Add Nginx Plus status

[mrkschan]

No description provided.

[mrkschan]

This is a WIP PR. Please let me know if the big chunk of copy-and-paste for reading the JSON from Nginx plus status endpoint is appropriate.

[elasticsearch-release]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

[elasticsearch-release]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

[ruflin]

I'm curious if we could move this to the nginx module? Long term we should probably make this even a general helper to be used for all http metricsets.

[ruflin]

Thanks for the additional metricsets. There is a new schema package which you could make use of to better parse the json documents: https://www.elastic.co/guide/en/beats/metricbeat/current/creating-metricsets.html#_parsing_and_normalizing_fields For an example have a look at the filebeat metricset. Let me know if you need some more details. In general the MetricSets should define all the fields that they are sending so we can make sure the types are correct.

It seems there is quite a bit of duplicated code in the Metricsets. I would hope part of it could be moved to the nginx module and then be reused.

[ruflin]

@mrkschan I just saw you pushed some additional commits. Can you ping me when it is ready for a review?

[mrkschan]

@ruflin, I'm trying to apply Schema to Nginx plus stream.upstreams. I'm not sure what to do with the peers array in each "upstream". Find below a sample stats retrieved from https://demo.nginx.com/status/stream/upstreams.

"upstreams": {
     "dns_udp_backends" : {
            "peers" : [
                 {
                        "connections" : 226042,
                        "active" : 0,
                        "state" : "up",
                        "unavail" : 0,
                        "downstart" : 0,
                        "fails" : 0,
                        "downtime" : 0,
                        "weight" : 2,
                        "health_checks" : {
                             "last_passed" : true,
                             "checks" : 118399,
                             "fails" : 0,
                             "unhealthy" : 0
                        }
                 },
             :
             :

[ruflin]

Arrays area always tricky. We have similar problems in other metricsets. So far we always tried to build a new metricset out of it that sends for each entry an event. That means in your case, each peer would be an event. If you go this way, you would probably have to add one more field to the peer which is type or something similar and contains for example unused_tcp_backends. The metricset could be called upstreampeer for example (I'm sure there are better name).

One of the questions that often helps to answer the question is on how this data will be consumed? What metrics is someone going to visualise? Will it be all peers with a specific ip and type? Or is it the number of peers per count? ...

[mrkschan]

Closing this ... will create a up-to-date PR later.

[ruflin]

@mrkschan Thanks