Flaky Test [Build&Test / metricbeat-pythonIntegTest / metricbeat.module.logstash.test_logstash.Test.test_node_stats]

[kaiyan-sheng]

Flaky Test

• Test Name: test_logstash.Test.test_node_stats
• Link:

  beats/metricbeat/module/logstash/test_logstash.py

  Line 28 in a78436b

  def test_node_stats(self):

• Branch: 7.x
• Artifact Link:
  • https://beats-ci.elastic.co/job/Beats/job/beats/job/7.x/lastFailedBuild/testReport/junit/metricbeat.module.logstash.test_logstash/Test/Build_Test___metricbeat_pythonIntegTest___test_node_stats/
  • https://beats-ci.elastic.co/job/Beats/job/beats/job/PR-26461/1/testReport/junit/metricbeat.module.logstash.test_logstash/Test/Build_Test___metricbeat_pythonIntegTest___test_node_stats/

Stack Trace

Error Message
AssertionError: Element counts were not equal:
First has 1, Second has 0:  'logstash'
First has 0, Second has 1:  'error'
Stacktrace
self = <test_logstash.Test testMethod=test_node_stats>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, "integration test")
    def test_node_stats(self):
        """
        logstash node_stats metricset test
        """
>       self.check_metricset("logstash", "node_stats", self.get_hosts(), self.FIELDS)

module/logstash/test_logstash.py:30: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/metricbeat.py:105: in check_metricset
    self.assertCountEqual(self.de_dot(fields), evt.keys())
E   AssertionError: Element counts were not equal:
E   First has 1, Second has 0:  'logstash'
E   First has 0, Second has 1:  'error'
REST API
Jenkins 2.289.1

[kaiyan-sheng]

Maybe this is related to #25043 ?

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[matschaffer]

@kaiyan-sheng sorry if this info is obvious and I'm just missing this but how would I run this test in a way I can also directly see the data it's trying to check?

[jsoriano]

@matschaffer when this test fails, it prints to stdout the checked events, I am not sure why this doesn't appear in the report, but it appears in the full logs, and it will appear if reproduced locally (though this may be difficult).

This is an example of stdout when these tests failed (from here):

[2021-06-24T06:39:36.859Z] [{'@timestamp': '2021-06-24T06:30:12.135Z', 'service': {'type': 'logstash', 'address': 'http://172.20.0.3:9600/_node/stats'}, 'ecs': {'version': '1.9.0'}, 'host': {'name': 'beats-ci-immutable-ubuntu-1804-1624511948878089174'}, 'agent': {'ephemeral_id': '8b89e757-00f4-4723-805f-e57bc15a8c98', 'id': '8eca0241-c2b4-474f-baf7-9ebcf6023ada', 'name': 'beats-ci-immutable-ubuntu-1804-1624511948878089174', 'type': 'metricbeat', 'version': '7.13.3', 'hostname': 'beats-ci-immutable-ubuntu-1804-1624511948878089174'}, 'event': {'dataset': 'logstash.node.stats', 'module': 'logstash', 'duration': 78804209}, 'error': {'message': 'failure applying node stats schema: 1 error: wrong format in `events.in`: expected integer, found <nil>'}, 'metricset': {'period': 1000, 'name': 'node_stats'}}]
[2021-06-24T06:39:36.859Z] {'@timestamp': '2021-06-24T06:30:12.135Z', 'service': {'type': 'logstash', 'address': 'http://172.20.0.3:9600/_node/stats'}, 'ecs': {'version': '1.9.0'}, 'host': {'name': 'beats-ci-immutable-ubuntu-1804-1624511948878089174'}, 'agent': {'ephemeral_id': '8b89e757-00f4-4723-805f-e57bc15a8c98', 'id': '8eca0241-c2b4-474f-baf7-9ebcf6023ada', 'name': 'beats-ci-immutable-ubuntu-1804-1624511948878089174', 'type': 'metricbeat', 'version': '7.13.3', 'hostname': 'beats-ci-immutable-ubuntu-1804-1624511948878089174'}, 'event': {'dataset': 'logstash.node.stats', 'module': 'logstash', 'duration': 78804209}, 'error': {'message': 'failure applying node stats schema: 1 error: wrong format in `events.in`: expected integer, found <nil>'}, 'metricset': {'period': 1000, 'name': 'node_stats'}}

This error seems relevant:

failure applying node stats schema: 1 error: wrong format in `events.in`: expected integer, found <nil>

I'd say that the problem here is that the healthcheck in the docker image is not waiting enough for the service to be available.