Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add linux disk IO metrics to system/process #37213

Merged

Conversation

fearful-symmetry
Copy link
Contributor

Proposed commit message

Part of #7461

This actually pulls in the changes from elastic-agent-system-metrics, and adds linux-only disk IO metrics to system/process.

No actual code changes here, see elastic/elastic-agent-system-metrics#114

Checklist

  • My code follows the style guidelines of this project
    - [ ] I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
    - [ ] I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@fearful-symmetry fearful-symmetry added the Team:Elastic-Agent Label for the Agent team label Nov 28, 2023
@fearful-symmetry fearful-symmetry self-assigned this Nov 28, 2023
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Nov 28, 2023
Copy link
Contributor

mergify bot commented Nov 28, 2023

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @fearful-symmetry? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@fearful-symmetry fearful-symmetry added the backport-skip Skip notification from the automated backport with mergify label Nov 28, 2023
@elasticmachine
Copy link
Collaborator

❕ Build Aborted

Either there was a build timeout or someone aborted the build.

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Duration: 8 min 39 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@elasticmachine
Copy link
Collaborator

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-11-28T04:10:12.754+0000

  • Duration: 128 min 12 sec

Test stats 🧪

Test Results
Failed 6
Passed 28158
Skipped 1882
Total 30046

Test errors 6

Expand to view the tests failures

Build&Test / metricbeat-pythonIntegTest / test_process – metricbeat.module.system.test_system.Test
    Expand to view the error details

     AssertionError: Element counts were not equal:
    First has 0, Second has 1:  'io' 
    

    Expand to view the stacktrace

     self = <test_system.Test testMethod=test_process>
    
        @unittest.skipUnless(re.match("(?i)win|linux|darwin|freebsd", sys.platform), "os")
        def test_process(self):
            """
            Test system/process output.
            """
            self.render_config_template(modules=[{
                "name": "system",
                "metricsets": ["process"],
                "period": "5s",
                "extras": {
                    "process.env.whitelist": ["PATH"],
                    "process.include_cpu_ticks": True,
        
                    # Remove 'percpu' prior to checking documented fields because its keys are dynamic.
                    "process.include_per_cpu": False,
                }
            }])
            self.run_beat_and_stop()
        
            output = self.read_output_json()
            self.assertGreater(len(output), 0)
        
            found_cmdline = False
            for evt in output:
                process = evt["system"]["process"]
                # Not all process will have 'cmdline' due to permission issues,
                # especially on Windows. Therefore we ensure at least some of
                # them will have it.
                found_cmdline |= "cmdline" in process
        
                # Remove 'env' prior to checking documented fields because its keys are dynamic.
                process.pop("env", None)
                self.assert_fields_are_documented(evt)
        
                # Remove optional keys.
                process.pop("cgroup", None)
                process.pop("fd", None)
                process.pop("cmdline", None)
                process.pop("num_threads", None)
        
    >           self.assertCountEqual(SYSTEM_PROCESS_FIELDS, process.keys())
    E           AssertionError: Element counts were not equal:
    E           First has 0, Second has 1:  'io'
    
    module/system/test_system.py:439: AssertionError 
    

Build&Test / metricbeat-pythonIntegTest / test_process_unix – metricbeat.module.system.test_system.Test
    Expand to view the error details

     AssertionError: Element counts were not equal:
    First has 0, Second has 1:  'io' 
    

    Expand to view the stacktrace

     self = <test_system.Test testMethod=test_process_unix>
    
        @unittest.skipUnless(re.match("(?i)linux|darwin|freebsd", sys.platform), "os")
        def test_process_unix(self):
            """
            Test system/process output checking it has got all expected fields specific of unix systems and no extra ones.
            """
        
            self.render_config_template(
                modules=[{
                    "name": "system",
                    "metricsets": ["process"],
                    "period": "5s",
                    "extras": {
                        "process.env.whitelist": ["PATH"],
                        "process.include_cpu_ticks": True,
        
                        # Remove 'percpu' prior to checking documented fields because its keys are dynamic.
                        "process.include_per_cpu": False,
                    },
                }],
                # Some info is only guaranteed in processes with permissions, check
                # only on own processes.
                processors=[{
                    "drop_event": {
                        "when": "not.equals.user.name: " + getpass.getuser(),
                    },
                }],
            )
            self.run_beat_and_stop()
        
            output = self.read_output_json()
            self.assertGreater(len(output), 0)
        
            found_fd = False
            found_env = False
            found_cwd = not sys.platform.startswith("linux")
            for evt in output:
                found_cwd |= "working_directory" in evt["process"]
        
                process = evt["system"]["process"]
                found_fd |= "fd" in process
                found_env |= "env" in process
        
                # Remove 'env' prior to checking documented fields because its keys are dynamic.
                process.pop("env", None)
                self.assert_fields_are_documented(evt)
        
                # Remove optional keys.
                process.pop("cgroup", None)
                process.pop("cmdline", None)
                process.pop("fd", None)
                process.pop("num_threads", None)
        
    >           self.assertCountEqual(SYSTEM_PROCESS_FIELDS, process.keys())
    E           AssertionError: Element counts were not equal:
    E           First has 0, Second has 1:  'io'
    
    module/system/test_system.py:497: AssertionError 
    

Build&Test / metricbeat-unitTest / test_process – metricbeat.module.system.test_system.Test
    Expand to view the error details

     AssertionError: Element counts were not equal:
    First has 0, Second has 1:  'io' 
    

    Expand to view the stacktrace

     self = <test_system.Test testMethod=test_process>
    
        @unittest.skipUnless(re.match("(?i)win|linux|darwin|freebsd", sys.platform), "os")
        def test_process(self):
            """
            Test system/process output.
            """
            self.render_config_template(modules=[{
                "name": "system",
                "metricsets": ["process"],
                "period": "5s",
                "extras": {
                    "process.env.whitelist": ["PATH"],
                    "process.include_cpu_ticks": True,
        
                    # Remove 'percpu' prior to checking documented fields because its keys are dynamic.
                    "process.include_per_cpu": False,
                }
            }])
            self.run_beat_and_stop()
        
            output = self.read_output_json()
            self.assertGreater(len(output), 0)
        
            found_cmdline = False
            for evt in output:
                process = evt["system"]["process"]
                # Not all process will have 'cmdline' due to permission issues,
                # especially on Windows. Therefore we ensure at least some of
                # them will have it.
                found_cmdline |= "cmdline" in process
        
                # Remove 'env' prior to checking documented fields because its keys are dynamic.
                process.pop("env", None)
                self.assert_fields_are_documented(evt)
        
                # Remove optional keys.
                process.pop("cgroup", None)
                process.pop("fd", None)
                process.pop("cmdline", None)
                process.pop("num_threads", None)
        
    >           self.assertCountEqual(SYSTEM_PROCESS_FIELDS, process.keys())
    E           AssertionError: Element counts were not equal:
    E           First has 0, Second has 1:  'io'
    
    module/system/test_system.py:439: AssertionError 
    

Build&Test / metricbeat-unitTest / test_process_unix – metricbeat.module.system.test_system.Test
    Expand to view the error details

     AssertionError: Element counts were not equal:
    First has 0, Second has 1:  'io' 
    

    Expand to view the stacktrace

     self = <test_system.Test testMethod=test_process_unix>
    
        @unittest.skipUnless(re.match("(?i)linux|darwin|freebsd", sys.platform), "os")
        def test_process_unix(self):
            """
            Test system/process output checking it has got all expected fields specific of unix systems and no extra ones.
            """
        
            self.render_config_template(
                modules=[{
                    "name": "system",
                    "metricsets": ["process"],
                    "period": "5s",
                    "extras": {
                        "process.env.whitelist": ["PATH"],
                        "process.include_cpu_ticks": True,
        
                        # Remove 'percpu' prior to checking documented fields because its keys are dynamic.
                        "process.include_per_cpu": False,
                    },
                }],
                # Some info is only guaranteed in processes with permissions, check
                # only on own processes.
                processors=[{
                    "drop_event": {
                        "when": "not.equals.user.name: " + getpass.getuser(),
                    },
                }],
            )
            self.run_beat_and_stop()
        
            output = self.read_output_json()
            self.assertGreater(len(output), 0)
        
            found_fd = False
            found_env = False
            found_cwd = not sys.platform.startswith("linux")
            for evt in output:
                found_cwd |= "working_directory" in evt["process"]
        
                process = evt["system"]["process"]
                found_fd |= "fd" in process
                found_env |= "env" in process
        
                # Remove 'env' prior to checking documented fields because its keys are dynamic.
                process.pop("env", None)
                self.assert_fields_are_documented(evt)
        
                # Remove optional keys.
                process.pop("cgroup", None)
                process.pop("cmdline", None)
                process.pop("fd", None)
                process.pop("num_threads", None)
        
    >           self.assertCountEqual(SYSTEM_PROCESS_FIELDS, process.keys())
    E           AssertionError: Element counts were not equal:
    E           First has 0, Second has 1:  'io'
    
    module/system/test_system.py:497: AssertionError 
    

Build&Test / metricbeat-windows-2022-windows-2022 / test_process – metricbeat.module.system.test_system.Test
    Expand to view the error details

     AssertionError: Element counts were not equal:
    First has 0, Second has 1:  'io' 
    

    Expand to view the stacktrace

     self = <test_system.Test testMethod=test_process>
    
        @unittest.skipUnless(re.match("(?i)win|linux|darwin|freebsd", sys.platform), "os")
        def test_process(self):
            """
            Test system/process output.
            """
            self.render_config_template(modules=[{
                "name": "system",
                "metricsets": ["process"],
                "period": "5s",
                "extras": {
                    "process.env.whitelist": ["PATH"],
                    "process.include_cpu_ticks": True,
        
                    # Remove 'percpu' prior to checking documented fields because its keys are dynamic.
                    "process.include_per_cpu": False,
                }
            }])
            self.run_beat_and_stop()
        
            output = self.read_output_json()
            self.assertGreater(len(output), 0)
        
            found_cmdline = False
            for evt in output:
                process = evt["system"]["process"]
                # Not all process will have 'cmdline' due to permission issues,
                # especially on Windows. Therefore we ensure at least some of
                # them will have it.
                found_cmdline |= "cmdline" in process
        
                # Remove 'env' prior to checking documented fields because its keys are dynamic.
                process.pop("env", None)
                self.assert_fields_are_documented(evt)
        
                # Remove optional keys.
                process.pop("cgroup", None)
                process.pop("fd", None)
                process.pop("cmdline", None)
                process.pop("num_threads", None)
        
    >           self.assertCountEqual(SYSTEM_PROCESS_FIELDS, process.keys())
    E           AssertionError: Element counts were not equal:
    E           First has 0, Second has 1:  'io'
    
    module\system\test_system.py:439: AssertionError 
    

Build&Test / metricbeat-windows-2016-windows-2016 / test_process – metricbeat.module.system.test_system.Test
    Expand to view the error details

     AssertionError: Element counts were not equal:
    First has 0, Second has 1:  'io' 
    

    Expand to view the stacktrace

     self = <test_system.Test testMethod=test_process>
    
        @unittest.skipUnless(re.match("(?i)win|linux|darwin|freebsd", sys.platform), "os")
        def test_process(self):
            """
            Test system/process output.
            """
            self.render_config_template(modules=[{
                "name": "system",
                "metricsets": ["process"],
                "period": "5s",
                "extras": {
                    "process.env.whitelist": ["PATH"],
                    "process.include_cpu_ticks": True,
        
                    # Remove 'percpu' prior to checking documented fields because its keys are dynamic.
                    "process.include_per_cpu": False,
                }
            }])
            self.run_beat_and_stop()
        
            output = self.read_output_json()
            self.assertGreater(len(output), 0)
        
            found_cmdline = False
            for evt in output:
                process = evt["system"]["process"]
                # Not all process will have 'cmdline' due to permission issues,
                # especially on Windows. Therefore we ensure at least some of
                # them will have it.
                found_cmdline |= "cmdline" in process
        
                # Remove 'env' prior to checking documented fields because its keys are dynamic.
                process.pop("env", None)
                self.assert_fields_are_documented(evt)
        
                # Remove optional keys.
                process.pop("cgroup", None)
                process.pop("fd", None)
                process.pop("cmdline", None)
                process.pop("num_threads", None)
        
    >           self.assertCountEqual(SYSTEM_PROCESS_FIELDS, process.keys())
    E           AssertionError: Element counts were not equal:
    E           First has 0, Second has 1:  'io'
    
    module\system\test_system.py:439: AssertionError 
    

Steps errors 13

Expand to view the steps failures

Show only the first 10 steps failures

metricbeat-pythonIntegTest - mage pythonIntegTest
  • Took 6 min 16 sec . View more details here
  • Description: mage pythonIntegTest
metricbeat-pythonIntegTest - mage pythonIntegTest
  • Took 2 min 1 sec . View more details here
  • Description: mage pythonIntegTest
metricbeat-pythonIntegTest - mage pythonIntegTest
  • Took 2 min 1 sec . View more details here
  • Description: mage pythonIntegTest
metricbeat-windows-2022-windows-2022 - mage build unitTest
  • Took 9 min 4 sec . View more details here
  • Description: mage build unitTest
metricbeat-windows-2022-windows-2022 - mage build unitTest
  • Took 4 min 42 sec . View more details here
  • Description: mage build unitTest
metricbeat-windows-2022-windows-2022 - mage build unitTest
  • Took 4 min 41 sec . View more details here
  • Description: mage build unitTest
metricbeat-windows-2016-windows-2016 - mage build unitTest
  • Took 9 min 26 sec . View more details here
  • Description: mage build unitTest
metricbeat-windows-2016-windows-2016 - mage build unitTest
  • Took 4 min 54 sec . View more details here
  • Description: mage build unitTest
metricbeat-windows-2016-windows-2016 - mage build unitTest
  • Took 4 min 55 sec . View more details here
  • Description: mage build unitTest
Error signal
  • Took 0 min 0 sec . View more details here
  • Description: Error 'hudson.AbortException: script returned exit code 1'

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors 6

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

  • Name: Build&Test / metricbeat-pythonIntegTest / test_process – metricbeat.module.system.test_system.Test
  • Name: Build&Test / metricbeat-pythonIntegTest / test_process_unix – metricbeat.module.system.test_system.Test
  • Name: Build&Test / metricbeat-unitTest / test_process – metricbeat.module.system.test_system.Test
  • Name: Build&Test / metricbeat-unitTest / test_process_unix – metricbeat.module.system.test_system.Test
  • Name: Build&Test / metricbeat-windows-2022-windows-2022 / test_process – metricbeat.module.system.test_system.Test
  • Name: Build&Test / metricbeat-windows-2016-windows-2016 / test_process – metricbeat.module.system.test_system.Test

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

metricbeat/docs/fields.asciidoc Outdated Show resolved Hide resolved
metricbeat/docs/fields.asciidoc Outdated Show resolved Hide resolved
metricbeat/module/system/process/_meta/fields.yml Outdated Show resolved Hide resolved
metricbeat/module/system/process/_meta/fields.yml Outdated Show resolved Hide resolved
@elasticmachine
Copy link
Collaborator

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Start Time: 2023-11-28T18:25:15.199+0000

  • Duration: 176 min 59 sec

Test stats 🧪

Test Results
Failed 0
Passed 28164
Skipped 1882
Total 30046

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-11-28T21:16:32.149+0000

  • Duration: 186 min 6 sec

Test stats 🧪

Test Results
Failed 0
Passed 28164
Skipped 1882
Total 30046

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Copy link
Contributor

mergify bot commented Nov 29, 2023

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add-diskio-to-process upstream/add-diskio-to-process
git merge upstream/main
git push upstream add-diskio-to-process

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-11-29T22:47:15.441+0000

  • Duration: 178 min 38 sec

Test stats 🧪

Test Results
Failed 0
Passed 28164
Skipped 1882
Total 30046

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Copy link
Contributor

mergify bot commented Nov 30, 2023

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add-diskio-to-process upstream/add-diskio-to-process
git merge upstream/main
git push upstream add-diskio-to-process

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-12-01T17:38:18.324+0000

  • Duration: 180 min 50 sec

Test stats 🧪

Test Results
Failed 0
Passed 28164
Skipped 1882
Total 30046

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Scholar-Li pushed a commit to Scholar-Li/beats that referenced this pull request Feb 5, 2024
* add diskio data from system-metrics

* add changelog

* fix python tests

* fix docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-skip Skip notification from the automated backport with mergify Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants