Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove deprecated spool queue from Beats #28869

Merged
merged 4 commits into from
Nov 23, 2021
Merged

Conversation

kvch
Copy link
Contributor

@kvch kvch commented Nov 8, 2021

What does this PR do?

This PR removes the deprecated disk spool queue from libbeat.

Why is it important?

The feature hasn't been GAed. Instead it was deprecated in favour of the disk queue: https://www.elastic.co/guide/en/beats/filebeat/current/configuring-internal-queue.html#configuration-internal-queue-disk

Checklist

- [ ] My code follows the style guidelines of this project
- [ ] I have commented my code, particularly in hard-to-understand areas

  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
    - [ ] I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@kvch kvch requested review from a team as code owners November 8, 2021 11:25
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 8, 2021
@kvch kvch requested a review from faec November 8, 2021 11:25
@kvch kvch added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Nov 8, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 8, 2021
@mergify
Copy link
Contributor

mergify bot commented Nov 8, 2021

This pull request does not have a backport label. Could you fix it @kvch? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@kvch kvch added backport-v8.0.0 Automated backport with mergify needs_team Indicates that the issue/PR needs a Team:* label labels Nov 8, 2021
@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Nov 8, 2021
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 8, 2021
@botelastic
Copy link

botelastic bot commented Nov 8, 2021

This pull request doesn't have a Team:<team> label.

@mergify mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Nov 8, 2021
@elasticmachine
Copy link
Collaborator

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-11-22T15:21:06.407+0000

  • Duration: 111 min 54 sec

  • Commit: d5023a9

Test stats 🧪

Test Results
Failed 1
Passed 19916
Skipped 1363
Total 21280

Test errors 1

Expand to view the tests failures

Build&Test / metricbeat-pythonIntegTest / test_metricsets_0_stats – metricbeat.tests.system.test_beat.Test
    Expand to view the error details

     Exception: Key 'beat.stats.beat.uuid' found in event is not documented! 
    

    Expand to view the stacktrace

     a = (<test_beat.Test testMethod=test_metricsets_0_stats>,)
    
        @wraps(func)
        def standalone_func(*a):
    >       return func(*(a + p.args), **p.kwargs)
    
    ../build/ve/docker/lib/python3.9/site-packages/parameterized/parameterized.py:518: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    tests/system/test_beat.py:21: in test_metricsets
        self.check_metricset("beat", metricset, [self.compose_host("metricbeat")], self.FIELDS + ["service"])
    tests/system/metricbeat.py:107: in check_metricset
        self.assert_fields_are_documented(evt)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    self = <test_beat.Test testMethod=test_metricsets_0_stats>
    evt = {'@timestamp': '2021-11-22T16:50:27.325Z', 'agent': {'ephemeral_id': '5e3090ed-ecc1-4a93-a70f-c41593a1c783', 'id': '7f...limit': {'hard': 1048576, 'soft': 1048576}, 'open': 10}, ...}, 'type': 'metricbeat'}, 'ecs': {'version': '8.0.0'}, ...}
    
        def assert_fields_are_documented(self, evt):
            """
            Assert that all keys present in evt are documented in fields.yml.
            This reads from the global fields.yml, means `make collect` has to be run before the check.
            """
            expected_fields, dict_fields, aliases = self.load_fields()
            flat = self.flatten_object(evt, dict_fields)
        
            def field_pattern_match(pattern, key):
                pattern_fields = pattern.split(".")
                key_fields = key.split(".")
                if len(pattern_fields) != len(key_fields):
                    return False
                for i in range(len(pattern_fields)):
                    if pattern_fields[i] == "*":
                        continue
                    if pattern_fields[i] != key_fields[i]:
                        return False
                return True
        
            def is_documented(key, docs):
                if key in docs:
                    return True
                for pattern in (f for f in docs if "*" in f):
                    if field_pattern_match(pattern, key):
                        return True
                return False
        
            for key in flat.keys():
                metaKey = key.startswith('@metadata.')
                # Range keys as used in 'date_range' etc will not have docs of course
                isRangeKey = key.split('.')[-1] in ['gte', 'gt', 'lte', 'lt']
                if not(is_documented(key, expected_fields) or metaKey or isRangeKey):
    >               raise Exception("Key '{}' found in event is not documented!".format(key))
    E               Exception: Key 'beat.stats.beat.uuid' found in event is not documented!
    
    ../libbeat/tests/system/beat/beat.py:732: Exception 
    

Steps errors 7

Expand to view the steps failures

libbeat-stress-tests - make STRESS_TEST_OPTIONS='-timeout=20m -race -v -parallel 1' -C libbeat stres
  • Took 10 min 17 sec . View more details here
  • Description: make STRESS_TEST_OPTIONS='-timeout=20m -race -v -parallel 1' -C libbeat stress-tests
libbeat-stress-tests - make STRESS_TEST_OPTIONS='-timeout=20m -race -v -parallel 1' -C libbeat stres
  • Took 10 min 3 sec . View more details here
  • Description: make STRESS_TEST_OPTIONS='-timeout=20m -race -v -parallel 1' -C libbeat stress-tests
libbeat-stress-tests - make STRESS_TEST_OPTIONS='-timeout=20m -race -v -parallel 1' -C libbeat stres
  • Took 10 min 4 sec . View more details here
  • Description: make STRESS_TEST_OPTIONS='-timeout=20m -race -v -parallel 1' -C libbeat stress-tests
metricbeat-pythonIntegTest - mage pythonIntegTest
  • Took 31 min 52 sec . View more details here
  • Description: mage pythonIntegTest
metricbeat-pythonIntegTest - mage pythonIntegTest
  • Took 22 min 46 sec . View more details here
  • Description: mage pythonIntegTest
metricbeat-pythonIntegTest - mage pythonIntegTest
  • Took 22 min 50 sec . View more details here
  • Description: mage pythonIntegTest
Error signal
  • Took 0 min 0 sec . View more details here
  • Description: Error 'hudson.AbortException: script returned exit code 2'

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors 1

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

  • Name: Build&Test / metricbeat-pythonIntegTest / test_metricsets_0_stats – metricbeat.tests.system.test_beat.Test

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@kvch
Copy link
Contributor Author

kvch commented Nov 23, 2021

Failing tests are fixed in #29094

Copy link
Contributor

@faec faec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

beautiful :D

@kvch kvch merged commit 260509f into elastic:master Nov 23, 2021
mergify bot pushed a commit that referenced this pull request Nov 23, 2021
v1v added a commit to v1v/beats that referenced this pull request Nov 24, 2021
…ws-on-file-changes

* upstream/master:
  override host on statsd metricset (elastic#29103)
  Skip config check in autodiscover for duplicated configurations (elastic#29048)
  Change "filebeat.config.modules.enabled" to "true" (elastic#28769)
  Remove deprecated spool queue from Beats (elastic#28869)
  Add `beat` field back to beat.stats (elastic#29094)
  Revert "Move labels and annotations under kubernetes.namespace. (elastic#27917)" (elastic#29069)
  heartbeat: remove w2008 in the CI (elastic#29093)
  Remove deprecated `--template` and `--index-policy` flags (elastic#28870)
  Fix parsing of apache trace log levels (elastic#28717)
  [Elastic-Agent] IUse itnernal port for local fleet server (elastic#28993)
  [Heartbeat] Log error on dupe monitor ID instead of strict req (elastic#29041)
  Enable pprof for elastic-agent and beats (elastic#28983)
kvch added a commit that referenced this pull request Nov 24, 2021
(cherry picked from commit 260509f)

Co-authored-by: Noémi Ványi <[email protected]>
v1v added a commit to v1v/beats that referenced this pull request Nov 30, 2021
* upstream/master: (577 commits)
  Remove Journalbeat (elastic#29131)
  Add note that there is no warranty or support for generator code (elastic#28797)
  packetbeat: preparation for npcap addition (elastic#29017)
  Use the generic helper for opening file to read in filestream (elastic#29180)
  Workflow for macos (elastic#29174)
  Fix `decode_json_fields` processor to always add error key (elastic#29107)
  Workflow for macos (elastic#29156)
  Fix agent download timeout values in yaml files (elastic#29039)
  Added workflow file for builds with macos (elastic#29148)
  CI: enable AWS cloud testing on a PR basis if changes in certain files (elastic#29047)
  Remove links to Journalbeat (elastic#29134)
  Fix rds metadata in cloudwatch metricset (elastic#29106)
  [mergify]: notify conflicts in PRs that are still open (elastic#29122)
  Use NamedWatcher in Agent's k8s provider (elastic#29095)
  override host on statsd metricset (elastic#29103)
  Skip config check in autodiscover for duplicated configurations (elastic#29048)
  Change "filebeat.config.modules.enabled" to "true" (elastic#28769)
  Remove deprecated spool queue from Beats (elastic#28869)
  Add `beat` field back to beat.stats (elastic#29094)
  Revert "Move labels and annotations under kubernetes.namespace. (elastic#27917)" (elastic#29069)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v8.0.0 Automated backport with mergify Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants