New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP]: 34125 output name issue #34143

Closed

AndersonQ wants to merge 8 commits into elastic:8.6 from AndersonQ:34125-output-name-issue

Member

AndersonQ commented Dec 29, 2022

See #34125 (comment) for context

What does this PR do?

Why is it important?

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

-->

AndersonQ added 5 commits

December 28, 2022 16:17


fix make clean

 - when cleaning the project it ignore 'permission denied' errors
 - remove broken make call


improve warning to surface the issue and make it a info log

6b8add5


properly set the beat name

c8c415a


reformat some comments

821741b

WIP

58b94b0

AndersonQ self-assigned this

botelastic bot added the needs_team label

AndersonQ added 3 commits

December 29, 2022 18:59

582d51a

9de3e57

Collaborator

elasticmachine commented Dec 29, 2022 •

edited by jenkins-beats-ci bot

Loading

❕ Build Aborted

Either there was a build timeout or someone aborted the build.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-12-29T18:04:31.809+0000
Duration: 364 min 35 sec

Test stats 🧪

Test	Results
Failed	2475
Passed	23302
Skipped	2047
Total	27824

Test errors

Expand to view the tests failures

> Show only the first 10 test failures

`Build&Test / x-pack/packetbeat-windows-2022-system-windows-2022 / TestWindowsNpcapInstaller – github.com/elastic/beats/v7/x-pack/packetbeat/tests/system`

Expand to view the error details

 Failed

Expand to view the stacktrace

 === RUN   TestWindowsNpcapInstaller
    app_test.go:46: Error:
         Exiting: error getting 'outputs' from config: missing field accessing 'outputs'
    app_test.go:51: could not stat install.log: CreateFile C:\Program Files\Npcap\install.log: The system cannot find the path specified.
    app_test.go:56: unexpected npcap version installed: want:1.71 have:WinPcap version 4.1.3 (packet.dll version 4.1.0.2980), based on libpcap version 1.0 branch 1_0_rel0b (20091008)
--- FAIL: TestWindowsNpcapInstaller (0.11s)

`Build&Test / x-pack/packetbeat-windows-2022-system-windows-2022 / TestDevices – github.com/elastic/beats/v7/x-pack/packetbeat/tests/system`

Expand to view the error details

 Failed

Expand to view the stacktrace

 === RUN   TestDevices
    app_test.go:63: Output:
         0: \Device\NPF_{857B4835-7F14-46DE-9157-8D82E3754E0F} (Red Hat) (10.224.1.181)
        PASS
        coverage: 60.0% of statements in ./...
    app_test.go:71: Expect interfaces:
         [5:Ethernet 3:up|broadcast|multicast 1:Loopback Pseudo-Interface 1:up|loopback|multicast]
    app_test.go:74: 
        	Error Trace:	C:\Users\jenkins\workspace\PR-34143-4-528000d0-9b99-4c73-9393-f1e593a61677\src\github.com\elastic\beats\x-pack\packetbeat\tests\system\app_test.go:74
        	Error:      	"0: \\Device\\NPF_{857B4835-7F14-46DE-9157-8D82E3754E0F} (Red Hat) (10.224.1.181)\nPASS\ncoverage: 60.0% of statements in ./..." does not contain "Ethernet 3"
        	Test:       	TestDevices
    app_test.go:74: 
        	Error Trace:	C:\Users\jenkins\workspace\PR-34143-4-528000d0-9b99-4c73-9393-f1e593a61677\src\github.com\elastic\beats\x-pack\packetbeat\tests\system\app_test.go:74
        	Error:      	"0: \\Device\\NPF_{857B4835-7F14-46DE-9157-8D82E3754E0F} (Red Hat) (10.224.1.181)\nPASS\ncoverage: 60.0% of statements in ./..." does not contain "Loopback Pseudo-Interface 1"
        	Test:       	TestDevices
--- FAIL: TestDevices (0.27s)

`Build&Test / x-pack/auditbeat-build / test_metricset_login – x-pack.auditbeat.tests.system.test_metricsets.Test`

Expand to view the error details

 beat.beat.WaitTimeoutError: Timeout waiting for condition 'cond'. Waited 20 seconds:

Expand to view the stacktrace

 self = <test_metricsets.Test testMethod=test_metricset_login>

    @unittest.skipUnless(sys.platform.startswith('linux'), "Only implemented for Linux")
    @unittest.skipIf(sys.byteorder != "little", "Test only implemented for little-endian systems")
    def test_metricset_login(self):
        """
        login metricset collects information about logins (successful and failed) and system restarts.
        """
    
        fields = ["event.origin", "event.outcome", "message", "process.pid", "source.ip",
                  "user.name", "user.terminal"]
    
        config = {
            "login.wtmp_file_pattern": os.path.abspath(os.path.join(self.beat_path, "module/system/login/testdata/wtmp*")),
            "login.btmp_file_pattern": os.path.abspath(os.path.join(self.beat_path, "module/system/login/testdata/btmp*")),
        }
    
        # Metricset is beta and that generates a warning, TODO: remove later
>       self.check_metricset("system", "login", COMMON_FIELDS + fields, config, warnings_allowed=True)

tests/system/test_metricsets.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/auditbeat_xpack.py:42: in check_metricset
    self.wait_until(lambda: self.output_lines() > 0)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_metricsets.Test testMethod=test_metricset_login>
cond = <function AuditbeatXPackTest.check_metricset.<locals>.<lambda> at 0x7faf6a195510>
max_timeout = 20, poll_interval = 0.1, name = 'cond', err_msg = ''

    def wait_until(self, cond, max_timeout=20, poll_interval=0.1, name="cond", err_msg=""):
        """
        TODO: this can probably be a "wait_until_output_count", among other things, since that could actually use `self`, and this can become an internal function
        Waits until the cond function returns true,
        or until the max_timeout is reached. Calls the cond
        function every poll_interval seconds.
    
        If the max_timeout is reached before cond() returns
        true, an exception is raised.
        """
        start = datetime.now()
        while not cond():
            if datetime.now() - start > timedelta(seconds=max_timeout):
                raise WaitTimeoutError(
>                   f"Timeout waiting for condition '{name}'. Waited {max_timeout} seconds: {err_msg}")
E               beat.beat.WaitTimeoutError: Timeout waiting for condition 'cond'. Waited 20 seconds:

../../libbeat/tests/system/beat/beat.py:447: WaitTimeoutError

`Build&Test / x-pack/auditbeat-build / test_metricset_package – x-pack.auditbeat.tests.system.test_metricsets.Test`

Expand to view the error details

 beat.beat.WaitTimeoutError: Timeout waiting for condition 'cond'. Waited 20 seconds:

Expand to view the stacktrace

 self = <test_metricsets.Test testMethod=test_metricset_package>

    @pytest.mark.flaky(reruns=1, reruns_delay=10)
    @unittest.skipIf(sys.platform == "win32", "Not implemented for Windows")
    @unittest.skipIf(sys.platform.startswith('linux') and not (os.path.isdir("/var/lib/dpkg") or os.path.isdir("/var/lib/rpm")),
                     "Only implemented for dpkg and rpm")
    @unittest.skipIf(sys.platform.startswith('darwin'), "See https:/elastic/beats/issues/21308")
    def test_metricset_package(self):
        """
        package metricset collects information about installed packages on a system.
        """
    
        fields = ["system.audit.package.entity_id", "system.audit.package.name", "system.audit.package.version"]
    
        # Metricset is beta and that generates a warning, TODO: remove later
>       self.check_metricset("system", "package", COMMON_FIELDS + fields, warnings_allowed=True)

tests/system/test_metricsets.py:60: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/auditbeat_xpack.py:42: in check_metricset
    self.wait_until(lambda: self.output_lines() > 0)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_metricsets.Test testMethod=test_metricset_package>
cond = <function AuditbeatXPackTest.check_metricset.<locals>.<lambda> at 0x7faf6a195840>
max_timeout = 20, poll_interval = 0.1, name = 'cond', err_msg = ''

    def wait_until(self, cond, max_timeout=20, poll_interval=0.1, name="cond", err_msg=""):
        """
        TODO: this can probably be a "wait_until_output_count", among other things, since that could actually use `self`, and this can become an internal function
        Waits until the cond function returns true,
        or until the max_timeout is reached. Calls the cond
        function every poll_interval seconds.
    
        If the max_timeout is reached before cond() returns
        true, an exception is raised.
        """
        start = datetime.now()
        while not cond():
            if datetime.now() - start > timedelta(seconds=max_timeout):
                raise WaitTimeoutError(
>                   f"Timeout waiting for condition '{name}'. Waited {max_timeout} seconds: {err_msg}")
E               beat.beat.WaitTimeoutError: Timeout waiting for condition 'cond'. Waited 20 seconds:

../../libbeat/tests/system/beat/beat.py:447: WaitTimeoutError

`Build&Test / x-pack/auditbeat-build / test_metricset_user – x-pack.auditbeat.tests.system.test_metricsets.Test`

Expand to view the error details

 beat.beat.WaitTimeoutError: Timeout waiting for condition 'cond'. Waited 20 seconds:

Expand to view the stacktrace

 self = <test_metricsets.Test testMethod=test_metricset_user>

    @unittest.skipUnless(sys.platform.startswith('linux'), "Only implemented for Linux")
    def test_metricset_user(self):
        """
        user metricset collects information about users on a server.
        """
    
        fields = ["user.entity_id", "system.audit.user.name"]
    
>       self.check_metricset("system", "user", COMMON_FIELDS + fields)

tests/system/test_metricsets.py:89: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/auditbeat_xpack.py:42: in check_metricset
    self.wait_until(lambda: self.output_lines() > 0)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_metricsets.Test testMethod=test_metricset_user>
cond = <function AuditbeatXPackTest.check_metricset.<locals>.<lambda> at 0x7faf6a195950>
max_timeout = 20, poll_interval = 0.1, name = 'cond', err_msg = ''

    def wait_until(self, cond, max_timeout=20, poll_interval=0.1, name="cond", err_msg=""):
        """
        TODO: this can probably be a "wait_until_output_count", among other things, since that could actually use `self`, and this can become an internal function
        Waits until the cond function returns true,
        or until the max_timeout is reached. Calls the cond
        function every poll_interval seconds.
    
        If the max_timeout is reached before cond() returns
        true, an exception is raised.
        """
        start = datetime.now()
        while not cond():
            if datetime.now() - start > timedelta(seconds=max_timeout):
                raise WaitTimeoutError(
>                   f"Timeout waiting for condition '{name}'. Waited {max_timeout} seconds: {err_msg}")
E               beat.beat.WaitTimeoutError: Timeout waiting for condition 'cond'. Waited 20 seconds:

../../libbeat/tests/system/beat/beat.py:447: WaitTimeoutError

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_index_management – x-pack.metricbeat.tests.system.test_xpack_base.Test`

Expand to view the error details

 AssertionError: assert 1 == 0

Expand to view the stacktrace

 self = <test_xpack_base.Test testMethod=test_index_management>

    @unittest.skipUnless(INTEGRATION_TESTS, "integration test")
    def test_index_management(self):
        """
        Test that the template can be loaded with `setup --index-management`
        """
        es = Elasticsearch([self.get_elasticsearch_url()])
        self.render_config_template(
            modules=[{
                "name": "apache",
                "metricsets": ["status"],
                "hosts": ["localhost"],
            }],
            elasticsearch={"host": self.get_elasticsearch_url()},
        )
        exit_code = self.run_beat(extra_args=["setup", "--index-management", "-E", "setup.template.overwrite=true"])
    
>       assert exit_code == 0
E       AssertionError: assert 1 == 0

../../metricbeat/tests/system/test_base.py:57: AssertionError

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_broker_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

Expand to view the error details

 beat.beat.WaitTimeoutError: Timeout waiting for condition 'cond'. Waited 20 seconds:

Expand to view the stacktrace

 self = <test_activemq.ActiveMqTest_0 testMethod=test_broker_metrics_collected>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
    def test_broker_metrics_collected(self):
        self.render_config_template(modules=[self.get_activemq_module_config('broker')])
        proc = self.start_beat(home=self.beat_path)
>       self.wait_until(lambda: self.output_lines() > 0)

module/activemq/test_activemq.py:81: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_activemq.ActiveMqTest_0 testMethod=test_broker_metrics_collected>
cond = <function ActiveMqTest.test_broker_metrics_collected.<locals>.<lambda> at 0x7ff841688ee0>
max_timeout = 20, poll_interval = 0.1, name = 'cond', err_msg = ''

    def wait_until(self, cond, max_timeout=20, poll_interval=0.1, name="cond", err_msg=""):
        """
        TODO: this can probably be a "wait_until_output_count", among other things, since that could actually use `self`, and this can become an internal function
        Waits until the cond function returns true,
        or until the max_timeout is reached. Calls the cond
        function every poll_interval seconds.
    
        If the max_timeout is reached before cond() returns
        true, an exception is raised.
        """
        start = datetime.now()
        while not cond():
            if datetime.now() - start > timedelta(seconds=max_timeout):
>               raise WaitTimeoutError(
                    f"Timeout waiting for condition '{name}'. Waited {max_timeout} seconds: {err_msg}")
E               beat.beat.WaitTimeoutError: Timeout waiting for condition 'cond'. Waited 20 seconds:

../../libbeat/tests/system/beat/beat.py:446: WaitTimeoutError

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_queue_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

Expand to view the error details

 beat.beat.WaitTimeoutError: Timeout waiting for condition 'cond'. Waited 20 seconds:

Expand to view the stacktrace

 self = <test_activemq.ActiveMqTest_0 testMethod=test_queue_metrics_collected>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
    def test_queue_metrics_collected(self):
>       self.verify_destination_metrics_collection('queue')

module/activemq/test_activemq.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
module/activemq/test_activemq.py:57: in verify_destination_metrics_collection
    self.wait_until(lambda: self.destination_metrics_collected(destination_type, destination_name))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_activemq.ActiveMqTest_0 testMethod=test_queue_metrics_collected>
cond = <function ActiveMqTest.verify_destination_metrics_collection.<locals>.<lambda> at 0x7ff841759550>
max_timeout = 20, poll_interval = 0.1, name = 'cond', err_msg = ''

    def wait_until(self, cond, max_timeout=20, poll_interval=0.1, name="cond", err_msg=""):
        """
        TODO: this can probably be a "wait_until_output_count", among other things, since that could actually use `self`, and this can become an internal function
        Waits until the cond function returns true,
        or until the max_timeout is reached. Calls the cond
        function every poll_interval seconds.
    
        If the max_timeout is reached before cond() returns
        true, an exception is raised.
        """
        start = datetime.now()
        while not cond():
            if datetime.now() - start > timedelta(seconds=max_timeout):
>               raise WaitTimeoutError(
                    f"Timeout waiting for condition '{name}'. Waited {max_timeout} seconds: {err_msg}")
E               beat.beat.WaitTimeoutError: Timeout waiting for condition 'cond'. Waited 20 seconds:

../../libbeat/tests/system/beat/beat.py:446: WaitTimeoutError

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_topic_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_0`

Expand to view the error details

 beat.beat.WaitTimeoutError: Timeout waiting for condition 'cond'. Waited 20 seconds:

Expand to view the stacktrace

 self = <test_activemq.ActiveMqTest_0 testMethod=test_topic_metrics_collected>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
    def test_topic_metrics_collected(self):
>       self.verify_destination_metrics_collection('topic')

module/activemq/test_activemq.py:100: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
module/activemq/test_activemq.py:57: in verify_destination_metrics_collection
    self.wait_until(lambda: self.destination_metrics_collected(destination_type, destination_name))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_activemq.ActiveMqTest_0 testMethod=test_topic_metrics_collected>
cond = <function ActiveMqTest.verify_destination_metrics_collection.<locals>.<lambda> at 0x7ff842819310>
max_timeout = 20, poll_interval = 0.1, name = 'cond', err_msg = ''

    def wait_until(self, cond, max_timeout=20, poll_interval=0.1, name="cond", err_msg=""):
        """
        TODO: this can probably be a "wait_until_output_count", among other things, since that could actually use `self`, and this can become an internal function
        Waits until the cond function returns true,
        or until the max_timeout is reached. Calls the cond
        function every poll_interval seconds.
    
        If the max_timeout is reached before cond() returns
        true, an exception is raised.
        """
        start = datetime.now()
        while not cond():
            if datetime.now() - start > timedelta(seconds=max_timeout):
>               raise WaitTimeoutError(
                    f"Timeout waiting for condition '{name}'. Waited {max_timeout} seconds: {err_msg}")
E               beat.beat.WaitTimeoutError: Timeout waiting for condition 'cond'. Waited 20 seconds:

../../libbeat/tests/system/beat/beat.py:446: WaitTimeoutError

`Build&Test / x-pack/metricbeat-pythonIntegTest / test_broker_metrics_collected – x-pack.metricbeat.module.activemq.test_activemq.ActiveMqTest_1`

Expand to view the error details

 beat.beat.WaitTimeoutError: Timeout waiting for condition 'cond'. Waited 20 seconds:

Expand to view the stacktrace

 self = <test_activemq.ActiveMqTest_1 testMethod=test_broker_metrics_collected>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, 'integration test')
    def test_broker_metrics_collected(self):
        self.render_config_template(modules=[self.get_activemq_module_config('broker')])
        proc = self.start_beat(home=self.beat_path)
>       self.wait_until(lambda: self.output_lines() > 0)

module/activemq/test_activemq.py:81: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_activemq.ActiveMqTest_1 testMethod=test_broker_metrics_collected>
cond = <function ActiveMqTest.test_broker_metrics_collected.<locals>.<lambda> at 0x7ff841688dc0>
max_timeout = 20, poll_interval = 0.1, name = 'cond', err_msg = ''

    def wait_until(self, cond, max_timeout=20, poll_interval=0.1, name="cond", err_msg=""):
        """
        TODO: this can probably be a "wait_until_output_count", among other things, since that could actually use `self`, and this can become an internal function
        Waits until the cond function returns true,
        or until the max_timeout is reached. Calls the cond
        function every poll_interval seconds.
    
        If the max_timeout is reached before cond() returns
        true, an exception is raised.
        """
        start = datetime.now()
        while not cond():
            if datetime.now() - start > timedelta(seconds=max_timeout):
>               raise WaitTimeoutError(
                    f"Timeout waiting for condition '{name}'. Waited {max_timeout} seconds: {err_msg}")
E               beat.beat.WaitTimeoutError: Timeout waiting for condition 'cond'. Waited 20 seconds:

../../libbeat/tests/system/beat/beat.py:446: WaitTimeoutError

Steps errors

Expand to view the steps failures

Show only the first 10 steps failures

`x-pack/metricbeat-windows-2022-windows-2022 - mage build unitTest`

Took 9 min 55 sec . View more details here
Description: mage build unitTest

`x-pack/metricbeat-windows-2022-windows-2022 - mage build unitTest`

Took 2 min 13 sec . View more details here
Description: mage build unitTest

`x-pack/metricbeat-windows-2022-windows-2022 - mage build unitTest`

Took 2 min 13 sec . View more details here
Description: mage build unitTest

`x-pack/metricbeat-windows-2016-windows-2016 - mage build unitTest`

Took 9 min 15 sec . View more details here
Description: mage build unitTest

`x-pack/metricbeat-windows-2016-windows-2016 - mage build unitTest`

Took 2 min 15 sec . View more details here
Description: mage build unitTest

`x-pack/metricbeat-windows-2016-windows-2016 - mage build unitTest`

Took 2 min 14 sec . View more details here
Description: mage build unitTest

`x-pack/packetbeat-windows-2022-system-windows-2022 - mage systemTest`

Took 1 min 29 sec . View more details here
Description: mage systemTest

`x-pack/packetbeat-windows-2022-system-windows-2022 - mage systemTest`

Took 0 min 11 sec . View more details here
Description: mage systemTest

`x-pack/packetbeat-windows-2022-system-windows-2022 - mage systemTest`

Took 0 min 11 sec . View more details here
Description: mage systemTest

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'hudson.AbortException: script returned exit code 1'

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

cmacknz reviewed

View reviewed changes

filebeat/beater/filebeat.go

- //All module configs need to be loaded to enable all the filesets
- //contained in the modules. The default glob just loads the enabled
- //ones. Switching the glob pattern from *.yml to * achieves this.
+ // All module configs need to be loaded to enable all the filesets

Member

cmacknz Dec 29, 2022 •

edited

Loading

FYI we explicitly disable modules (and filesets) when running under agent: https:/elastic/elastic-agent/blob/7f7faf365aeb303c6e4bb557f3f4b7f4de4025e7/specs/filebeat.spec.yml#L41

This functionality is replaced with integrations.

cmacknz reviewed

View reviewed changes

filebeat/beater/filebeat.go

 // setupPipelineLoaderCallback sets the callback function for loading pipelines during setup.
 func (fb *Filebeat) setupPipelineLoaderCallback(b *beat.Beat) error {
  if b.Config.Output.Name() != "elasticsearch" {
- logp.Warn(pipelinesWarning)
+ logp.Info(fmt.Sprintf(pipelinesWarning, b.Config.Output.Name()))

Member

cmacknz Dec 29, 2022

Ingest pipelines (that run on Elasticsearch) are handled by integration packages for agent. For standalone Filebeat it can load them itself, but the agent doesn't use the functionality.

Member

cmacknz Dec 29, 2022

https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#pipelines-for-fleet-elastic-agent for how this works under agent.

This entire section should be getting bypassed.

cmacknz reviewed

View reviewed changes

libbeat/beat/beat.go

@@ @@ -84,7 +84,7 @@ type Beat struct { @@
 // BeatConfig struct contains the basic configuration of every beat
 type BeatConfig struct {
  // output/publishing related configurations
- Output config.Namespace `config:"output"`
+ Output config.Namespace `config:"outputs"` // this seems to be the issue. But right now it seems that somehow it works when loading the config, but later on Output is empty again

Member

cmacknz Dec 29, 2022

This was also the source of the problem in elastic/elastic-agent#1860 (comment)

The Beats assume they are reading a complete configuration file at startup to populate that. This is not true for the V2 agent. There is no output configured at startup.

In V1 we actually did read the reference Filebeat configuration at startup, and this worked by coincidence because the default output is Elasticsearch.

Ideally what we do in V2 is correct, don't assume an output until the agent configures it. This avoids problems where the Beat loads something from the reference config that conflicts with what the agent sent it, or problems where the user notices the Beats are reaching out to localhost:9200 at startup when a Logstash output is configured.

The problem is the Beats were written assuming they could read a valid config file at startup, so this is causing some weird problems in code that assumes this is populated when it isn't necessarily.

The original issue for this change was #31901

Member

cmacknz Dec 29, 2022

This is where a blank default configuration is created:

beats/libbeat/cfgfile/cfgfile.go

Line 166 in d8204a2

c = config.NewConfig()

Member

cmacknz Dec 29, 2022 •

edited

Loading

If the problem is that the ingest pipeline setup is being skipped, that should not actually be the cause of this problem. See https:/elastic/beats/pull/34143/files#r1059158293

endorama added the Team:Elastic-Agent-Control-Plane label

botelastic bot removed the needs_team label

Member Author

AndersonQ commented Apr 5, 2023

old, it was fixed on another PR

AndersonQ closed this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment