x-pack/filebeat: AWS test failure

[oakrizan]

Flaky Test

• Test Name: github.com/elastic/beats/v7/x-pack/filebeat/input/cel

• Link: Link to file/line number in github.

• Branch: main

• Artifact Link: If available, attach the generated zip artifact associated with the stack trace for this failure.

• Notes: Additional details about the test. e.g. theory as to failure cause
  In Jenkins AWS tests for x-pack/filebeat were disabled on main/8.*:

• stage cloud was disabled within the [CI] disable aws cloud testing by default #23186
• stage cloudAWS (which was introduced as replacement for cloud stage in CI: enable AWS cloud testing on a PR basis if changes in certain files #29047) was commented out due to flakiness within the Skip flaky Filebeat cloudAWS tests #36558

On 7.17 branch AWS tests were enabled for specific changeset and were successful (eg. #35885).
After Beats migration from Jenkins to Buildkite, AWS tests were temporarily re-enabled for x-pack/filebeat on 8.*/main branches for validation purposes.

AWS test failure context: #36425

• main/8.* branches
  On main/8.* branches previously reported github.com/elastic/beats/v7/x-pack/filebeat/input/cel was still occasionally failing with same error. Once retried the test was successful.
  Though the build failed while was running cometd container for localstack integration tests.

There was a similar issue opened when running test on Windows: #39657. It was fixed by increasing timeout from 5 to 10 (#39713). Apparently it's sometimes not enough when running tests on AWS.

I have created a #40162, where timeout is 20 and tests seems to be successful when executed on AWS: https://buildkite.com/elastic/beats-xpack-filebeat/builds/4118. Basically this problem can be bypassed either by enabling retry for AWS step, either by increasing timeout again.

beats-xpack-filebeat_build_3351_ubuntu-x-pack-slash-filebeat-aws-tests.log
beats-xpack-filebeat_build_3351_ubuntu-x-pack-slash-filebeat-aws-tests-retry.log

• 7.17 branch
  Go Integration tests fail for x-pack/filebeat/input/awss3 which seems to be incorrect/missing bucket policy related issue, since tests fail with Access Denied status code: 403.
  beats-xpack-filebeat_build_3348_ubuntu-x-pack-slash-filebeat-aws-tests.log

• Issue reproducing on CI

• Add aws label to beats-related PR
• Commit changes for x-pack/filebeat & build will be triggered in Buildkite

Tasks

Beta Give feedback

1. Enable AWS tests on main/8.* once fixed
2. Enable AWS tests on 7.17 once fixed

Stack Trace

=== Failed
=== FAIL: x-pack/filebeat/input/cel TestInput/retry_failure (0.01s)
    input_test.go:1602: unexpected result for event 0: got:- want:+
          mapstr.M{
        - 	"error": map[string]any{
        - 		"message": string("failed eval: ERROR: <input>:2:19: failed to unmarshal JSON message: unexpected end of JSON input\n |  get(state.url).as(resp, {\n "...),
        - 	},
        + 	"hello": string("world"),
          }
=== FAIL: x-pack/filebeat/input/cel TestInput (38.59s)

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[rowlandgeoff]

During the migration of beats-ci from Jenkins to Buildkite, a number of tests were failing consistently due to issues unrelated to the migration. Those tests were disabled to stabilize the CI, with the intent to revisit them post-migration. @oakrizan has reviewed them all in her draft PRs linked above in the description, and has opened tickets such as this one to highlight to the product teams the tests that are currently still disabled and could use some attention.

[oakrizan]

I have potentially fixed problem with CURL by updating the version of observability/stream from v0.6.1 to v0.7.0 in testing/environments/docker/cometd/Dockerfile. But now AWS test step fails with AuthorizationError: https://buildkite.com/elastic/beats-xpack-filebeat/builds/5127#01915b0f-c5ba-41ba-9c70-453222f11cf3.
Will look for the solution for that.

In addition removed retry in initCloudEnv.sh since there is no log in case of failure of terraformApply.
Related PR: #40549