Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Add resilience when installing required tooling #24542

Merged
merged 2 commits into from
Mar 17, 2021

Conversation

v1v
Copy link
Member

@v1v v1v commented Mar 15, 2021

What does this PR do?

Add more resilience when installing the required tooling with the retry and the context for Windows

Why is it important?

Reduce environmental build failures

For instance

first retry failed with

image

And then

image

@v1v v1v added Team:Automation Label for the Observability productivity team skip-ci Skip the build in the CI but linting labels Mar 15, 2021
@v1v v1v requested a review from a team March 15, 2021 14:55
@v1v v1v self-assigned this Mar 15, 2021
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Mar 15, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Mar 15, 2021

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #24542 updated

  • Start Time: 2021-03-15T17:02:54.744+0000

  • Duration: 133 min 8 sec

  • Commit: 1bc6246

Test stats 🧪

Test Results
Failed 0
Passed 45592
Skipped 5096
Total 50688

Trends 🧪

Image of Build Times

Image of Tests

Steps errors 6

Expand to view the steps failures

Prepare test output
  • Took 0 min 5 sec . View more details on here
  • Description: python .ci/scripts/pre_archive_test.py
Prepare test output
  • Took 0 min 5 sec . View more details on here
  • Description: python .ci/scripts/pre_archive_test.py
Prepare test output
  • Took 0 min 2 sec . View more details on here
  • Description: python .ci/scripts/pre_archive_test.py
Prepare test output
  • Took 0 min 7 sec . View more details on here
  • Description: python .ci/scripts/pre_archive_test.py
x-pack/libbeat-build - mage build test
  • Took 107 min 37 sec . View more details on here
  • Description: mage build test
Error signal
  • Took 0 min 0 sec . View more details on here
  • Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-03-15T19:15:28.362Z]  Go version:        go1.13.15
[2021-03-15T19:15:28.362Z]  Git commit:        48d30b5
[2021-03-15T19:15:28.362Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-03-15T19:15:28.362Z]  OS/Arch:           linux/amd64
[2021-03-15T19:15:28.362Z]  Context:           default
[2021-03-15T19:15:28.362Z]  Experimental:      true
[2021-03-15T19:15:28.362Z] 
[2021-03-15T19:15:28.362Z] Server: Docker Engine - Community
[2021-03-15T19:15:28.362Z]  Engine:
[2021-03-15T19:15:28.362Z]   Version:          20.10.3
[2021-03-15T19:15:28.362Z]   API version:      1.41 (minimum version 1.12)
[2021-03-15T19:15:28.362Z]   Go version:       go1.13.15
[2021-03-15T19:15:28.362Z]   Git commit:       46229ca
[2021-03-15T19:15:28.362Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-03-15T19:15:28.362Z]   OS/Arch:          linux/amd64
[2021-03-15T19:15:28.362Z]   Experimental:     false
[2021-03-15T19:15:28.362Z]  containerd:
[2021-03-15T19:15:28.362Z]   Version:          1.4.4
[2021-03-15T19:15:28.362Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-03-15T19:15:28.362Z]  runc:
[2021-03-15T19:15:28.362Z]   Version:          1.0.0-rc93
[2021-03-15T19:15:28.362Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-03-15T19:15:28.362Z]  docker-init:
[2021-03-15T19:15:28.362Z]   Version:          0.19.0
[2021-03-15T19:15:28.362Z]   GitCommit:        de40ad0
[2021-03-15T19:15:28.362Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-03-15T19:15:28.362Z] Unable to find image 'alpine:3.4' locally
[2021-03-15T19:15:29.079Z] 3.4: Pulling from library/alpine
[2021-03-15T19:15:29.356Z] c1e54eec4b57: Pulling fs layer
[2021-03-15T19:15:29.652Z] c1e54eec4b57: Verifying Checksum
[2021-03-15T19:15:29.652Z] c1e54eec4b57: Download complete
[2021-03-15T19:15:29.652Z] c1e54eec4b57: Pull complete
[2021-03-15T19:15:29.652Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-03-15T19:15:29.652Z] Status: Downloaded newer image for alpine:3.4
[2021-03-15T19:15:31.623Z] Change permissions with write access of all files inside the specific folder
[2021-03-15T19:15:32.902Z] + python .ci/scripts/pre_archive_test.py
[2021-03-15T19:15:32.902Z] Copy ./x-pack/libbeat/build into build/x-pack/libbeat/build
[2021-03-15T19:15:32.919Z] Running in /var/lib/jenkins/workspace/PR-24542-3-a33afe18-b004-4ba8-bc8f-12e506fe784f/src/github.com/elastic/beats/build
[2021-03-15T19:15:33.222Z] + rm -rf ve
[2021-03-15T19:15:33.222Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-03-15T19:15:33.236Z] Recording test results
[2021-03-15T19:15:36.010Z] [Checks API] No suitable checks publisher found.
[2021-03-15T19:15:36.325Z] + tar --version
[2021-03-15T19:15:36.632Z] + tar --exclude=test-build-artifacts-x-pack/libbeat-build.tgz -czf test-build-artifacts-x-pack/libbeat-build.tgz .
[2021-03-15T19:15:38.354Z] + gsutil --version
[2021-03-15T19:15:38.354Z] /var/lib/jenkins/workspace/PR-24542-3-a33afe18-b004-4ba8-bc8f-12e506fe784f/src/github.com/elastic/beats/build@tmp/durable-879686df/script.sh: 1: /var/lib/jenkins/workspace/PR-24542-3-a33afe18-b004-4ba8-bc8f-12e506fe784f/src/github.com/elastic/beats/build@tmp/durable-879686df/script.sh: gsutil: not found
[2021-03-15T19:15:38.397Z] Running in /var/lib/jenkins/workspace/PR-24542-3-a33afe18-b004-4ba8-bc8f-12e506fe784f/src/github.com/elastic/beats/build@tmp
[2021-03-15T19:15:38.702Z] + wget --version
[2021-03-15T19:15:39.014Z] + wget -q -O gsutil.tar.gz https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-319.0.0-linux-x86_64.tar.gz
[2021-03-15T19:15:39.897Z] + tar -xpf gsutil.tar.gz --strip-components=1
[2021-03-15T19:15:42.585Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-03-15T19:15:42.942Z] + gcloud auth activate-service-account --key-file ****
[2021-03-15T19:15:44.317Z] Activated service account credentials for: [[email protected]]
[2021-03-15T19:15:44.885Z] + gsutil -m -q cp -a public-read test-build-artifacts-x-pack/libbeat-build.tgz gs://beats-ci-temp/Beats/beats/PR-24542-3
[2021-03-15T19:15:47.391Z] + python .ci/scripts/search_system_tests.py
[2021-03-15T19:15:47.405Z] [INFO] system-tests=''. If no empty then let's create a tarball
[2021-03-15T19:15:47.750Z] + go clean -modcache
[2021-03-15T19:15:51.453Z] Cleaning up /var/lib/jenkins/workspace/PR-24542-3-a33afe18-b004-4ba8-bc8f-12e506fe784f
[2021-03-15T19:15:51.453Z] Client: Docker Engine - Community
[2021-03-15T19:15:51.453Z]  Version:           20.10.3
[2021-03-15T19:15:51.453Z]  API version:       1.41
[2021-03-15T19:15:51.453Z]  Go version:        go1.13.15
[2021-03-15T19:15:51.453Z]  Git commit:        48d30b5
[2021-03-15T19:15:51.453Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-03-15T19:15:51.454Z]  OS/Arch:           linux/amd64
[2021-03-15T19:15:51.454Z]  Context:           default
[2021-03-15T19:15:51.454Z]  Experimental:      true
[2021-03-15T19:15:51.454Z] 
[2021-03-15T19:15:51.454Z] Server: Docker Engine - Community
[2021-03-15T19:15:51.454Z]  Engine:
[2021-03-15T19:15:51.454Z]   Version:          20.10.3
[2021-03-15T19:15:51.454Z]   API version:      1.41 (minimum version 1.12)
[2021-03-15T19:15:51.454Z]   Go version:       go1.13.15
[2021-03-15T19:15:51.454Z]   Git commit:       46229ca
[2021-03-15T19:15:51.454Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-03-15T19:15:51.454Z]   OS/Arch:          linux/amd64
[2021-03-15T19:15:51.454Z]   Experimental:     false
[2021-03-15T19:15:51.454Z]  containerd:
[2021-03-15T19:15:51.454Z]   Version:          1.4.4
[2021-03-15T19:15:51.454Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-03-15T19:15:51.454Z]  runc:
[2021-03-15T19:15:51.454Z]   Version:          1.0.0-rc93
[2021-03-15T19:15:51.454Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-03-15T19:15:51.454Z]  docker-init:
[2021-03-15T19:15:51.454Z]   Version:          0.19.0
[2021-03-15T19:15:51.454Z]   GitCommit:        de40ad0
[2021-03-15T19:15:51.454Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-03-15T19:15:55.648Z] Change permissions with write access of all files inside the specific folder
[2021-03-15T19:15:55.694Z] Running in /var/lib/jenkins/workspace/PR-24542-3-a33afe18-b004-4ba8-bc8f-12e506fe784f
[2021-03-15T19:16:00.420Z] Failed in branch x-pack/libbeat-build
[2021-03-15T19:16:00.562Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-03-15T19:16:00.626Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-24542/src/github.com/elastic/beats
[2021-03-15T19:16:01.653Z] Running on worker-395930 in /var/lib/jenkins/workspace/Beats_beats_PR-24542
[2021-03-15T19:16:01.696Z] [INFO] getVaultSecret: Getting secrets
[2021-03-15T19:16:01.798Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-03-15T19:16:03.651Z] + chmod 755 generate-build-data.sh
[2021-03-15T19:16:03.651Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24542/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24542/runs/3 FAILURE 7987518
[2021-03-15T19:16:03.651Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24542/runs/3/steps/?limit=10000 -o steps-info.json
[2021-03-15T19:16:11.814Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24542/runs/3/tests/?status=FAILED -o tests-errors.json
[2021-03-15T19:16:14.132Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24542/runs/3/log/ -o pipeline-log.txt

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 45592
Skipped 5096
Total 50688

@v1v v1v removed the skip-ci Skip the build in the CI but linting label Mar 15, 2021
@v1v
Copy link
Member Author

v1v commented Mar 15, 2021

/test

@v1v v1v marked this pull request as ready for review March 17, 2021 09:57
@v1v v1v merged commit 5390d69 into elastic:master Mar 17, 2021
v1v added a commit to v1v/beats that referenced this pull request Mar 17, 2021
v1v added a commit to v1v/beats that referenced this pull request Mar 17, 2021
v1v added a commit to v1v/beats that referenced this pull request Mar 17, 2021
@v1v v1v deleted the feature/environmental-issues-go-windows branch March 17, 2021 12:09
narph pushed a commit to narph/beats that referenced this pull request Mar 17, 2021
narph added a commit that referenced this pull request Mar 18, 2021
* mofidy doc

* work on fix

* update changelog

* update notice

* Update notice

* [Ingest Manager] Sync on rename on windows (#24504)

* Add tests for encoding settings of filestream input (#24426)

* [Elastic Agent] Add the ability to provide custom CA's inside of docker. (#24486)

* Add the ability to provide custom CA's for Elastic Agent docker.

* Add changelog.

* Update Golang to 1.15.9 (#24442)

* Add syntax for multiple selector logging (#24207) (#24497)

* Add syntax for multiple selector logging

* Update libbeat/docs/loggingconfig.asciidoc

Co-authored-by: EamonnTP <[email protected]>

Co-authored-by: EamonnTP <[email protected]>

Co-authored-by: AndyHunt66 <[email protected]>

* chore(ci): use beat_version instead of PR version (#24446)

* Add test for close.reader.after_interval to filestream input (#24423)

* Refactor use of system.hostfs to fix cgroup metrics (#24334)

* refactor use of system.hostfs to fix cgroup metrics

* add changelog

* remove comment

* add cfgwarn

* move changelog

* shift around CLI config location and rep warning

* add comment about system.hostfs usage

* update docs

* capitalization

* fix grammar, add conditional

* change docs phrasing

* [Elastic Agent] Add verification check when updating communication to Kibana. (#24489)

* Add verification check when updating communication to Kibana.

* Add changelog.

* Add const.

* Fix typo in mqtt input docs (#24509)

* Update input-http-endpoint.asciidoc (#24490)

* [Ingest Manager] Move logging defaults to agent (#24535)

[Ingest Manager] Move logging defaults to agent (#24535)

* Clarify that the Tomcat module is for ingesting access logs (#24543)

The Tomcat module is for ingesting access logs, not Catalina or localhost logs.

* [Auditbeat] btmp offset check (#24515)

* auditbeat btmp offset check

Add check that saved offset is not larger than the current file size
to prevent seeking past the end of file

* [Heartbeat] Produce error rather than panic on missing source (#24404)

Fixes #24403.

With the changes to the heartbeat config syntax in 7.12 the `source`
field is now required. Our config validation code didn't actually check
for this field's presence, which caused an NPE.

This PR adds a validation checking for that config's presence. It also
adds tests for the validation code for config sub-fields. There were no
defects found in the validations for source.inline, or source.browser,
but a few tests were missing.

Instead of the panic seen in #24403 users will now get the error seen
below.

```
2021-03-05T15:41:40.146-0600	ERROR	instance/beat.go:952	Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
```

* Fix default scope in add_nomad_metadata (#24559)

Fix default scope in add_nomad_metadata. It is set to local, but
it should be node. Fix also error message that showed that local
is a valid value.

* [Filebeat] Add Dashboards to Threat Intel Module (#24488)

* added dashboards & docs

* ran mage fmt update

* [CI] bump gvm version and use the binary (#24571)

* [CI] Add resilience when installing required tooling (#24542)

* [CI] enable new flaky detector (#24464)

* chore: do not pass beat version (#24586)

We will be delegating the version calculation to the e2e tests, using
target branch values as defaults

Co-authored-by: Jaime Soriano Pastor <[email protected]>
Co-authored-by: Michal Pristas <[email protected]>
Co-authored-by: Noémi Ványi <[email protected]>
Co-authored-by: Blake Rouse <[email protected]>
Co-authored-by: EamonnTP <[email protected]>
Co-authored-by: AndyHunt66 <[email protected]>
Co-authored-by: Manuel de la Peña <[email protected]>
Co-authored-by: Alex K <[email protected]>
Co-authored-by: DeDe Morton <[email protected]>
Co-authored-by: Lee Hinman <[email protected]>
Co-authored-by: Andrew Cholakian <[email protected]>
Co-authored-by: Andrew Pease <[email protected]>
Co-authored-by: Victor Martinez <[email protected]>
narph added a commit to narph/beats that referenced this pull request Mar 18, 2021
* mofidy doc

* work on fix

* update changelog

* update notice

* Update notice

* [Ingest Manager] Sync on rename on windows (elastic#24504)

* Add tests for encoding settings of filestream input (elastic#24426)

* [Elastic Agent] Add the ability to provide custom CA's inside of docker. (elastic#24486)

* Add the ability to provide custom CA's for Elastic Agent docker.

* Add changelog.

* Update Golang to 1.15.9 (elastic#24442)

* Add syntax for multiple selector logging (elastic#24207) (elastic#24497)

* Add syntax for multiple selector logging

* Update libbeat/docs/loggingconfig.asciidoc

Co-authored-by: EamonnTP <[email protected]>

Co-authored-by: EamonnTP <[email protected]>

Co-authored-by: AndyHunt66 <[email protected]>

* chore(ci): use beat_version instead of PR version (elastic#24446)

* Add test for close.reader.after_interval to filestream input (elastic#24423)

* Refactor use of system.hostfs to fix cgroup metrics (elastic#24334)

* refactor use of system.hostfs to fix cgroup metrics

* add changelog

* remove comment

* add cfgwarn

* move changelog

* shift around CLI config location and rep warning

* add comment about system.hostfs usage

* update docs

* capitalization

* fix grammar, add conditional

* change docs phrasing

* [Elastic Agent] Add verification check when updating communication to Kibana. (elastic#24489)

* Add verification check when updating communication to Kibana.

* Add changelog.

* Add const.

* Fix typo in mqtt input docs (elastic#24509)

* Update input-http-endpoint.asciidoc (elastic#24490)

* [Ingest Manager] Move logging defaults to agent (elastic#24535)

[Ingest Manager] Move logging defaults to agent (elastic#24535)

* Clarify that the Tomcat module is for ingesting access logs (elastic#24543)

The Tomcat module is for ingesting access logs, not Catalina or localhost logs.

* [Auditbeat] btmp offset check (elastic#24515)

* auditbeat btmp offset check

Add check that saved offset is not larger than the current file size
to prevent seeking past the end of file

* [Heartbeat] Produce error rather than panic on missing source (elastic#24404)

Fixes elastic#24403.

With the changes to the heartbeat config syntax in 7.12 the `source`
field is now required. Our config validation code didn't actually check
for this field's presence, which caused an NPE.

This PR adds a validation checking for that config's presence. It also
adds tests for the validation code for config sub-fields. There were no
defects found in the validations for source.inline, or source.browser,
but a few tests were missing.

Instead of the panic seen in elastic#24403 users will now get the error seen
below.

```
2021-03-05T15:41:40.146-0600	ERROR	instance/beat.go:952	Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
```

* Fix default scope in add_nomad_metadata (elastic#24559)

Fix default scope in add_nomad_metadata. It is set to local, but
it should be node. Fix also error message that showed that local
is a valid value.

* [Filebeat] Add Dashboards to Threat Intel Module (elastic#24488)

* added dashboards & docs

* ran mage fmt update

* [CI] bump gvm version and use the binary (elastic#24571)

* [CI] Add resilience when installing required tooling (elastic#24542)

* [CI] enable new flaky detector (elastic#24464)

* chore: do not pass beat version (elastic#24586)

We will be delegating the version calculation to the e2e tests, using
target branch values as defaults

Co-authored-by: Jaime Soriano Pastor <[email protected]>
Co-authored-by: Michal Pristas <[email protected]>
Co-authored-by: Noémi Ványi <[email protected]>
Co-authored-by: Blake Rouse <[email protected]>
Co-authored-by: EamonnTP <[email protected]>
Co-authored-by: AndyHunt66 <[email protected]>
Co-authored-by: Manuel de la Peña <[email protected]>
Co-authored-by: Alex K <[email protected]>
Co-authored-by: DeDe Morton <[email protected]>
Co-authored-by: Lee Hinman <[email protected]>
Co-authored-by: Andrew Cholakian <[email protected]>
Co-authored-by: Andrew Pease <[email protected]>
Co-authored-by: Victor Martinez <[email protected]>
(cherry picked from commit 5c6f1b6)
narph added a commit to narph/beats that referenced this pull request Mar 18, 2021
* mofidy doc

* work on fix

* update changelog

* update notice

* Update notice

* [Ingest Manager] Sync on rename on windows (elastic#24504)

* Add tests for encoding settings of filestream input (elastic#24426)

* [Elastic Agent] Add the ability to provide custom CA's inside of docker. (elastic#24486)

* Add the ability to provide custom CA's for Elastic Agent docker.

* Add changelog.

* Update Golang to 1.15.9 (elastic#24442)

* Add syntax for multiple selector logging (elastic#24207) (elastic#24497)

* Add syntax for multiple selector logging

* Update libbeat/docs/loggingconfig.asciidoc

Co-authored-by: EamonnTP <[email protected]>

Co-authored-by: EamonnTP <[email protected]>

Co-authored-by: AndyHunt66 <[email protected]>

* chore(ci): use beat_version instead of PR version (elastic#24446)

* Add test for close.reader.after_interval to filestream input (elastic#24423)

* Refactor use of system.hostfs to fix cgroup metrics (elastic#24334)

* refactor use of system.hostfs to fix cgroup metrics

* add changelog

* remove comment

* add cfgwarn

* move changelog

* shift around CLI config location and rep warning

* add comment about system.hostfs usage

* update docs

* capitalization

* fix grammar, add conditional

* change docs phrasing

* [Elastic Agent] Add verification check when updating communication to Kibana. (elastic#24489)

* Add verification check when updating communication to Kibana.

* Add changelog.

* Add const.

* Fix typo in mqtt input docs (elastic#24509)

* Update input-http-endpoint.asciidoc (elastic#24490)

* [Ingest Manager] Move logging defaults to agent (elastic#24535)

[Ingest Manager] Move logging defaults to agent (elastic#24535)

* Clarify that the Tomcat module is for ingesting access logs (elastic#24543)

The Tomcat module is for ingesting access logs, not Catalina or localhost logs.

* [Auditbeat] btmp offset check (elastic#24515)

* auditbeat btmp offset check

Add check that saved offset is not larger than the current file size
to prevent seeking past the end of file

* [Heartbeat] Produce error rather than panic on missing source (elastic#24404)

Fixes elastic#24403.

With the changes to the heartbeat config syntax in 7.12 the `source`
field is now required. Our config validation code didn't actually check
for this field's presence, which caused an NPE.

This PR adds a validation checking for that config's presence. It also
adds tests for the validation code for config sub-fields. There were no
defects found in the validations for source.inline, or source.browser,
but a few tests were missing.

Instead of the panic seen in elastic#24403 users will now get the error seen
below.

```
2021-03-05T15:41:40.146-0600	ERROR	instance/beat.go:952	Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
```

* Fix default scope in add_nomad_metadata (elastic#24559)

Fix default scope in add_nomad_metadata. It is set to local, but
it should be node. Fix also error message that showed that local
is a valid value.

* [Filebeat] Add Dashboards to Threat Intel Module (elastic#24488)

* added dashboards & docs

* ran mage fmt update

* [CI] bump gvm version and use the binary (elastic#24571)

* [CI] Add resilience when installing required tooling (elastic#24542)

* [CI] enable new flaky detector (elastic#24464)

* chore: do not pass beat version (elastic#24586)

We will be delegating the version calculation to the e2e tests, using
target branch values as defaults

Co-authored-by: Jaime Soriano Pastor <[email protected]>
Co-authored-by: Michal Pristas <[email protected]>
Co-authored-by: Noémi Ványi <[email protected]>
Co-authored-by: Blake Rouse <[email protected]>
Co-authored-by: EamonnTP <[email protected]>
Co-authored-by: AndyHunt66 <[email protected]>
Co-authored-by: Manuel de la Peña <[email protected]>
Co-authored-by: Alex K <[email protected]>
Co-authored-by: DeDe Morton <[email protected]>
Co-authored-by: Lee Hinman <[email protected]>
Co-authored-by: Andrew Cholakian <[email protected]>
Co-authored-by: Andrew Pease <[email protected]>
Co-authored-by: Victor Martinez <[email protected]>
(cherry picked from commit 5c6f1b6)
narph added a commit that referenced this pull request Mar 26, 2021
* mofidy doc

* work on fix

* update changelog

* update notice

* Update notice

* [Ingest Manager] Sync on rename on windows (#24504)

* Add tests for encoding settings of filestream input (#24426)

* [Elastic Agent] Add the ability to provide custom CA's inside of docker. (#24486)

* Add the ability to provide custom CA's for Elastic Agent docker.

* Add changelog.

* Update Golang to 1.15.9 (#24442)

* Add syntax for multiple selector logging (#24207) (#24497)

* Add syntax for multiple selector logging

* Update libbeat/docs/loggingconfig.asciidoc

Co-authored-by: EamonnTP <[email protected]>

Co-authored-by: EamonnTP <[email protected]>

Co-authored-by: AndyHunt66 <[email protected]>

* chore(ci): use beat_version instead of PR version (#24446)

* Add test for close.reader.after_interval to filestream input (#24423)

* Refactor use of system.hostfs to fix cgroup metrics (#24334)

* refactor use of system.hostfs to fix cgroup metrics

* add changelog

* remove comment

* add cfgwarn

* move changelog

* shift around CLI config location and rep warning

* add comment about system.hostfs usage

* update docs

* capitalization

* fix grammar, add conditional

* change docs phrasing

* [Elastic Agent] Add verification check when updating communication to Kibana. (#24489)

* Add verification check when updating communication to Kibana.

* Add changelog.

* Add const.

* Fix typo in mqtt input docs (#24509)

* Update input-http-endpoint.asciidoc (#24490)

* [Ingest Manager] Move logging defaults to agent (#24535)

[Ingest Manager] Move logging defaults to agent (#24535)

* Clarify that the Tomcat module is for ingesting access logs (#24543)

The Tomcat module is for ingesting access logs, not Catalina or localhost logs.

* [Auditbeat] btmp offset check (#24515)

* auditbeat btmp offset check

Add check that saved offset is not larger than the current file size
to prevent seeking past the end of file

* [Heartbeat] Produce error rather than panic on missing source (#24404)

Fixes #24403.

With the changes to the heartbeat config syntax in 7.12 the `source`
field is now required. Our config validation code didn't actually check
for this field's presence, which caused an NPE.

This PR adds a validation checking for that config's presence. It also
adds tests for the validation code for config sub-fields. There were no
defects found in the validations for source.inline, or source.browser,
but a few tests were missing.

Instead of the panic seen in #24403 users will now get the error seen
below.

```
2021-03-05T15:41:40.146-0600	ERROR	instance/beat.go:952	Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
```

* Fix default scope in add_nomad_metadata (#24559)

Fix default scope in add_nomad_metadata. It is set to local, but
it should be node. Fix also error message that showed that local
is a valid value.

* [Filebeat] Add Dashboards to Threat Intel Module (#24488)

* added dashboards & docs

* ran mage fmt update

* [CI] bump gvm version and use the binary (#24571)

* [CI] Add resilience when installing required tooling (#24542)

* [CI] enable new flaky detector (#24464)

* chore: do not pass beat version (#24586)

We will be delegating the version calculation to the e2e tests, using
target branch values as defaults

Co-authored-by: Jaime Soriano Pastor <[email protected]>
Co-authored-by: Michal Pristas <[email protected]>
Co-authored-by: Noémi Ványi <[email protected]>
Co-authored-by: Blake Rouse <[email protected]>
Co-authored-by: EamonnTP <[email protected]>
Co-authored-by: AndyHunt66 <[email protected]>
Co-authored-by: Manuel de la Peña <[email protected]>
Co-authored-by: Alex K <[email protected]>
Co-authored-by: DeDe Morton <[email protected]>
Co-authored-by: Lee Hinman <[email protected]>
Co-authored-by: Andrew Cholakian <[email protected]>
Co-authored-by: Andrew Pease <[email protected]>
Co-authored-by: Victor Martinez <[email protected]>
(cherry picked from commit 5c6f1b6)
narph added a commit that referenced this pull request Mar 26, 2021
* Update gosigar package after fix (#24502)

* mofidy doc

* work on fix

* update changelog

* update notice

* Update notice

* [Ingest Manager] Sync on rename on windows (#24504)

* Add tests for encoding settings of filestream input (#24426)

* [Elastic Agent] Add the ability to provide custom CA's inside of docker. (#24486)

* Add the ability to provide custom CA's for Elastic Agent docker.

* Add changelog.

* Update Golang to 1.15.9 (#24442)

* Add syntax for multiple selector logging (#24207) (#24497)

* Add syntax for multiple selector logging

* Update libbeat/docs/loggingconfig.asciidoc

Co-authored-by: EamonnTP <[email protected]>

Co-authored-by: EamonnTP <[email protected]>

Co-authored-by: AndyHunt66 <[email protected]>

* chore(ci): use beat_version instead of PR version (#24446)

* Add test for close.reader.after_interval to filestream input (#24423)

* Refactor use of system.hostfs to fix cgroup metrics (#24334)

* refactor use of system.hostfs to fix cgroup metrics

* add changelog

* remove comment

* add cfgwarn

* move changelog

* shift around CLI config location and rep warning

* add comment about system.hostfs usage

* update docs

* capitalization

* fix grammar, add conditional

* change docs phrasing

* [Elastic Agent] Add verification check when updating communication to Kibana. (#24489)

* Add verification check when updating communication to Kibana.

* Add changelog.

* Add const.

* Fix typo in mqtt input docs (#24509)

* Update input-http-endpoint.asciidoc (#24490)

* [Ingest Manager] Move logging defaults to agent (#24535)

[Ingest Manager] Move logging defaults to agent (#24535)

* Clarify that the Tomcat module is for ingesting access logs (#24543)

The Tomcat module is for ingesting access logs, not Catalina or localhost logs.

* [Auditbeat] btmp offset check (#24515)

* auditbeat btmp offset check

Add check that saved offset is not larger than the current file size
to prevent seeking past the end of file

* [Heartbeat] Produce error rather than panic on missing source (#24404)

Fixes #24403.

With the changes to the heartbeat config syntax in 7.12 the `source`
field is now required. Our config validation code didn't actually check
for this field's presence, which caused an NPE.

This PR adds a validation checking for that config's presence. It also
adds tests for the validation code for config sub-fields. There were no
defects found in the validations for source.inline, or source.browser,
but a few tests were missing.

Instead of the panic seen in #24403 users will now get the error seen
below.

```
2021-03-05T15:41:40.146-0600	ERROR	instance/beat.go:952	Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
```

* Fix default scope in add_nomad_metadata (#24559)

Fix default scope in add_nomad_metadata. It is set to local, but
it should be node. Fix also error message that showed that local
is a valid value.

* [Filebeat] Add Dashboards to Threat Intel Module (#24488)

* added dashboards & docs

* ran mage fmt update

* [CI] bump gvm version and use the binary (#24571)

* [CI] Add resilience when installing required tooling (#24542)

* [CI] enable new flaky detector (#24464)

* chore: do not pass beat version (#24586)

We will be delegating the version calculation to the e2e tests, using
target branch values as defaults

Co-authored-by: Jaime Soriano Pastor <[email protected]>
Co-authored-by: Michal Pristas <[email protected]>
Co-authored-by: Noémi Ványi <[email protected]>
Co-authored-by: Blake Rouse <[email protected]>
Co-authored-by: EamonnTP <[email protected]>
Co-authored-by: AndyHunt66 <[email protected]>
Co-authored-by: Manuel de la Peña <[email protected]>
Co-authored-by: Alex K <[email protected]>
Co-authored-by: DeDe Morton <[email protected]>
Co-authored-by: Lee Hinman <[email protected]>
Co-authored-by: Andrew Cholakian <[email protected]>
Co-authored-by: Andrew Pease <[email protected]>
Co-authored-by: Victor Martinez <[email protected]>
(cherry picked from commit 5c6f1b6)

* upadte changelog
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…lastic#24627)

* Update gosigar package after fix (elastic#24502)

* mofidy doc

* work on fix

* update changelog

* update notice

* Update notice

* [Ingest Manager] Sync on rename on windows (elastic#24504)

* Add tests for encoding settings of filestream input (elastic#24426)

* [Elastic Agent] Add the ability to provide custom CA's inside of docker. (elastic#24486)

* Add the ability to provide custom CA's for Elastic Agent docker.

* Add changelog.

* Update Golang to 1.15.9 (elastic#24442)

* Add syntax for multiple selector logging (elastic#24207) (elastic#24497)

* Add syntax for multiple selector logging

* Update libbeat/docs/loggingconfig.asciidoc

Co-authored-by: EamonnTP <[email protected]>

Co-authored-by: EamonnTP <[email protected]>

Co-authored-by: AndyHunt66 <[email protected]>

* chore(ci): use beat_version instead of PR version (elastic#24446)

* Add test for close.reader.after_interval to filestream input (elastic#24423)

* Refactor use of system.hostfs to fix cgroup metrics (elastic#24334)

* refactor use of system.hostfs to fix cgroup metrics

* add changelog

* remove comment

* add cfgwarn

* move changelog

* shift around CLI config location and rep warning

* add comment about system.hostfs usage

* update docs

* capitalization

* fix grammar, add conditional

* change docs phrasing

* [Elastic Agent] Add verification check when updating communication to Kibana. (elastic#24489)

* Add verification check when updating communication to Kibana.

* Add changelog.

* Add const.

* Fix typo in mqtt input docs (elastic#24509)

* Update input-http-endpoint.asciidoc (elastic#24490)

* [Ingest Manager] Move logging defaults to agent (elastic#24535)

[Ingest Manager] Move logging defaults to agent (elastic#24535)

* Clarify that the Tomcat module is for ingesting access logs (elastic#24543)

The Tomcat module is for ingesting access logs, not Catalina or localhost logs.

* [Auditbeat] btmp offset check (elastic#24515)

* auditbeat btmp offset check

Add check that saved offset is not larger than the current file size
to prevent seeking past the end of file

* [Heartbeat] Produce error rather than panic on missing source (elastic#24404)

Fixes elastic#24403.

With the changes to the heartbeat config syntax in 7.12 the `source`
field is now required. Our config validation code didn't actually check
for this field's presence, which caused an NPE.

This PR adds a validation checking for that config's presence. It also
adds tests for the validation code for config sub-fields. There were no
defects found in the validations for source.inline, or source.browser,
but a few tests were missing.

Instead of the panic seen in elastic#24403 users will now get the error seen
below.

```
2021-03-05T15:41:40.146-0600	ERROR	instance/beat.go:952	Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
```

* Fix default scope in add_nomad_metadata (elastic#24559)

Fix default scope in add_nomad_metadata. It is set to local, but
it should be node. Fix also error message that showed that local
is a valid value.

* [Filebeat] Add Dashboards to Threat Intel Module (elastic#24488)

* added dashboards & docs

* ran mage fmt update

* [CI] bump gvm version and use the binary (elastic#24571)

* [CI] Add resilience when installing required tooling (elastic#24542)

* [CI] enable new flaky detector (elastic#24464)

* chore: do not pass beat version (elastic#24586)

We will be delegating the version calculation to the e2e tests, using
target branch values as defaults

Co-authored-by: Jaime Soriano Pastor <[email protected]>
Co-authored-by: Michal Pristas <[email protected]>
Co-authored-by: Noémi Ványi <[email protected]>
Co-authored-by: Blake Rouse <[email protected]>
Co-authored-by: EamonnTP <[email protected]>
Co-authored-by: AndyHunt66 <[email protected]>
Co-authored-by: Manuel de la Peña <[email protected]>
Co-authored-by: Alex K <[email protected]>
Co-authored-by: DeDe Morton <[email protected]>
Co-authored-by: Lee Hinman <[email protected]>
Co-authored-by: Andrew Cholakian <[email protected]>
Co-authored-by: Andrew Pease <[email protected]>
Co-authored-by: Victor Martinez <[email protected]>
(cherry picked from commit b5b66f5)

* upadte changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Automation Label for the Observability productivity team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants