Cherry-pick #23240 to 7.11: libbeat/template: duplicate entries in fields.yml leads to repeated dynamic templates #23899
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* docs: update docs * chore: update test environment * docs: update doc-branch * chore: make update updates
…d.pct value by removing *100 (elastic#23168) (cherry picked from commit 1e9b07c)
… modules (elastic#23130) (elastic#23199) The experimental Citrix Netscaler and Symantec Endpoint modules are not parsing logs as expected. They were created by converting Netwitness log parser definitions and didn't have test data. Let's remove them since they don't work as expected and hopefully we can provide replacements built on Ingest Node. Closes elastic#23129 (cherry picked from commit 8497b12)
…onfig (elastic#23197) * [docs] Fix typo in elastic agent config (elastic#23185) * fix typo in elastic agent config * update changelog (cherry picked from commit 4cbb168) * fix changelog
* add MADVDONTNEED to builds * delay setting godebug in init scripts (cherry picked from commit 8a68dc9)
…lastic#23174) (cherry picked from commit 9f975c5)
…) (elastic#23207) - only match on /CloudTrail/, /CloudTrail-Digest/ and /CloudTrail-Insight/, ignore variable prefix Closes elastic#23203 (cherry picked from commit 6935dc6)
Co-authored-by: Roman Nagorkin <[email protected]>
…ic#23242) (cherry picked from commit cc8f870) Co-authored-by: Adrian Serrano <[email protected]>
…3215) (elastic#23245) * Add pubsub_alternative_host to gcp pubsub input * Apply suggestions * Add changelog entry * Add new option comment * Make error more descriptive and reorder imports (cherry picked from commit 545598f)
Both inputs gcp-pubsub and httpjson used []byte fields as part of their configurations to receive json blobs. This caused issues because the config values never get parsed properly since literal JSON strings arrived as string and objects as maps, causing errors similar to can not convert 'string' into 'uint8' accessing 'auth.oauth2.google.credentials_json' or can not convert 'object' into 'uint8' accessing 'auth.oauth2.google.credentials_json'. This creates a JSONBlob type that can be unpacked from literal json strings or from config objects into a raw json message. (cherry picked from commit 9022e19) Co-authored-by: Marc Guasch <[email protected]>
* Reorder headers for infoblox module This reorders the syslog headers parsers for the infoblox/nios dataset so that the simpler header is picked up first. Otherwise it will fail to properly parse logs. Fixes elastic#23272 * Changelog entry (cherry picked from commit 27d0f08)
…stic#23317) (cherry picked from commit feb6cbf)
…lastic#23299) (cherry picked from commit 8dbe24a) Co-authored-by: alankis <[email protected]>
For event 4778 (A session was reconnected to a Window Station) the `winlog.event_data.ClientAddress` could be "LOCAL" which is obviosuly not a valid IP so we don't want to copy it into `source.ip` in that case. Fixes elastic#19627 (cherry picked from commit 8c992c5)
…23361 to 7.x: [Filebeat] Fixing test logs for Ciscao ASA logs mysqlenterprise and changing date format (elastic#23375) Co-authored-by: Marius Iversen <[email protected]> (cherry picked from commit d12c906)
…ic#23352) (elastic#23435) * Clean up naming conventions for namespace and dataset settings * Add link to index naming restrictions and add limitation
…3444) (elastic#23446) * Do not take ownership when custom log path is defined in spec. * Add changelog. * Empty commit. (cherry picked from commit 71e406b)
(cherry picked from commit 4c2911f)
…ic#23817) Co-authored-by: Brandon Morelli <[email protected]> Co-authored-by: Felix Barnsteiner <[email protected]> Co-authored-by: DeDe Morton <[email protected]>
…lastic#23825) * Add unix socket to HAProxy stats configuration info * Run make update
) (elastic#23836) * Fix(docs): Undocumented ssl.key_passphrase option Added documentation for an undocumented option: Use of an encrypted client certificate key If you're using an encrypted certificate for X509 authentication in the Kafka module, the kafka module in metricbeat throws a `no PEM blocks` error. Obviously, we need to configure the key's passphrase somewhere, but there's no obvious way to do that. I dug around a little in the libbeat source code to figure out how it's configuring TLS, and sure enough, you can just set the tls.key_passphrase` option. So I've updated the docs to add this parameter, and save others the trouble of figuring this out on their own. Hope this helps. * Update meta files and run make update * Remove extra whitespace * Run make update Co-authored-by: Joost De Cock <[email protected]> Co-authored-by: Joost De Cock <[email protected]>
Co-authored-by: Brandon Morelli <[email protected]> Co-authored-by: Kiju Kim <[email protected]>
…handling (elastic#23807) This fixes several errors processing EVE logs. Fix null dereference of suricata.eve.http.status. Even for http events it can be null. Remove unused field mapping for suricata.eve.flow.end Improve error.message details in pipeline on_failure handlers. Add tags to script processors to improve error messages. Rename suricata.eve.http.http_port to url.port. Add remove processor to on_failure handler to remove any alias field to prevent indexing errors. Ignore errors parsing TLS version. The pipeline got a value of "UNDETERMINED" that caused an error. Ignore errors parsing TLS subject/issuer DNs with kv. If a value contain the value_split string like "Nutanix, Inc." it can fail. (cherry picked from commit 11c5367)
* Update system.asciidoc The default metricsets have more than current documentation and it makes some confusion. Please check the latest `system.yml` configuration and consider to add all default metricset in the documentation. Feel free to change my suggestion. * Update docs under _meta directory to fix build error Co-authored-by: dedemorton <[email protected]> Co-authored-by: Insuk (Chris) Cho <[email protected]>
…ation (elastic#23722) (elastic#23804) Stop input v1 runners created to check config. `CheckConfig` for v1 inputs actually calls the constructors of the inputs. In some cases, as in the log input, the constructor creates resources that are never released unless the runner is stopped. This causes goroutines leaks with autodiscover or other dynamic configurations. (cherry picked from commit e6bb5c9)
…astic#23864) (cherry picked from commit 82968b0)
* docs: Close changelog for 7.11.0 * Revert changes not included in BC * Remove empty sections * Apply suggestions from code review Co-authored-by: Brandon Morelli <[email protected]> * Apply suggestions from code review Co-authored-by: Brandon Morelli <[email protected]> Co-authored-by: Andres Rodriguez <[email protected]> Co-authored-by: Andres Rodriguez <[email protected]> Co-authored-by: Brandon Morelli <[email protected]>
…ynamic templates (elastic#23240) * libbeat/template: deduplicate dynamic templates In case of duplicate fields, do not generate duplicate dynamic_template items. We already deduplicate field mappings because we update a map. * libbeat/template: preserve dynamic template order (cherry picked from commit c0bfea4)
axw
added
[zube]: In Review
backport
Team:Services
|
Pinging @elastic/integrations-services (Team:Services) |
botelastic
bot
added
needs_team
|
Backporting for 7.11.1, to fix the downstream issue in APM Server. |
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
💚 Flaky test reportTests succeeded. Expand to view the summary
Test stats 🧪
|
simitt
approved these changes
Feb 24, 2021
|
This pull request is now in conflicts. Could you fix it? 🙏 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-pick of PR #23240 to 7.11 branch. Original message:
What does this PR do?
In case of duplicate fields, do not generate duplicate dynamic_template items. We already deduplicate field mappings because there we update a map.
There's some light refactoring here to stop using a global list of dynamic templates, and instead maintain a map of dynamic templates on the processor.
Why is it important?
As part of APM Server's migration to Fleet and data streams, we now duplicate fields for each data stream. We still need to generate legacy templates for 7.x, so we gather all of these data stream fields.yml files together to generate docs, templates, etc. The combined fields.yml will therefore have duplicate entries.
Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration files- [ ] I have added an entry inCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.How to test this PR locally
N/A
Related issues
elastic/apm-server#4576