Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[auditbeat] Use shared process cache in add_session_metadata processor #40934

Merged
merged 60 commits into from
Oct 16, 2024

Conversation

mjwolf
Copy link
Contributor

@mjwolf mjwolf commented Sep 20, 2024

Proposed commit message

This changes to use a shared process cache in the add_session_metadata processor. This cache is provided by quark and go-quark.

The are currently several process caches in auditbeat. The long term intention is to move all process caches to the shared cache provided by quark. This will reduce resource usage, and improve maintainability by not having multiple implementations of a process cache within Auditbeat.

With this change, the process cache that was previously being used by the ebpf backend is no longer used, and quark will provide process data that's required for enrichment. Rather than needing to track processes from within this processor, quark handles everything, so the processor will now only need to request process data from quark when enrichment happens.

The add_session_metadata process DB code isn't removed, since it's still used by the procfs backend. That backend is intended to be used on systems that aren't supported by the modern backend. Still, quark also supports as far back as CentOS 7, so there will be few systems that will actually use the procfs backend now. The procfs backend could potentially be removed entirely, along with the process DB cache code in the processor, in the future.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

In the add_session_metadata processor config, modern backend replaced ebpf (auto is still the preferred config setting, and does not change). Anyone that has manually set epbf backend will need to change to modern. This processor is in beta, so I think this change is OK.

How to test this PR locally

For users, this change should be transparent, it can be tested in the same way as the existing add_session_metadata processor.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 20, 2024
@mergify mergify bot assigned mjwolf Sep 20, 2024
@mjwolf mjwolf requested a review from AndersonQ October 15, 2024 05:14
@mjwolf mjwolf enabled auto-merge (squash) October 15, 2024 17:27
@mjwolf mjwolf disabled auto-merge October 15, 2024 18:18
@mjwolf mjwolf enabled auto-merge (squash) October 15, 2024 18:46
@mjwolf
Copy link
Contributor Author

mjwolf commented Oct 16, 2024

@andrewkroh Thanks for the review! I've address all the comments now

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not review much of the logic kernel trace provider. I was focused mostly on general Go issues. Some areas to follow-up on:

  • Multi-threaded testing with the Go race detector enabled.
  • Review log verbosity while running the processor.
  • Ensure the backoff/wait logic has full test coverage.

@mjwolf mjwolf merged commit 9992eb5 into elastic:main Oct 16, 2024
143 checks passed
mergify bot pushed a commit that referenced this pull request Oct 16, 2024
…sor (#40934)

This changes to use a shared process cache in the add_session_metadata processor. This cache is provided by quark and go-quark.

The are currently several process caches in auditbeat. The long term intention is to move all process caches to the shared cache provided by quark. This will reduce resource usage, and improve maintainability by not having multiple implementations of a process cache within Auditbeat.

With this change, the process cache that was previously being used by the ebpf backend is no longer used, and quark will provide process data that's required for enrichment. Rather than needing to track processes from within this processor, quark handles everything, so the processor will now only need to request process data from quark when enrichment happens.

The add_session_metadata process DB code isn't removed, since it's still used by the procfs backend. That backend is intended to be used on systems that aren't supported by the modern backend. Still, quark also supports as far back as CentOS 7, so there will be few systems that will actually use the procfs backend now. The procfs backend could potentially be removed entirely, along with the process DB cache code in the processor, in the future.

(cherry picked from commit 9992eb5)
@mjwolf mjwolf deleted the modern-session-view branch October 16, 2024 04:46
pierrehilbert pushed a commit that referenced this pull request Oct 16, 2024
…sor (#40934) (#41250)

This changes to use a shared process cache in the add_session_metadata processor. This cache is provided by quark and go-quark.

The are currently several process caches in auditbeat. The long term intention is to move all process caches to the shared cache provided by quark. This will reduce resource usage, and improve maintainability by not having multiple implementations of a process cache within Auditbeat.

With this change, the process cache that was previously being used by the ebpf backend is no longer used, and quark will provide process data that's required for enrichment. Rather than needing to track processes from within this processor, quark handles everything, so the processor will now only need to request process data from quark when enrichment happens.

The add_session_metadata process DB code isn't removed, since it's still used by the procfs backend. That backend is intended to be used on systems that aren't supported by the modern backend. Still, quark also supports as far back as CentOS 7, so there will be few systems that will actually use the procfs backend now. The procfs backend could potentially be removed entirely, along with the process DB cache code in the processor, in the future.

(cherry picked from commit 9992eb5)

Co-authored-by: Michael Wolf <[email protected]>
belimawr pushed a commit to belimawr/beats that referenced this pull request Oct 18, 2024
…sor (elastic#40934)

This changes to use a shared process cache in the add_session_metadata processor. This cache is provided by quark and go-quark.

The are currently several process caches in auditbeat. The long term intention is to move all process caches to the shared cache provided by quark. This will reduce resource usage, and improve maintainability by not having multiple implementations of a process cache within Auditbeat.

With this change, the process cache that was previously being used by the ebpf backend is no longer used, and quark will provide process data that's required for enrichment. Rather than needing to track processes from within this processor, quark handles everything, so the processor will now only need to request process data from quark when enrichment happens.

The add_session_metadata process DB code isn't removed, since it's still used by the procfs backend. That backend is intended to be used on systems that aren't supported by the modern backend. Still, quark also supports as far back as CentOS 7, so there will be few systems that will actually use the procfs backend now. The procfs backend could potentially be removed entirely, along with the process DB cache code in the processor, in the future.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
8.16-candidate Auditbeat backport-8.x Automated backport to the 8.x branch with mergify enhancement Team:Security-Linux Platform Linux Platform Team in Security Solution
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants