Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x-pack/filebeat/module/threatintel/misp: add support for secondary object attribute handling #28124

Merged
merged 9 commits into from
Oct 7, 2021
Merged

x-pack/filebeat/module/threatintel/misp: add support for secondary object attribute handling #28124

merged 9 commits into from
Oct 7, 2021

Conversation

efd6
Copy link
Contributor

@efd6 efd6 commented Sep 27, 2021

What does this PR do?

This change adds support for ingestion of secondary Object list Attribute data and more reliably handles cases where the primary Attribute data is missing. It does not currently promote attributes in the Object lists and drops some data from Objects (I have retained the same fields that are kept in the primary Attributes list.

Naming of the Object data is open for bikeshedding.

Why is it important?

This addresses a user issue (#26008), allowing indexing of Object lists in ThreatIntel MISP data.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

Standard testing.

Related issues

Use cases

See related issue.

Screenshots

N/A

Logs

N/A

@efd6 efd6 requested a review from P1llus September 27, 2021 00:49
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 27, 2021
@mergify
Copy link
Contributor

mergify bot commented Sep 27, 2021

This pull request does not have a backport label. Could you fix it @efd6? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Sep 27, 2021
@efd6 efd6 force-pushed the threatintel/misp-enhancement branch from deb2d0f to 7ef42bf Compare September 27, 2021 00:51
@efd6 efd6 mentioned this pull request Sep 27, 2021
Closed
3 tasks
@elasticmachine
Copy link
Collaborator

elasticmachine commented Sep 27, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-10-07T09:55:13.426+0000

  • Duration: 96 min 37 sec

  • Commit: 6033972

Test stats 🧪

Test Results
Failed 0
Passed 14772
Skipped 2318
Total 17090

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

@efd6 efd6 force-pushed the threatintel/misp-enhancement branch from 7ef42bf to 97dbdba Compare September 27, 2021 01:21
@mergify
Copy link
Contributor

mergify bot commented Sep 28, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b threatintel/misp-enhancement upstream/threatintel/misp-enhancement
git merge upstream/master
git push upstream threatintel/misp-enhancement

@efd6 efd6 force-pushed the threatintel/misp-enhancement branch from 3049e73 to 80a8b30 Compare September 29, 2021 00:46
@efd6
Copy link
Contributor Author

efd6 commented Sep 30, 2021

/test

@mergify
Copy link
Contributor

mergify bot commented Sep 30, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b threatintel/misp-enhancement upstream/threatintel/misp-enhancement
git merge upstream/master
git push upstream threatintel/misp-enhancement

@efd6 efd6 force-pushed the threatintel/misp-enhancement branch from f2b013f to 686dfe5 Compare September 30, 2021 21:42
@efd6
Copy link
Contributor Author

efd6 commented Oct 3, 2021

PTAL

@efd6 efd6 force-pushed the threatintel/misp-enhancement branch from 63ff6ab to 32fae9e Compare October 4, 2021 00:56
@efd6
Copy link
Contributor Author

efd6 commented Oct 4, 2021

/test

efd6
efd6 commented Oct 4, 2021
View reviewed changes
filebeat/docs/fields.asciidoc Outdated Show resolved Hide resolved
P1llus
P1llus approved these changes Oct 5, 2021
View reviewed changes
Copy link
Member

@P1llus P1llus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to merge once you have done your last changes and CI passes.

@P1llus P1llus added backport-v7.16.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Oct 5, 2021
@efd6 efd6 merged commit 22d1742 into elastic:master Oct 7, 2021
@efd6 efd6 deleted the threatintel/misp-enhancement branch October 7, 2021 11:42
mergify bot pushed a commit that referenced this pull request Oct 7, 2021
@efd6
x-pack/filebeat/module/threatintel/misp: add support for secondary ob…
3079fea 
…ject attribute handling (#28124)

Co-authored-by: Marius Iversen <[email protected]>
(cherry picked from commit 22d1742)

# Conflicts:
#	x-pack/filebeat/module/threatintel/misp/test/misp_sample.ndjson.log-expected.json
@mergify mergify bot mentioned this pull request Oct 7, 2021
Merged
efd6 added a commit that referenced this pull request Oct 11, 2021
@mergify @efd6
[7.x](backport #28124) x-pack/filebeat/module/threatintel/misp: add s…
a3d9bd1 
…upport for secondary object attribute handling (#28305)

* x-pack/filebeat/module/threatintel/misp: add support for secondary object attribute handling (#28124)

Co-authored-by: Marius Iversen <[email protected]>
(cherry picked from commit 22d1742)

# Conflicts:
#	x-pack/filebeat/module/threatintel/misp/test/misp_sample.ndjson.log-expected.json

* Fix merge conflict

Co-authored-by: Dan Kortschak <[email protected]>
Co-authored-by: Dan Kortschak <[email protected]>
v1v added a commit to v1v/beats that referenced this pull request Oct 11, 2021
@v1v
Merge remote-tracking branch 'upstream/master' into feature/e2e-fleet
86c1cf8 
* upstream/master: (73 commits)
  Remove GCP support from Functionbeat (elastic#28253)
  Move labels and annotations under kubernetes.namespace. (elastic#27917)
  Update go release version 1.17.1 (elastic#27543)
  Osquerybeat: Runner and Fetcher unit tests (elastic#28290)
  Osquerybeat: Improve handling of osquery.autoload file, allow customizations (elastic#28289)
  seccomp: allow clone3 syscall for x86 (elastic#28117)
  packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (elastic#28297)
  [7.x] [DOCS] Update api_key example on elasticsearch output (elastic#28288)
  [cloud][docker] use the private docker namespace (elastic#28286)
  Update aws-lambda-go library version to 1.13.3 (elastic#28236)
  Deprecate common.Float (elastic#28280)
  Filebeat: Change compatibility test stage to test against previous minor instead of 7.11 (elastic#28274)
  x-pack/filebeat/module/threatintel/misp: add support for secondary object attribute handling (elastic#28124)
  Explicitly pass http config to doppler consumer (elastic#28277)
  processors/actions/add_fields: Do not panic if event.Fields is nil map (elastic#28219)
  Resolved timestamp for defender atp (elastic#28272)
  [Winlogbeat] Tolerate faults when Windows Event Log session is interrupted (elastic#28191)
  [elastic-agent] proxy requests to subprocesses to their metrics endpoints (elastic#28165)
  Build cloud docker images for elastic-agent (elastic#28134)
  Upgrade k8s go-client library (elastic#28228)
  ...
This was referenced Oct 19, 2021
Closed
Closed
Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021
@efd6 @P1llus
x-pack/filebeat/module/threatintel/misp: add support for secondary ob…
225108a 
…ject attribute handling (elastic#28124)

Co-authored-by: Marius Iversen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@P1llus P1llus P1llus approved these changes

Assignees
No one assigned
Labels
7.16-candidate backport-v7.16.0 Automated backport with mergify enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@efd6 @elasticmachine @P1llus