Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Fix threatintel.indicator.url.full field not populating #26508

Merged
merged 4 commits into from
Jun 29, 2021

Conversation

legoguy1000
Copy link
Contributor

@legoguy1000 legoguy1000 commented Jun 27, 2021

What does this PR do?

Properly sets the threatintel.indicator.url.full field for URL events for threatintel.abuseurl,misp,anomali,anomalithreatstream,otx

Why is it important?

url.full field wasn't being properly set.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

cd beats/x-pack/filebeat
TESTING_FILEBEAT_MODULES=threatintel mage -v pythonIntegTest

Related issues

Screenshots

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 27, 2021
@legoguy1000 legoguy1000 marked this pull request as ready for review June 27, 2021 16:37
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 27, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: P1llus commented: /test

  • Start Time: 2021-06-29T12:21:36.503+0000

  • Duration: 109 min 21 sec

  • Commit: f02b12a

Test stats 🧪

Test Results
Failed 0
Passed 7400
Skipped 1201
Total 8601

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 7400
Skipped 1201
Total 8601

@P1llus
Copy link
Member

P1llus commented Jun 29, 2021

I think there is a few different ways to approach the same issue, but I would rather we depend more on the fields generated by the uri_parts processor. Instead of removing the remove_if_sucessfull setting, we should copy the value from url.original, so all the threatintel filesets should use uri_parts processor in combination with a set processor to copy_from the threatintel.indicator.url.original field. Do you think we could make that consistent over all the TI filesets?

- uri_parts:
    field: json.url
    target_field: threatintel.indicator.url
    keep_original: true
    remove_if_sucessful: true
    if: 'ctx.json.url != null'
- set:
    field: threatintel.indicator.url.full
    copy_from: threatintel.indicator.url.original
    ignore_empty_value: true

@P1llus
Copy link
Member

P1llus commented Jun 29, 2021

/test

@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 29, 2021
@P1llus P1llus added Filebeat Filebeat needs_integration_sync Changes in this PR need synced to elastic/integrations. and removed needs_integration_sync Changes in this PR need synced to elastic/integrations. labels Jun 29, 2021
@P1llus P1llus merged commit c45aba5 into elastic:master Jun 29, 2021
@P1llus P1llus added the backport-v7.14.0 Automated backport with mergify label Jun 29, 2021
mergify bot pushed a commit that referenced this pull request Jun 29, 2021
…26508)

* #26351: Fix Threat Intel Full URL field

* update changelog

* remove commented items

* updated pipelines per comments

(cherry picked from commit c45aba5)
@legoguy1000 legoguy1000 deleted the 26351-threatintel-url branch June 29, 2021 14:14
P1llus pushed a commit that referenced this pull request Jun 29, 2021
…26508) (#26569)

* #26351: Fix Threat Intel Full URL field

* update changelog

* remove commented items

* updated pipelines per comments

(cherry picked from commit c45aba5)

Co-authored-by: Alex Resnick <[email protected]>
v1v added a commit to v1v/beats that referenced this pull request Jun 29, 2021
…arwin-arm64

* upstream/master: (295 commits)
  Update urllib to 1.26.5. (elastic#26380)
  Update golang.org/x/crypto (elastic#26448)
  [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816)
  Move parsers outside of filestream input so others can use them as well (elastic#26541)
  [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508)
  [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620)
  Logging code cleanup related to Nomad auto-discovery (elastic#26498)
  [Metricbeat] Add Couchbase's Sync Gateway module (elastic#25599)
  Refactor add_cloud_metadata to handle ECS fields easier (elastic#26438)
  [Elastic Agent] Improper casting of int64 (elastic#26520)
  [Elastic Agent] Enable configuring monitoring namespace (elastic#26439)
  [Heartbeat] configure permissions for synthetics config (elastic#26393)
  Osquerybeat: set the raw index name to supress the timestamp suffix (elastic#26545)
  [Heartbeat] add screenshots config to synthetics (elastic#26455)
  [Elastic Agent] Use http2 to connect to Fleet Server. (elastic#26474)
  Remove all docs about  Beats central management (elastic#26399)
  update data.json for gcp billing (elastic#26506)
  Skip x-pack metricbeat tests (elastic#26537)
  [Elastic Agent] Fix issue with FLEET_CA not being used with Fleet Server in container (elastic#26529)
  Add changelog entry for  elastic#26224 (elastic#26531)
  ...
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jun 30, 2021
* master: (25 commits)
  fix: Force PLATFORMS environment variable when we build Elastic Agent dependencies on arm64 (elastic#26415)
  macos for metricbeat to run in the extended meta-stage (elastic#26573)
  Packaging: add arm7 platform in the main pipeline (elastic#26575)
  [Heartbeat] Skip flakey timer queue test (elastic#26592)
  Update to "read_pipeline" permission (elastic#26465) (elastic#26580)
  API keys do not reflect the need for read_pipeline (elastic#26466) (elastic#26582)
  Add Fleet agent.id to Agent monitoring data (elastic#26548)
  Add kinesis metricset (elastic#25989)
  Refactor of system/memory metricset (elastic#26334)
  Introduce httpcommon package in libbeat (add support for Proxy) (elastic#25219)
  [Filebeat] change multiline configuration in awss3 input to parsers (elastic#25873)
  docs: Hint for the error "Error extracting container id" (elastic#25824)
  [Docs] Fixed metricbeat redis exported field CPU descriptions (elastic#25846) (elastic#26496)
  Update urllib to 1.26.5. (elastic#26380)
  Update golang.org/x/crypto (elastic#26448)
  [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816)
  Move parsers outside of filestream input so others can use them as well (elastic#26541)
  [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508)
  [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620)
  Logging code cleanup related to Nomad auto-discovery (elastic#26498)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v7.14.0 Automated backport with mergify Filebeat Filebeat
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants