ti_misp - Fix duplicate requests for page 1 #6495

andrewkroh · 2023-06-06T15:55:00Z

Avoid requesting page 1 more than once. The page counter within httpjson begins at 0 and the page parameter within MISP starts at 1.

The results were verified using this against the request tracer log from the system test.

jq -c '{"http.request.body.content":.["http.request.body.content"], "transaction.id":.["transaction.id"]}' tracer.ndjson | grep -v null
{"http.request.body.content":"{\"limit\":\"10\",\"page\":\"1\",\"returnFormat\":\"json\",\"timestamp\":\"1686065582\"}","transaction.id":"L1FFIRQ23HJ1E-1"}
{"http.request.body.content":"{\"limit\":\"10\",\"page\":\"2\",\"returnFormat\":\"json\",\"timestamp\":\"1686065582\"}","transaction.id":"L1FFIRQ23HJ1E-2"}
{"http.request.body.content":"{\"limit\":\"10\",\"page\":\"3\",\"returnFormat\":\"json\",\"timestamp\":\"1686065584\"}","transaction.id":"L1FFIRQ23HJ1E-3"}

Relates #6479

What does this PR do?

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

elasticmachine · 2023-06-06T15:55:03Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh · 2023-06-06T15:57:48Z

packages/ti_misp/_dev/deploy/docker/files/config.yml

@@ -9,70 +9,77 @@ rules:
 - status_code: 200
 body: |-
 {
- "response": [{


NOTE: I formatted this JSON body and then split the existing events across page 1 and page 2.

elasticmachine · 2023-06-06T16:21:53Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-06-08T04:08:48.880+0000
Duration: 14 min 22 sec

Test stats 🧪

Test	Results
Failed	0
Passed	15
Skipped	0
Total	15

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

Avoid requesting page 1 more than once. The page counter within httpjson begins at 0 and the page parameter within MISP starts at 1. The results were verified using this against the request tracer log from the system test. jq -c '{"http.request.body.content":.["http.request.body.content"], "transaction.id":.["transaction.id"]}' tracer.ndjson | grep -v null {"http.request.body.content":"{\"limit\":\"10\",\"page\":\"1\",\"returnFormat\":\"json\",\"timestamp\":\"1686065582\"}","transaction.id":"L1FFIRQ23HJ1E-1"} {"http.request.body.content":"{\"limit\":\"10\",\"page\":\"2\",\"returnFormat\":\"json\",\"timestamp\":\"1686065582\"}","transaction.id":"L1FFIRQ23HJ1E-2"} {"http.request.body.content":"{\"limit\":\"10\",\"page\":\"3\",\"returnFormat\":\"json\",\"timestamp\":\"1686065584\"}","transaction.id":"L1FFIRQ23HJ1E-3"} Relates elastic#6479

elasticmachine · 2023-06-06T20:54:25Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`2/2`)	💚 1.258
Classes	100.0% (`2/2`)	💚 1.258
Methods	100.0% (`30/30`)	💚 5.508
Lines	86.344% (`607/703`)	👎 -7.237
Conditionals	100.0% (`0/0`)	💚

efd6 · 2023-06-06T21:58:14Z

packages/ti_misp/_dev/deploy/docker/files/config.yml

 "Galaxy": [],
 "ShadowAttribute": [],
 "category": "Payload delivery",
- "comment": "",
+ "comment": "filename contect for test event 3",


That diff is a bit misleading in that I did not add this string. Viewing without whitespace is better for this file.

But my guess would be "content". 🤷

Changed to "content".

packages/ti_misp/data_stream/threat/_dev/test/system/test-default-config.yml

packages/ti_misp/data_stream/threat/agent/stream/httpjson.yml.hbs

packages/ti_misp/data_stream/threat_attributes/_dev/test/system/test-default-config.yml

packages/ti_misp/data_stream/threat_attributes/agent/stream/httpjson.yml.hbs

elasticmachine · 2023-06-08T05:06:40Z

Package ti_misp - 1.15.2 containing this change is available at https://epr.elastic.co/search?package=ti_misp

Avoid requesting page 1 more than once. The page counter within httpjson begins at 0 and the page parameter within MISP starts at 1. The results were verified using this against the request tracer log from the system test. jq -c '{"http.request.body.content":.["http.request.body.content"], "transaction.id":.["transaction.id"]}' tracer.ndjson | grep -v null {"http.request.body.content":"{\"limit\":\"10\",\"page\":\"1\",\"returnFormat\":\"json\",\"timestamp\":\"1686065582\"}","transaction.id":"L1FFIRQ23HJ1E-1"} {"http.request.body.content":"{\"limit\":\"10\",\"page\":\"2\",\"returnFormat\":\"json\",\"timestamp\":\"1686065582\"}","transaction.id":"L1FFIRQ23HJ1E-2"} {"http.request.body.content":"{\"limit\":\"10\",\"page\":\"3\",\"returnFormat\":\"json\",\"timestamp\":\"1686065584\"}","transaction.id":"L1FFIRQ23HJ1E-3"} Relates #6479

Update the HTTP JSON input configuration for the Threat Intel module's misp fileset with pagination fixes that were done earlier in the Agent-based MISP integration, in these PRs: - Fix timestamp format sent to API elastic/integrations#6482 - Fix duplicate requests for page 1 elastic/integrations#6495 - Keep the same timestamp for later pages elastic/integrations#6649 - Pagination fixes elastic/integrations#9073

Update the HTTP JSON input configuration for the Threat Intel module's misp fileset with pagination fixes that were done earlier in the Agent-based MISP integration, in these PRs: - Fix timestamp format sent to API elastic/integrations#6482 - Fix duplicate requests for page 1 elastic/integrations#6495 - Keep the same timestamp for later pages elastic/integrations#6649 - Pagination fixes elastic/integrations#9073 (cherry picked from commit b7fc69a)

…#37923) [filebeat][threatintel] MISP pagination fixes (#37898) Update the HTTP JSON input configuration for the Threat Intel module's misp fileset with pagination fixes that were done earlier in the Agent-based MISP integration, in these PRs: - Fix timestamp format sent to API elastic/integrations#6482 - Fix duplicate requests for page 1 elastic/integrations#6495 - Keep the same timestamp for later pages elastic/integrations#6649 - Pagination fixes elastic/integrations#9073

…#37924) [filebeat][threatintel] MISP pagination fixes (#37898) Update the HTTP JSON input configuration for the Threat Intel module's misp fileset with pagination fixes that were done earlier in the Agent-based MISP integration, in these PRs: - Fix timestamp format sent to API elastic/integrations#6482 - Fix duplicate requests for page 1 elastic/integrations#6495 - Keep the same timestamp for later pages elastic/integrations#6649 - Pagination fixes elastic/integrations#9073

andrewkroh added bug Something isn't working, use only for issues Team:Security-External Integrations Integration:ti_misp MISP labels Jun 6, 2023

andrewkroh requested a review from a team as a code owner June 6, 2023 15:55

andrewkroh commented Jun 6, 2023

View reviewed changes

andrewkroh added 2 commits June 6, 2023 12:02

stream config - format JSON body

b3d4cfc

stream config - split page 1 into page 1 and 2

8ba3ac2

andrewkroh force-pushed the ti_misp/bugfix/stop-duplicate-page1-request branch from e855605 to 94d908f Compare June 6, 2023 16:10

andrewkroh mentioned this pull request Jun 6, 2023

[ti_misp] timestamp should not change while paginating #6479

Closed

andrewkroh force-pushed the ti_misp/bugfix/stop-duplicate-page1-request branch from 94d908f to 7a3309a Compare June 6, 2023 19:54

andrewkroh force-pushed the ti_misp/bugfix/stop-duplicate-page1-request branch from 7a3309a to e7d0645 Compare June 6, 2023 20:39

efd6 reviewed Jun 7, 2023

View reviewed changes

andrewkroh added 5 commits June 7, 2023 23:54

contect -> content

594010f

test-default-config.yml - add final newline

ba0513f

httpjson.yml.hbs - Add comment about zero-based httpjson page counter

bb64760

elastic-package format

9e26c71

Fix changelog description

7c521b0

andrewkroh requested a review from efd6 June 8, 2023 04:09

efd6 approved these changes Jun 8, 2023

View reviewed changes

andrewkroh merged commit 6b43275 into elastic:main Jun 8, 2023

chrisberkhout mentioned this pull request Feb 7, 2024

[filebeat][threatintel] MISP pagination fixes elastic/beats#37898

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ti_misp - Fix duplicate requests for page 1 #6495

ti_misp - Fix duplicate requests for page 1 #6495

andrewkroh commented Jun 6, 2023

elasticmachine commented Jun 6, 2023

andrewkroh Jun 6, 2023

elasticmachine commented Jun 6, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jun 6, 2023 •

edited

Loading

efd6 Jun 6, 2023

andrewkroh Jun 8, 2023

andrewkroh Jun 8, 2023

elasticmachine commented Jun 8, 2023

ti_misp - Fix duplicate requests for page 1 #6495

ti_misp - Fix duplicate requests for page 1 #6495

Conversation

andrewkroh commented Jun 6, 2023

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Jun 6, 2023

andrewkroh Jun 6, 2023

Choose a reason for hiding this comment

elasticmachine commented Jun 6, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Jun 6, 2023 • edited Loading

🌐 Coverage report

efd6 Jun 6, 2023

Choose a reason for hiding this comment

andrewkroh Jun 8, 2023

Choose a reason for hiding this comment

andrewkroh Jun 8, 2023

Choose a reason for hiding this comment

elasticmachine commented Jun 8, 2023

elasticmachine commented Jun 6, 2023 •

edited

Loading

elasticmachine commented Jun 6, 2023 •

edited

Loading