[ti_misp] Wrong timestamp cause no events fetched from MISP

[kowalczyk-p]

I'm struggling with MISP integration to ElasticAgent, it worked in integration version 1.10.1 but in version 1.27.0 there are no events or attributes I enabled request tracing and found this:

{
  "log.level": "debug",
  "@timestamp": "2023-11-21T14:16:47.068+0100",
  "message": "HTTP request",
  "transaction.id": "5BJTL1H1J2CHE-51",
  "url.original": "https://misp.my.internal.domain.pl/events/restSearch?timestamp=1700572607",
  "url.scheme": "https",
  "url.path": "/events/restSearch",
  "url.domain": "misp.my.internal.domain.pl",
  "url.port": "",
  "url.query": "timestamp=1700572607",
  "http.request.method": "POST",
  "user_agent.original": "Elastic-Filebeat/8.6.2 (linux; amd64; 9b77c2c135c228c2eedc310f6e975bb1a76169b1; 2023-02-12 04:37:19 +0000 UTC)",
  "http.request.body.content": "{\"limit\":\"10\",\"page\":\"1\",\"returnFormat\":\"json\",\"timestamp\":\"1700572607\"}",
  "http.request.body.bytes": 72,
  "http.request.mime_type": "application/json",
  "event.original": "POST /events/restSearch?timestamp=1700572607 HTTP/1.1\r\nHost: misp.soc.abs.assecobs.pl\r\nUser-Agent: Elastic-Filebeat/8.6.2 (linux; amd64; 9b77c2c135c228c2eedc310f6e975bb1a76169b1; 2023-02-12 04:37:19 +0000 UTC)\r\nContent-Length: 72\r\nAccept: application/json\r\nAuthorization: YfbGWjx7blfaRQD9Ptggl7ZgvzNWMYC7MptcG9MQ\r\nContent-Type: application/json\r\nAccept-Encoding: gzip\r\n\r\n{\"limit\":\"10\",\"page\":\"1\",\"returnFormat\":\"json\",\"timestamp\":\"1700572607\"}",
  "ecs.version": "1.6.0"
}

I think the problem is value of timestamp filter: 1700572607 which is "Tue 21 November 2023 13:16:47 UTC" and this is exactly same time as log timestamp "2023-11-21T14:16:47.068+0100".

As per MISP/MISP#6803

When filtering on timestamp, according to the documentation it is supposed to only return any event newer than the defined timestamp. (Using Unix timestamp).

timestamp filter should have value of last api query minus of some look-back time.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[htioekb]

Same finding as me in https://discuss.elastic.co/t/bug-threatintel-misp-plugin-runs-in-endless-loop/344798/2

[ebeahan]

@kowalczyk-p @htioekb for troubleshooting purposes, what version(s) of MISP are you using the integration alongside?

[kowalczyk-p]

@ebeahan I'm using 2.4.173.

[htioekb]

@ebeahan I'm using 2.4.178. The bug reported in MISP/MISP#9359 is fixed in more recent versions only.

[chrisberkhout]

@kowalczyk-p @htioekb Thanks for the reports!

There are several issues. I'll try to fix them here first, in the MISP integration. Then I'll try to apply those and some earlier changes to the Filebeat Threat Intel module's misp fileset.

These reports refer to the data from /events/restSearch, which is referred to as "threat" data. In the integration there is now also a threat_attributes data stream that fetches data from /attributes/restSearch. That is similar and should probably get the same fixes applied to the treat data stream.

@kowalczyk-p's Issue: No events, timestamp is now

Several changes between v1.10.1 and v1.27.0 may affect the observed behavior, especially:

• ti_misp: fix timestamp format sent to API #6482
• [ti_misp] Keep the same timestamp for later pages #6649

Looking at the current logic I can see the following problem, which matches the observed behavior:
With an initial_interval setting of zero hours, the timestamp parameter of the first request will be set to the current time. That will produce no results, so no cursor value will be saved. Since there is no saved value, for the next request the timestamp will again be set to the current time, and again produce no results. This will repeat.

FIX 1: Initialize the timestamp parameter once only. If response data doesn't provide a timestamp value for the next request, the last request's timestamp value should be reused.

@htioekb's Issue: Endless loop

Observation: an endless loop that generates a lot of data, which may have been triggered by an event with a lot of attributes, and which was stopped by removing the split logic.

An exact explanation of this is unclear, but there are several issues that may played a role.

The pagination logic in the Threat Intel module wasn't kept up with fixes in the MISP integration, so that may have contributed to some different behavior.

Sort order is not requested. When requested it was not applied, until fixed in MISP v2.4.179. From what I could see in the code of MISP itself, no default order is applied unless one is explicitly set. Under certain conditions, this would be enough to cause an infinite loop over a section of data.

FIX 2: Set the order parameter in requests. This is mentioned in MISP's main documentation but not in it's OpenAI documentation. Apparently it's "Only available for /events/restSearch", but maybe it can/will work for attributes as well.

Revisiting the same data multiple times should not cause multiple copies of the same document to be indexed, since each event document has the _id field set, and that will deduplicate documents within an index. That is the case for both the integration and the Threat Intel module, but not for the integration's threat_attributes data stream.

FIX 3: Set _id in the threat_attributes data stream. This is for de-duplication purposes, to match what is done in the threat data stream.

MISP's bug Events restSearch API does not filter out events that has same timestamp value as filter (>= timestamp instead of > timestamp) is apparently not fixed yet. If we see the same event multiple times, it should still only appear once in the index, due to the _id de-duplication due to the IOC transform. Redundant events may cause a pagination sequence to have a non-empty first page, but the sequence will terminate when a later page is returned empty.

The observed behavior may have been related to a problem handling the large amount of data generated by an event with so many attributes (27759 attributes and 3863 objects). Disabling the split logic may reduce such a problem. I think we should revisit a potential problem with processing large events if that is observed again after fixing the other issues.

[chrisberkhout]

@kowalczyk-p @htioekb I've fixed several pagination bugs in #9073. That includes the two fixes described above (the third fix isn't necessary since we have another de-duplication mechanism for the threat_attributes data stream), and a couple more that I discovered in the process.

This issue will close when that PR is merged. If there are remaining issues, either handing large data or other cases, let's handle them in a new issue.