Threat Intel Module | Support for MISP attributes

[jamiehynds]

MISP has a hierarchy of ‘Events’, ‘attributes’ and ‘objects’ and threat intel can be represented in MISP in a number of ways.

Use case 1
An event can be created which contains an attribute or multiple attributes. Filebeat ThreatIntel correctly processes event/attribute relationships for 1 to n attributes and stores the type/values in an index.

Event: 1234

attribute type: MD5
attribute value: e16ddaf4f91df524c28bf4f2e4b1ac09

Use case 2
An event can also be created which contains attributes and objects also containing attributes. Filebeat ThreatIntel correctly processes event/attribute relationships for 1 to n attributes however any event/object/attributes fail to strore in an index. In the example text ‘Malicious executable file” would be included but none of the file:object attributes would be processed.

Event: 4321

attribute type: text
attribute value: Malicious executable file
object: file
attribute type: filename
attribute value: badexe.exe
attribute type: size-in-bytes
attribute value: 46382
attribute type: md5
attribute value: e16ddaf4f91df524b27bf4f2e4b1ac09
attribute type: sha1
attribute value: 988eb4980904d42862f7dc9d4eed52469312ac20
attribute type: sha256
attribute value: 986051385feae5b9850804db2d701c0b029ad24f09bce340c12aee7a5c8a0391

Please see attached sample MISP events:

• 3631,3632 – should behave correctly
• 3634,3635 - fails to import object attribute values but do see records created with ‘error.message’ ‘Illegal list shortcut value [type]
• 3633 only imports the filename ‘thetestfile.txt’
• 3636 only imports the windows-service-name ‘badmojopipe’

misp.sample.events_json.txt

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[LeonBirk]

Have the same issue, would love to see that implemented!

[nicpenning]

This is high priority for us. Are there any workarounds with the Ingest Node pipeline to bring in the MISP events that use the objects?

[P1llus]

Did another dive into this issue today, and there is a few things to note.
We actually store one document in Elastic per attribute, and the initial testing data would usually just have an array of attributes, which fits nicely into how we store TI indicators.

However in these cases, it can have both attributes and object attributes. At first I fixed the pipeline to simply make sure that objects are also included in the document, but then we end up with an array of attributes/indicators in a single document, which won't work when we map it to ECS.

Instead we are working now on some small additions to the httpjson input, so we can easily do multiple splits, resulting in one document per attribute and per object attribute, which should resolve this issue.

[nicpenning]

That's great Marius. Happy to test anything that is available 😊

[P1llus]

That's great Marius. Happy to test anything that is available 😊

Will let you know @nicpenning ! The PR to update httpjson will most likely be linked to this in some way :)

[nicpenning]

Excellent!

[P1llus]

Support has been added, so this can now be closed.