missing info in the harmonization file: hash.md5, hash.sha1, etc.

[aaronkaplan]

There is a problem with assuming the implicit declaration of hash functions by prefixing them with $1$ etc . in the events table / data harmonization config file:

• assume you are given a hash (sha1) of a piece of malware and you want to find it in the events table.
  However, you only stored the md5 since that is what you received even though the sender sent you both fields (sha1 and md4 - such as the n6 feed). Then you can not ever find the right entry again.

Solution: we unfortunately need to extend the harmonization.conf file:
Include

• malware.hash.sha1
• malware.hash.md5
• malware.hash.sha256

Sorry...

[sebix]

The next field where we can possibly have multiple values...

[aaronkaplan]

On 09 Nov 2015, at 22:04, Sebastian [email protected] wrote:

The next field where we can possible have multiple values…

True but multiple values might be OK for JSON (-> array of values) but not so much for a SQL DB.

[mauroasilva]

Yeah... this is something we are always fighting with... the problem is that we are trying to have a multivalue world while supporting singlevalue formats (like relational tables). One could argue that you could always split into multiple events whenever you are converting to a singlevalue reality but if you have multiple fields with multiple values, how do you pair them together?

I think in the future we will have to decide wether we want to support multivalue and disregard a bit of the singlevalue reality or if we want to stick with a singlevalue format.

Right now I think most of our sources are using singlevalue formats like CSV, so it isn't a big deal but this is a decision we will have to make sooner rather than later I think.

[SYNchroACK]

IMHO:

• v1.0 should follow the current format (singlevalue formats - relational tables)
• v1.1 should moved to multivalue format because in the current days is a real requirement. Since the beginning of the project I tried to stick to the singlevalue but with all harmonization issues I think we need to move to the next step (multivalue). For me, is a MUST in v1.1.

[aaronkaplan]

On 11 Nov 2015, at 14:26, Tomás Lima [email protected] wrote:

IMHO:

• v1.0 should follow the current format (singlevalue formats - relational tables)
• v1.1 should moved to multivalue format because in the current days is a real requirement. Since the begging of the project I tried to stick to the singlevalue but with all harmonization issues I think we need to move to the next step (multivalue). For me, is a MUST in v1.1.

TBD… I think this is a major discussion involving multiple projects (not just intelmq)

[ghost]

fixed by #885