Parsers for Bambenek Consulting and Netlab 360 OSINT feeds #772

jgedeon120 · 2016-11-12T21:29:41Z

I've created the required parsers for the feeds from Bambenek Consulting; C2 IP Feed, C2 Domain Feed, and DGA Domain Feed. The required test scripts are also included. Please let me know if you have any questions.

codecov-io · 2016-11-14T00:33:18Z

Current coverage is 75.23% (diff: 92.50%)

Merging #772 into master will increase coverage by 4.43%

@@             master       #772   diff @@
==========================================
  Files           206        217    +11   
  Lines          7849       8031   +182   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           5557       6042   +485   
+ Misses         2292       1989   -303   
  Partials          0          0

Powered by Codecov. Last update 2162d51...16c11ad

jgedeon120 · 2016-11-14T00:46:35Z

Parsers have now been added to this pull request for Netlab 360 Magnitude EK and DGA feeds.

http://data.netlab.360.com/

dmth · 2016-11-14T10:45:05Z

Hi thanks for your contribution.

I've a question concerning your parsers: Why are all IPs/Domains written as "destination" and not as "source"?

jgedeon120 · 2016-11-14T11:34:47Z

For these I have written them as the destination due to the relation to
where the connection was headed. If these lists would have been scanners
or something like that then they would have been written as source.

On Mon, Nov 14, 2016 at 5:45 AM, Dustin Demuth [email protected]
wrote:

Hi thanks for your contribution.

I've a question concerning your parsers: Why are all IPs/Domains written
as "destination" and not as "source"?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#772 (comment),
or mute the thread
https:/notifications/unsubscribe-auth/AIHVQsfz3FsXnDlmciIyvcdAfMmZxfKIks5q-DuygaJpZM4KwiF5
.

Registered Linux User # 379282

jgedeon120 · 2016-11-14T14:36:20Z

With putting these into a production state I found that Intelmq was filtering out some of the events. I will close this request and submit a new request once the issues have been worked out so that the proper information is recorded.

sebix · 2016-11-14T14:46:43Z

First, thanks for your contribution. I added some comments inline, most of them apply to all bots of course. Once you think it's ready, I will try it with real data too.

You can reuse this PR, just push your fixes here (you could even overwrite the history).

sebix · 2016-11-14T14:22:38Z

intelmq/bots/parsers/bambenek/parser_c2ipmasterlist.py

+
+ event = Event(report)
+
+ event.add('destination.ip', row_split[0])


Should be source.ip, see https:/certtools/intelmq/blob/master/docs/Data-Harmonization.md#classification-1 (the second table and the text below).

sebix · 2016-11-14T14:24:09Z

intelmq/bots/parsers/bambenek/parser_c2dommasterlist.py

@@ -0,0 +1,42 @@
+# -*- coding: utf-8 -*-
+"""
+http://osint.bambenekconsulting.com/feeds/c2-dommasterlist.txt


Please also document it (with more details) at docs/Feeds.md

sebix · 2016-11-14T14:30:43Z

intelmq/bots/parsers/bambenek/parser_c2dommasterlist.py

+
+class Bambenekc2dommasterlistParserBot(Bot):
+
+ def process(self):


Please use the ParserBot, see https:/certtools/intelmq/blob/master/docs/Developers-Guide.md#parsers
This provides much better error handling.

As the iteration over rows is already implemented, we only need lines 25-35 + a yield in parse_lines.

Example:

intelmq/intelmq/bots/parsers/malwaredomainlist/parser.py

Line 21 in 9996db6

def parse_line(self, row, report):

jgedeon120 · 2016-11-14T18:50:42Z

Sebix,

Thanks again for the tips. The requested changed have been made and the bug that I found this morning corrected.

sebix · 2016-11-15T11:49:16Z

intelmq/bots/parsers/netlab_360/parser_dga.py

+ if FQDN.is_valid(lvalue[1]):
+ event.add('source.fqdn', lvalue[1])
+ else:
+ event.add('source.ip', lvalue[1])


This line is uncovered by the tests.

Changed since this feed only has FQDN's and not IP addresses. It may have been carried over from the dga feed.

sebix · 2016-11-15T11:51:09Z

intelmq/bots/parsers/netlab_360/parser_dga.py

+ event.add('source.ip', lvalue[1])
+
+ event.add('raw', line)
+ event.add('classification.type', 'malware')


Isn't it a c&c?

Yes, yes it is, not sure why I typed malware. Corrected.

sebix · 2016-11-15T11:54:16Z

intelmq/bots/parsers/bambenek/parser_c2dommasterlist.py

+ event.add('time.source', lvalue[2] + " UTC")
+ event.add('event_description.url', lvalue[3])
+ event.add('classification.type', 'c&c')
+ event.add('status', 'online')


Puh, not sure if this field is intended for a status reported by the source. @aaronkaplan can you comment on this?

yes, according to the DHO that's for what it is. However, we have not clearly defined valid values in the DHO for this field ;-) Means: we might have to refactor this later.... But it's ok for now IMHO.

How should we proceed with this now?

How should we proceed with this now?

It's fine.

sebix · 2016-11-15T12:01:58Z

intelmq/bots/parsers/bambenek/parser_dgafeed.py

+ event.add('event_description.text', lvalue[1])
+ event.add('time.source', lvalue[2] + " 00:00 UTC")
+ event.add('event_description.url', lvalue[3])
+ event.add('classification.type', 'ransomware')


Description of ransomware is This IOC refers to a specific type of compromized machine, where the computer has been hijacked for ransom by the criminals. but the description of the feed says "known DGA generated
domains used by malware".

Please have a look @aaronkaplan

this even might change over time ... I have no good solution for this at the moment.
Currently it seems to cover ransomeware DGA domains, if I understand it correctly.
But... I think we should call it "dga domain" since that is what we are actually addressing with this feed. ("source.fqdn")

Should I change the classification.type to "dga domain" and also make the needed changes to harmonization.py to allow dga domain and also map it to Malicious Code in the Taxonomy?

Yes please :)

sebix · 2016-11-15T12:04:25Z

intelmq/bots/parsers/netlab_360/parser_magnitude.py

+
+ event.add('classification.identifier', lvalue[0].lower())
+ event.add('time.source',
+ datetime.utcfromtimestamp(int(lvalue[1])).strftime('%Y-%m-%dT%H:%M:%S+00:00'))


We already have a function in the libs for this: https:/certtools/intelmq/blob/master/intelmq/lib/harmonization.py#L211 :)

I'll look into this more, before making the change to what it is now I kept getting an error in the tests about being out of range or something.

Let me know if the implementation is incomplete or erroneous, I'd like to fix it.

I must have had something wrong when I first tried it. It is working now.

sebix · 2016-11-15T12:10:09Z

docs/Feeds.md

+
+### Magnitude EK Feed
+
+Status: Unknown


AFAIU the upstream description (http://data.netlab.360.com/ek), this feed lists URLs with exploits? Please be more detailed here.

Parsers for Bambenek Consulting and Netlab 360 OSINT feeds Signed-off-by: Sebastian Wagner <[email protected]>

jgedeon120 added 9 commits November 12, 2016 08:43

Added Bambenek c2-dommasterlist parser

3b5a392

Updated Bambenek parsers

5d99003

Added Bambenek test files

8fc4934

Added Bambenek corrected test files

92020df

Added Bambenek corrected test files

18b5dc9

Added Bambenek corrected test files

8b31b7d

Added Bambenek corrected test files

b8e9962

Added Bambenek corrected test files

7189ce5

Added Bambenek corrected test files

a1be3ca

sebix self-assigned this Nov 13, 2016

sebix added component: bots feature Indicates new feature requests or new features labels Nov 13, 2016

sebix added this to the v1.1 Feature release milestone Nov 13, 2016

jgedeon120 added 5 commits November 13, 2016 09:11

Added netlab 360 parsers

3f8667f

Updated Netlab 360 parsers

276f57f

Correction to Netlab 360 test_parser_dga.py

f159cfb

Corrected pep8 issues.

472295b

Corrected pep8 error.

154b9bc

jgedeon120 changed the title ~~Parsers for Bambenek Consulting OSINT feeds~~ Parsers for Bambenek Consulting and Netlab 360 OSINT feeds Nov 14, 2016

jgedeon120 closed this Nov 14, 2016

sebix reviewed Nov 14, 2016

View reviewed changes

jgedeon120 added 3 commits November 14, 2016 13:45

Updated Bambenek parser_c2dommasterlist.py

8027959

Updated Bambenek parser_c2dommasterlist.py

a50eb86

Updated Bambenek parser_c2dommasterlist.py

d0a4bf6

jgedeon120 added 5 commits November 14, 2016 13:45

Updated Bambenek parser_c2dommasterlist.py

c03198a

Updated Netlab 360 parsers

18396f0

Updated Bambenek and Netlab 360 test parsers.

b80e4f7

Updated Feeds.md with Bambenek and Netlab 360 information.

62b14fe

Corrected unittest issue

c6ce6fe

jgedeon120 reopened this Nov 14, 2016

sebix requested changes Nov 15, 2016

View reviewed changes

jgedeon120 added 2 commits November 15, 2016 08:53

Some of the corrections requested

c5a0e2c

Moved Bambenek dga feed to dga domain classification type

16c11ad

sebix approved these changes Nov 21, 2016

View reviewed changes

sebix merged commit 16c11ad into certtools:master Nov 21, 2016

sebix added a commit that referenced this pull request Nov 21, 2016

Merge pull request #772 from jgedeon120/master

f48c638

Parsers for Bambenek Consulting and Netlab 360 OSINT feeds Signed-off-by: Sebastian Wagner <[email protected]>

ghost modified the milestones: v1.1 Feature release, v1.0 Stable Release Jul 5, 2017

ghost unassigned sebix Jul 5, 2017

ghost mentioned this pull request Sep 29, 2020

Re-classification of 'DGA domain' #1613

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parsers for Bambenek Consulting and Netlab 360 OSINT feeds #772

Parsers for Bambenek Consulting and Netlab 360 OSINT feeds #772

jgedeon120 commented Nov 12, 2016

codecov-io commented Nov 14, 2016 •

edited

Loading

jgedeon120 commented Nov 14, 2016

dmth commented Nov 14, 2016

jgedeon120 commented Nov 14, 2016

jgedeon120 commented Nov 14, 2016

sebix commented Nov 14, 2016

sebix Nov 14, 2016

sebix Nov 14, 2016

sebix Nov 14, 2016

jgedeon120 commented Nov 14, 2016

sebix Nov 15, 2016

jgedeon120 Nov 15, 2016

sebix Nov 15, 2016

jgedeon120 Nov 15, 2016

sebix Nov 15, 2016

aaronkaplan Nov 16, 2016

jgedeon120 Nov 21, 2016

sebix Nov 21, 2016 •

edited

Loading

sebix Nov 15, 2016 •

edited

Loading

aaronkaplan Nov 16, 2016

jgedeon120 Nov 21, 2016

sebix Nov 21, 2016

sebix Nov 15, 2016

jgedeon120 Nov 15, 2016

sebix Nov 15, 2016

jgedeon120 Nov 15, 2016

sebix Nov 15, 2016


		event = Event(report)

		event.add('destination.ip', row_split[0])


		class Bambenekc2dommasterlistParserBot(Bot):

		def process(self):

Parsers for Bambenek Consulting and Netlab 360 OSINT feeds #772

Parsers for Bambenek Consulting and Netlab 360 OSINT feeds #772

Conversation

jgedeon120 commented Nov 12, 2016

codecov-io commented Nov 14, 2016 • edited Loading

Current coverage is 75.23% (diff: 92.50%)

jgedeon120 commented Nov 14, 2016

dmth commented Nov 14, 2016

jgedeon120 commented Nov 14, 2016

jgedeon120 commented Nov 14, 2016

sebix commented Nov 14, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgedeon120 commented Nov 14, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sebix Nov 21, 2016 • edited Loading

Choose a reason for hiding this comment

sebix Nov 15, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Nov 14, 2016 •

edited

Loading

sebix Nov 21, 2016 •

edited

Loading

sebix Nov 15, 2016 •

edited

Loading