shadowserver parser: open mongodb parser broken

[aaronkaplan]

openmongodb-parser: Failed to parse line: '"2015-08-09 11:11:53","1.2.3.4","tcp",27017,"046206184112.atmpu0022.XXXXXXX.net","mongodb","2.4.6",16305,"AT","WIEN","VIENNA",0,0,"nogitversion","Linux allspice 3.2.0-37-generic #58-Ubuntu SMP Thu Jan 24 15:28:10 UTC 2013 x86_64 BOOST_LIB_VERSION=1_53",,"tcmalloc","V8",64,16777216,1,"session | client | ssv | local | presentation"'
Traceback (most recent call last):
  File "/home/aaron/intelmq/intelmq/lib/bot.py", line 461, in process
    events = list(filter(bool, self.parse_line(line, report)))
  File "/home/aaron/intelmq/intelmq/lib/bot.py", line 449, in parse_line
    raise NotImplementedError
NotImplementedError
openmongodb-parser: Dumping message from pipeline to dump file.
openmongodb-parser: Bot has found a problem.
Traceback (most recent call last):
  File "/home/aaron/intelmq/intelmq/lib/bot.py", line 125, in start
    self.process()
  File "/home/aaron/intelmq/intelmq/lib/bot.py", line 469, in process
    self._dump_message(exc, self.recover_line(line))
  File "/home/aaron/intelmq/intelmq/lib/bot.py", line 337, in _dump_message
    json.dump(dump_data, fp, indent=4, sort_keys=True)
  File "/usr/lib/python3.4/json/__init__.py", line 178, in dump
    for chunk in iterable:
  File "/usr/lib/python3.4/json/encoder.py", line 422, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.4/json/encoder.py", line 396, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.4/json/encoder.py", line 396, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.4/json/encoder.py", line 429, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.4/json/encoder.py", line 173, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: NotImplementedError() is not JSON serializable

[aaronkaplan]

New bug in the mongodb parser:

2016-07-25 18:16:52,175 - openmongodb-parser - INFO - Trying to start processing again.
2016-07-25 18:16:52,175 - openmongodb-parser - DEBUG - Waiting for incoming message.
2016-07-25 18:16:52,176 - openmongodb-parser - DEBUG - Received message {'feed.name': 'Shadowserver Open MongoDB', 'rtir_report_id': 582847, 'feed.accuracy': 100.0, 'time.observation': '2016-07-25T16:16:37+00:00'}.
2016-07-25 18:16:52,176 - openmongodb-parser - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/home/aaron/intelmq/intelmq/lib/bot.py", line 125, in start
    self.process()
  File "/home/aaron/intelmq/intelmq/lib/bot.py", line 460, in process
    for line in self.parse(report):
  File "/home/aaron/intelmq/intelmq/bots/parsers/shadowserver/parser.py", line 52, in parse
    raw_report = utils.base64_decode(report["raw"])
KeyError: 'raw'
2016-07-25 18:16:52,176 - openmongodb-parser - INFO - Current Message(event): {'feed.name': 'Shadowserver Open MongoDB', 'rtir_report_id': 582847, 'feed.accuracy': 100.0, 'time.observation': '2016-07-25T16:16:37+00:00'}.
2016-07-25 18:16:52,176 - openmongodb-parser - INFO - Bot will restart in 15 seconds.

[sebix]

Can we ignore reports without raw field?

[aaronkaplan]

Yes Indeed we should :)

[sebix]

Most probably this invalid data has been introduced by this "quick fix" 468f4aa

[aaronkaplan]

On 26 Jul 2016, at 09:08, Sebastian [email protected] wrote:

Most probably this invalid data has been introduced by this "quick fix" 468f4aa

Might be. I'll inspect now.

[aaronkaplan]

On 26 Jul 2016, at 09:08, Sebastian [email protected] wrote:

Most probably this invalid data has been introduced by this "quick fix" 468f4aa

No. that was not the reason.
The problem must come from somewhere else.