Turris greylist has been updated

[ClevenL]

It seems that this morning the Turris graylist has been updated. The CSV file location has been changed from https://project.turris.cz/greylist-data/greylist-latest.csv to https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv.

http_url parameter in bots.collectors.http.collector_http bot should be changed to https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv to receive the correct input for the parser bot.

They also removed IP lookup data from the CSV file. (CSV header: Address,Country,Tags,ASN -> Address,Tags)
So you would also need to modify the bots.parsers.turris.parser file and remove source.geolocation.cc and source.asn from the columns list.

[sebix]

Thanks for the report.

That requires a few updates in IntelMQ. Additionally to what you already mentioned, we also need updated docs and upgrade instructions. And an automated upgrade for changing the URL (lib/upgrades.py).

[ClevenL]

It seems that Turris changed the graylist again by adding a comment in the beginning of the CSV file.
I modified bots.parsers.turris.parser to skip the first two lines(comment and header) with next() function.

I wrote something like this (included ignore_lines list for readability):

ignore_lines = ["comment", "header"]
csvreader = csv.reader(io.StringIO(raw_report))
for _ in ignore_lines: next(csvreader)

[sebix]

Instead of calling next() I proposed to use a csv parser with headers and ignore all lines starting with #. If the tags-column wouldn't contain multiple strings, we could use the generic csv parser.
Also, these two lines (or, if using the csv parser with headers, just the first one), should be added to self.tempdata so that the raw-data is correct (especially relevant if the parser is failing).

And while we are it: The tags actually contain multiple protocol.application fields, so one row results in one or more events.