Merge tag '1.0.5' into 2021-06-14-upstream-1.0.5-merge

1.0.5 ### Core - `lib/message`: `Report()` can now create a Report instance from Event instances (certtools#1225). - `lib/bot`: * The first word in the log line `Processed ... messages since last logging.` is now adaptible and set to `Forwarded` in the existing filtering bots (certtools#1237). * Kills oneself again after proper shutdown if the bot is XMPP collector or output (certtools#970). Previously these two bots needed two stop commands to get actually stopped. - `lib/utils`: log: set the name of the `py.warnings` logger to the bot name (certtools#1184). ### Bots #### Collectors - `bots.collectors.mail.collector_mail_url`: handle empty downloaded reports (certtools#988). - `bots.collectos.file.collector_file`: handle empty files (certtools#1244). #### Parsers - Shadowserver parser: * SSL FREAK: Remove optional column `device_serial` and add several new ones. * Fixed HTTP URL parsing for multiple feeds (certtools#1243). - Spamhaus CERT parser: * add support for `smtpauth`, `l_spamlink`, `pop`, `imap`, `rdp`, `smb`, `iotscan`, `proxyget`, `iotmicrosoftds`, `automatedtest`, `ioturl`, `iotmirai`, `iotcmd`, `iotlogin` and `iotuser` (certtools#1254). * fix `extra.destination.local_port` -> `extra.source.local_port`. #### Experts - `bots.experts.filter`: Pre-compile regex at bot initialization. ### Tests - Ensure that the bots did process all messages (certtools#291). ### Tools - `intelmqctl`: * `intelmqctl run` has a new parameter `-l` `--loglevel` to overwrite the log level for the run (certtools#1075). * `intelmqctl run [bot-id] mesage send` can now send report messages (certtools#1077). - `intelmqdump`: * has now command completion for bot names, actions and queue names in interacive console. * automatically converts messages from events to reports if the queue the message is being restored to is the source queue of a parser (certtools#1225). * is now capable to read messages in dumps that are dictionaries as opposed to serialized dicts as strings and does not convert them in the show command (certtools#1256). * truncated messages are no longer used/saved to the file after being shown (certtools#1255). * now again denies recovery of dumps if the corresponding bot is running. The check was broken (certtools#1258). * now sorts the dump by the time of the dump. Previously, the list was in random order (certtools#1020). ### Known issues no known issues
chorsley · Jun 14, 2021 · 233d478 · 233d478
2 parents b57223c + b2fa783
commit 233d478
Show file tree

Hide file tree

Showing 83 changed files with 1,972 additions and 1,079 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,102 @@ CHANGELOG
 ==========
 
 
+1.0.5 Bugfix release (2018-06-21)
+---------------------------------
+
+### Core
+- `lib/message`: `Report()` can now create a Report instance from Event instances (#1225).
+- `lib/bot`:
+ * The first word in the log line `Processed ... messages since last logging.` is now adaptible and set to `Forwarded` in the existing filtering bots (#1237).
+ * Kills oneself again after proper shutdown if the bot is XMPP collector or output (#970). Previously these two bots needed two stop commands to get actually stopped.
+- `lib/utils`: log: set the name of the `py.warnings` logger to the bot name (#1184).
+
+### Harmonization
+- Added new types `unauthorized-command` and `unauthorized-login` to `intrusions` taxonomy.
+
+### Bots
+#### Collectors
+- `bots.collectors.mail.collector_mail_url`: handle empty downloaded reports (#988).
+- `bots.collectos.file.collector_file`: handle empty files (#1244).
+
+#### Parsers
+- Shadowserver parser:
+ * SSL FREAK: Remove optional column `device_serial` and add several new ones.
+ * Fixed HTTP URL parsing for multiple feeds (#1243).
+- Spamhaus CERT parser:
+ * add support for `smtpauth`, `l_spamlink`, `pop`, `imap`, `rdp`, `smb`, `iotscan`, `proxyget`, `iotmicrosoftds`, `automatedtest`, `ioturl`, `iotmirai`, `iotcmd`, `iotlogin` and `iotuser` (#1254).
+ * fix `extra.destination.local_port` -> `extra.source.local_port`.
+
+#### Experts
+- `bots.experts.filter`: Pre-compile regex at bot initialization.
+
+### Tests
+- Ensure that the bots did process all messages (#291).
+
+### Tools
+- `intelmqctl`:
+ * `intelmqctl run` has a new parameter `-l` `--loglevel` to overwrite the log level for the run (#1075).
+ * `intelmqctl run [bot-id] mesage send` can now send report messages (#1077).
+- `intelmqdump`:
+ * has now command completion for bot names, actions and queue names in interacive console.
+ * automatically converts messages from events to reports if the queue the message is being restored to is the source queue of a parser (#1225).
+ * is now capable to read messages in dumps that are dictionaries as opposed to serialized dicts as strings and does not convert them in the show command (#1256).
+ * truncated messages are no longer used/saved to the file after being shown (#1255).
+ * now again denies recovery of dumps if the corresponding bot is running. The check was broken (#1258).
+ * now sorts the dump by the time of the dump. Previously, the list was in random order (#1020).
+
+### Known issues
+no known issues
+
+
+1.0.4 Bugfix release (2018-04-20)
+---------------------------------
+- make code style compatible to pycodestyle 2.4.0
+- fixed permissions of some files (they were executable but shouldn't be)
+
+### Core
+- lib/harmonization:
+ * FQDN validation now handles None correctly (raised an Exception).
+ * Fixed several sanitize() methods, the generic sanitation method were called by is_valid, not the sanitize methods (#1219).
+
+### Bots
+* Use the new pypi website at https://pypi.org/ everywhere.
+
+#### Parsers
+- Shadowserver parser:
+ * The fields `url` and `http_url` now handle HTTP URL paths and HTTP requests for all feeds (#1204).
+ * The conversion function `validate_fqdn` now handles empty strings correctly.
+ * Feed 'drone (hadoop)':
+ * Correct validation of field `cc_dns`, will now only be added as `destination.fqdn` if correct FQDN, otherwise ignored. Previously this field could be saved in extra containing an IP address.
+ * Adding more mappings for added columns.
+ * A lot of newly added fields and fixed conversions.
+ * Add newly added columns of `Ssl-Scan` feed to parser
+- Spamhaus CERT parser:
+ * fix parsing and classification for bot names 'openrelay', 'iotrdp', 'sshauth', 'telnetauth', 'iotcmd', 'iotuser', 'wpscanner', 'w_wplogin', 'iotscan'
+ see the NEWS file - Postgresql section - for all changes.
+- CleanMX phishing parser: handle FQDNs in IP column (#1162).
+
+#### Experts
+- `bots.experts.ripencc_abuse_contact`: Add existing parameter `mode` to BOTS file.
+
+### Tools
+- intelmqctl check: Fixed and extended message for 'run_mode' check.
+- `intelmqctl start` botnet. When using `--type json`, no non-json information about wrong bots are output because that would confuse eg. intelmq-manager
+
+### Tests
+- lib/bot: No dumps will be written during tests (#934).
+- lib/test: Expand regular expression on python version to match pre-releases (debian testing).
+
+### Packaging
+* Static data is now included in source tarballs, development files are excluded
+
+### Known issues
+- `bots.collectors/outputs.xmpp` must be killed two times (#970).
+- When running bots with `intelmqctl run [bot-id]` the log level is always INFO (#1075).
+- `intelmqctl run [bot-id] message send [msg]` does only support Events, not Reports (#1077).
+- A warning issued by the python warnings module is logged without the bot-id (#1184).
+
+
 1.0.3 Bugfix release (2018-02-05)
 ---------------------------------
 ### Contrib
@@ -319,7 +415,7 @@ Changes between 0.9 and 1.0.0.dev6
 -`classification.taxonomy` is now lower case only
 
 ### Known issues
- - Harmonization: hashes are not normalized and classified, see also issue #394 and pull #634
+- Harmonization: hashes are not normalized and classified, see also issue #394 and pull #634
 
 ### Contrib
 - ansible and vagrant scripts added
@@ -358,6 +454,6 @@ Changes between 0.9 and 1.0.0.dev6
 2015/06/03 (aaron)
 ------------------
 
- * fixed the license to AGPL in setup.py
- * moved back the documentation from the wiki repo to `docs/`. See #205.
- * added python-zmq as a setup requirement in UserGuide . See #206
+* fixed the license to AGPL in setup.py
+* moved back the documentation from the wiki repo to `docs/`. See #205.
+* added python-zmq as a setup requirement in UserGuide . See #206
diff --git a/MANIFEST.in b/MANIFEST.in
@@ -1,7 +1,13 @@
-exclude .gitignore
-exclude .travis.yml
 graft contrib
 graft docs
+graft intelmq/bots
+graft intelmq/etc
+graft intelmq/tests
 include COPYRIGHT
 include LICENSE
 include CHANGELOG.md
+recursive-exclude intelmq/bin intelmq_gen_feeds_docs.py intelmq_gen_harm_docs.py rewrite_config_files.py
+exclude .*
+global-exclude *~
+global-exclude *.py[co]
+global-exclude __pycache__
diff --git a/NEWS.md b/NEWS.md
@@ -3,6 +3,56 @@ NEWS
 
 See the changelog for a full list of changes.
 
+1.0.5 Bugfix release (2018-06-21)
+---------------------------------
+### Postgres databases
+Use the following statement carefully to upgrade your database.
+Adapt your feedname in the query to the one used in your setup.
+```SQL
+UPDATE events
+ SET "extra" = json_build_object('source.local_port', "extra"->'destination.local_port')
+ WHERE "feed.name" = 'Spamhaus CERT' AND "classification.type" = 'brute-force' AND "classification.identifier" = 'telnet';
+```
+
+1.0.4 Bugfix release (2018-04-20)
+---------------------------------
+
+### Postgres databases
+Use the following statement carefully to upgrade your database.
+Adapt your feedname in the query to the one used in your setup.
+```SQL
+UPDATE events
+ SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'brute-force', "classification.identifier" = 'rdp', "protocol.application" = 'rdp', "malware.name" = NULL
+ WHERE "malware.name" = 'iotrdp' AND "feed.name" = 'Spamhaus CERT';
+UPDATE events
+ SET "classification.taxonomy" = 'vulnerable', "classification.type" = 'vulnerable service', "classification.identifier" = 'openrelay', "protocol.application" = 'smtp', "malware.name" = NULL
+ WHERE "malware.name" = 'openrelay' AND "feed.name" = 'Spamhaus CERT';
+UPDATE events
+ SET "protocol.application" = 'portmapper'
+ WHERE "classification.identifier" = 'openportmapper' AND "feed.name" = 'Open-Portmapper';
+UPDATE events
+ SET "protocol.application" = 'netbios-nameservice'
+ WHERE "classification.identifier" = 'opennetbios' AND "feed.name" = 'Open-NetBIOS-Nameservice';
+UPDATE events
+ SET "protocol.application" = 'ipsec'
+ WHERE "classification.identifier" = 'openike' AND "feed.name" = 'Vulnerable-ISAKMP';
+UPDATE events
+ SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'brute-force', "classification.identifier" = 'ssh', "malware.name" = NULL, "protocol.application" = 'ssh'
+ WHERE "malware.name" = 'sshauth' AND "feed.name" = 'Spamhaus CERT';
+UPDATE events
+ SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'brute-force', "classification.identifier" = 'telnet', "malware.name" = NULL, "protocol.application" = 'ssh'
+ WHERE ("malware.name" = 'telnetauth' OR "malware.name" = 'iotcmd' OR "malware.name" = 'iotuser') AND "feed.name" = 'Spamhaus CERT';
+UPDATE events
+ SET "classification.taxonomy" = 'information gathering', "classification.type" = 'scanner', "classification.identifier" = 'wordpress-vulnerabilities', "malware.name" = NULL, "event_description.text" = 'scanning for wordpress vulnerabilities', "protocol.application" = 'http'
+ WHERE "malware.name" = 'wpscanner' AND "feed.name" = 'Spamhaus CERT';
+UPDATE events
+ SET "classification.taxonomy" = 'information gathering', "classification.type" = 'scanner', "classification.identifier" = 'wordpress-login', "malware.name" = NULL, "event_description.text" = 'scanning for wordpress login pages', "protocol.application" = 'http'
+ WHERE "malware.name" = 'w_wplogin' AND "feed.name" = 'Spamhaus CERT';
+UPDATE events
+ SET "classification.taxonomy" = 'intrusion attempts', "classification.type" = 'scanner', "classification.identifier" = 'scanner-generic', "malware.name" = NULL, "event_description.text" = 'infected IoT device scanning for other vulnerable IoT devices'
+ WHERE "malware.name" = 'iotscan' AND "feed.name" = 'Spamhaus CERT';
+```
+
 1.0.3 Bugfix release (2018-02-05)
 ---------------------------------
 ### Configuration
@@ -12,24 +62,24 @@ See the changelog for a full list of changes.
 | n6 classification | Previous classification | | | Current classification | | | Notes |
 |-|-|-|-|-|-|-|-|
 | | taxonomy | type | identifier | taxonomy | type | identifier |
-| dns-query | Other | other | ignore me | Other | other | dns-query |
-| proxy | Vulnerable | proxy | open proxy | Other | proxy | openproxy |
+| dns-query | other | other | ignore me | other | other | dns-query |
+| proxy | vulnerable | proxy | open proxy | other | proxy | openproxy |
 | sandbox-url | ignore | ignore | ignore me | malicious code | malware | sandboxurl | As this previous taxonomy did not exist, these events have been rejected |
-| other | Vulnerable | unknow | unknown | Other | other | other |
+| other | vulnerable | unknow | unknown | other | other | other |
 
 ### Postgres databases
 Use the following statement carefully to upgrade your database.
 Adapt your feedname in the query to the one used in your setup.
 ```SQL
 UPDATE events
- SET "classification.identifier" = "dns-query"
- WHERE "feed.name" = 'n6' AND "classification.taxonomy" = "Other" AND "classification.type" = "other" AND "classification.identifier" = "ignore me";
+ SET "classification.identifier" = 'dns-query'
+ WHERE "feed.name" = 'n6' AND "classification.taxonomy" = 'other' AND "classification.type" = 'other' AND "classification.identifier" = 'ignore me';
 UPDATE events
- SET "classification.taxonomy" = "malicious code" AND "classification.type" = "malware" AND "classification.identifier" = "sandboxurl"
- WHERE "feed.name" = 'n6' AND "classification.taxonomy" = "Vulnerable" AND "classification.type" = "ignore" AND "classification.identifier" = "ignore me";
+ SET "classification.taxonomy" = 'malicious code' AND "classification.type" = 'malware' AND "classification.identifier" = 'sandboxurl'
+ WHERE "feed.name" = 'n6' AND "classification.taxonomy" = 'vulnerable' AND "classification.type" = 'ignore' AND "classification.identifier" = 'ignore me';
 UPDATE events
- SET "classification.taxonomy" = "Other" AND "classification.type" = "other" AND "classification.identifier" = "other"
- WHERE "feed.name" = 'n6' AND "classification.taxonomy" = "Vulnerable" AND "classification.type" = "unknow" AND "classification.identifier" = "unknow";
+ SET "classification.taxonomy" = 'other' AND "classification.type" = 'other' AND "classification.identifier" = 'other'
+ WHERE "feed.name" = 'n6' AND "classification.taxonomy" = 'vulnerable' AND "classification.type" = 'unknow' AND "classification.identifier" = 'unknow';
 ```
 
 1.0.2 Bugfix release

diff --git a/debian/changelog b/debian/changelog
@@ -1,3 +1,15 @@
+intelmq (1.0.5-1) stable; urgency=medium
+
+ * Update to version 1.0.5
+
+ -- Wagner Sebastian <[email protected]> Thu, 21 Jun 2018 16:53:48 +0200
+
+intelmq (1.0.4-1) stable; urgency=low
+
+ * Update to version 1.0.4
+
+ -- Wagner Sebastian <[email protected]> Fri, 20 Apr 2018 15:25:55 +0200
+
 intelmq (1.0.3-1) stable; urgency=low
 
  * Update to version 1.0.3

diff --git a/docs/Bots.md b/docs/Bots.md
@@ -511,6 +511,10 @@ none
 
 ### IDEA
 
+Converts the event to IDEA format and saves it as JSON in the field `output`. All other fields are not modified.
+
+Documentation about IDEA: https://idea.cesnet.cz/en/index
+
 #### Information:
 * `name:` idea
 * `lookup:` local config
@@ -711,11 +715,11 @@ Sources:
 
 #### Configuration Parameters:
 
+* `mode`: either `append` (default) or `replace`
 * `query_ripe_db_asn`: Query for IPs at `http://rest.db.ripe.net/abuse-contact/%s.json`, default `true`
 * `query_ripe_db_ip`: Query for ASNs at `http://rest.db.ripe.net/abuse-contact/as%s.json`, default `true`
 * `query_ripe_stat_asn`: Query for ASNs at `https://stat.ripe.net/data/abuse-contact-finder/data.json?resource=%s`, default `true`
 * `query_ripe_stat_ip`: Query for IPs at `https://stat.ripe.net/data/abuse-contact-finder/data.json?resource=%s`, default `true`
-* `mode`: either `append` (default) or `replace`
 
 * * *