Sieve Filter expert

intelmq/bots/BOTS

@@ -572,6 +572,13 @@
  "module": "intelmq.bots.experts.taxonomy.expert",
  "parameters": {}
  },
+ "Sievebot": {
+ "description": "Sievebot is the bot responsible to filter and modify intelMQ events.",
+ "module": "intelmq.bots.experts.sieve.expert",
+ "parameters": {
+ "file" :"/opt/intelmq/var/lib/bots/filter.sieve"
+ }
+ },
  "Tor Nodes": {
  "description": "Tor Nodes is the bot responsible to check if an IP is an Tor Exit Node.",
  "module": "intelmq.bots.experts.tor_nodes.expert",

intelmq/bots/experts/sieve/.gitignore

@@ -0,0 +1,2 @@
+*.dot
+

intelmq/bots/experts/sieve/README.md

@@ -0,0 +1,166 @@
+# Sieve Bot
+
+The sieve bot is used to filter and/or modify events based on a set of rules. The
+rules are specified in an external configuration file and with a syntax similar
+to the [Sieve language](http://sieve.info/) used for mail filtering.
+
+Each rule defines a set of matching conditions on received events. Events can be
+matched based on keys and values in the event. If the processed event matches a
+rule's conditions, the corresponding actions are performed. Actions can specify
+whether the event should be kept or dropped in the pipeline (filtering actions)
+or if keys and values should be changed (modification actions).
+
+
+## Examples
+
+The following excerpts illustrate some of the basic features of the sieve file
+format:
+
+```
+if :exists source.fqdn {
+ keep // aborts processing of subsequent rules and forwards the event.
+}
+
+
+if :notexists source.abuse_contact || source.abuse_contact =~ '.*@example.com' {
+ drop // aborts processing of subsequent rules and drops the event.
+}
+
+if source.ip << 192.0.0.0/24 {
+ add! comment = 'bogon'
+}
+
+if classification.type == ['phishing', 'malware'] && source.fqdn =~ '.*\.(ch|li)$' {
+ add! comment = 'domainabuse'
+ keep
+} elsif classification.type == 'scanner' {
+ add! comment = 'ignore'
+ drop
+} else {
+ remove comment
+}
+```
+
+
+## Parameters
+
+The sieve bot only takes one parameter:
+ * `file` - filesystem path the the sieve file
+
+
+## Reference
+
+### Sieve File Structure
+
+The sieve file contains an arbitrary number of rules of the form:
+
+```
+if EXPRESSION {
+ ACTIONS
+} elif EXPRESSION {
+ ACTIONS
+} else {
+ ACTIONS
+}
+```
+
+
+### Expressions
+
+Each rule specifies on or more expressions to match an event based on its keys
+and values. Event keys are specified as strings without quotes. String values
+must be enclosed in single quotes. Numeric values can be specified as integers
+or floats and are unquoted. IP addresses and network ranges (IPv4 and IPv6) are
+specified with quotes. Following operators may be used to match events:
+
+ * `:exists` and `:notexists` match if a given key exists, for example:
+
+ ```if :exists source.fqdn { ... }```
+
+ * `==` and `!=` match for equality of strings and numbers, for example:
+
+ ```if feed.name != 'acme-security' || feed.accuracy == 100 { ... }```
+
+ * `:contains` matches on substrings.
+
+ * `=~` matches strings based on the given regex. `!~` is the inverse regex
+ match.
+
+ * Numerical comparisons are evaluated with `<`, `<=`, `>`, `>=`.
+
+ * `<<` matches if an IP address is contained in the specified network range:
+
+ ```if source.ip << '10.0.0.0/8' { ... }```
+
+ * Values to match against can also be specified as list, in which case any one
+ of the values will result in a match:
+
+ ```if source.ip == ['8.8.8.8', '8.8.4.4'] { ... }```
+
+ In this case, the event will match if it contains a key `source.ip` with
+ either value `8.8.8.8` or `8.8.4.4`.
+
+
+### Actions
+
+If part of a rule matches the given conditions, the actions enclosed in `{` and
+`}` are applied. By default, all events that are matched or not matched by rules
+in the sieve file will be forwarded to the next bot in the pipeline, unless the
+`drop` action is applied.
+
+ * `add` adds a key value pair to the event. This action only applies if the key
+ is not yet defined in the event. If the key is already defined, the action is
+ ignored. Example:
+
+ ```add comment = 'hello, world'```
+
+ * `add!` same as above, but will force overwrite the key in the event.
+
+ * `modify` modifies an existing value for a key. Only applies if the key is
+already defined. If the key is not defined in the event, this action is ignored.
+Example:
+
+ ```modify feed.accuary = 50```
+
+ * `remove` removes a key/value from the event. Action is ignored if the key is
+ not defined in the event. Example:
+
+ ```remove extra.comments```
+
+ * `keep` marks the event to be forwarded to the next bot in the pipeline
+ (same as the default behaviour), but in addition the sieve file processing is
+ interrupted upon reaching this action.
+
+ * `drop` marks the event to be dropped. The event will not be forwarded to the
+ next bot in the pipeline. The sieve file processing is interrupted upon
+ reaching this action. No other actions may be specified besides the `drop`
+ action within `{` and `}`.
+
+
+### Comments
+
+Comments may be used in the sieve file: all characters after `//` and until the end of the line will be ignored.
+
+
+## Validating a sieve file
+
+Use the following command to validate your sieve files:
+```
+$ python intelmq/bots/experts/sieve/validator.py -h
+usage: validator.py [-h] sievefile
+
+Validates the syntax of sievebot files.
+
+positional arguments:
+ sievefile Sieve file
+
+optional arguments:
+ -h, --help show this help message and exit
+```
+
+## Installation
+
+To use this bot, you need to install the required dependencies:
+```
+$ pip install -r REQUIREMENTS.txt
+```

intelmq/bots/experts/sieve/REQUIREMENTS.txt

@@ -0,0 +1 @@
+textX>=1.5.1