Deprecations 3.0

CHANGELOG.md

@@ -27,6 +27,7 @@ CHANGELOG
 - `intelmq.lib.utils`:
  - New function `list_all_bots` to list all available/installed bots as replacement for the BOTS file (#368, #552, #644, #757, #1069, #1750, PR#1751 by Sebastian Waldbauer).
  - New function `get_bots_settings` to return the effective bot parameters, with global parameters applied.
+ - Removed deprecated function `create_request_session_from_bot` (PR#1997 by Sebastian Wagner, #1404).
 - `intelmq.lib.bot_debugger`:
  - Set bot's `logging_level` directly in `__init__` before the bot's initialization by changing the default value (by Sebastian Wagner).
  - Rewrite `load_configuration_patch` by adapting it to the parameter and configuration rewrite (by Sebastian Wagner).
@@ -73,6 +74,8 @@ Update allowed classification fields to 2020-01-28 version (#1409, #1476).
 - `intelmq.bots.collectors.api.collector_api` (PR#1987 by Mikk Margus Möll, fixes #1986):
  - Added UNIX socket capability.
  - Correctly close the IOLoop in the shutdown method to fix reload.
+- `intelmq.bots.collectors.rt.collector_rt` (PR#1997 by Sebastian Wagner, #1404):
+ - compatibility with the deprecated parameter `unzip_attachment` (removed in 2.1.0) was removed.
 
 #### Parsers
 - Added `intelmq.bots.parsers.fireeye`: A bot that parses hashes and URLs from Fireeye MAS indicators (PR#1745 by Christopher Schappelwein).
@@ -105,6 +108,9 @@ Update allowed classification fields to 2020-01-28 version (#1409, #1476).
  - Split string and numeric matches into single- and multivalued variants, with the relevant new operators `:in`, `:containsany` and `:regexin` for string lists, and `:in` for numeric value lists (PR#1957 by Mikk Margus Möll).
  - Removed the `==` operator for lists, with the previous meaning of `:in`. Have a look at the NEWS.md for more information.
 - Added `intelmq.bots.experts.uwhoisd`: A bot that fetches the whois entry from a uwhois-instance (PR#1918 by Raphaël Vinot).
+- Removed deprecated `intelmq.bots.experts.ripencc_abuse_contact.expert`. It was replaced by `intelmq.bots.experts.ripe.expert` and marked as deprecated in 2.0.0.beta1 (PR#1997 by Sebastian Wagner, #1404).
+- `intelmq.bots.experts.modify.expert`:
+ - Removed compatibility with deprecated configuration format before 1.0.0.dev7 (PR#1997 by Sebastian Wagner, #1404).
 
 #### Outputs
 - Remove `intelmq.bots.outputs.xmpp`: one of the dependencies of the bot was deprecated and according to a short survey on the IntelMQ
@@ -129,12 +135,22 @@ Update allowed classification fields to 2020-01-28 version (#1409, #1476).
  - Compare content of the `output` field as dictionaries, not as string in `assertMessageEqual` (PR#1975 by Karl-Johan Karlsson).
 
 ### Tools
+- intelmqdump (PR#1997 by Sebastian Wagner, #1404):
+ - The command `e` for deleting single entries by given IDs has been merged into the command `d` ("delete"), which can now delete either entries by ID or the whole file.
+ - The command `v` for editing entries has been renamed to `e` ("edit").
 
 ### Contrib
 - eventdb:
  - Added `separate-raws-table.sql` (PR#1985 by Sebastian Wagner).
+- cron-jobs: Removed the deprecated update scripts (PR#1997 by Sebastian Wagner, #1404):
+ - `update-asn-data`
+ - `update-geoip-data`
+ - `update-tor-nodes`
+ - `update-rfiprisk-data`
+ in favor of the built-in update-mechanisms (see the bots' documentation). A crontab file for calling all new update command can be found in `contrib/cron-jobs/intelmq-update-database`.
 
 ### Known issues
+- ParserBot: erroneous raw line recovery in error handling (#1850).
 
 
 2.3.3 (2021-05-31)

NEWS.md

@@ -17,6 +17,18 @@ IntelMQ now uses YAML for the runtime configuration and therefore needs the `rua
 
 ### Tools
 
+#### intelmqdump
+The command `e` for deleting single entries by given IDs has been merged into the command `d` ("delete"), which can now delete either entries by ID or the whole file.
+The command `v` for editing entries has been renamed to `e` ("edit").
+
+#### Cronjobs
+The deprecated shell scripts
+- `update-asn-data`
+- `update-geoip-data`
+- `update-tor-nodes`
+- `update-rfiprisk-data`
+have been removed in favor of the built-in update-mechanisms (see the bots' documentation). A crontab file for calling all new update command can be found in `contrib/cron-jobs/intelmq-update-database`.
+
 ### Bots
 
 Both the XMPP collector bot and the XMPP output bot were removed. This [was evaluated on the mailinglist](https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html)
@@ -85,6 +97,12 @@ The `pipeline.conf` file was removed. The source- and destination-queues of the
 The `intelmqctl upgrade-config` command migrates the existing configuration from the `pipeline.conf` file to the individual bot configurations in the `runtime.conf` configuration file.
 The `runtime.conf` file was replaced by a `runtime.yaml` file. IntelMQ moves the file for you if it does not find a runtime.conf but a runtime.yaml file. When IntelMQ changes the file, it now writes YAML syntax.
 
+#### Removal of deprecated bots and behaviour
+- The bot `intelmq.bots.experts.ripencc_abuse_contact.expert` has been removed. It was replaced by `intelmq.bots.experts.ripe.expert` and marked as deprecated in 2.0.0.beta1.
+- Modify expert: Compatibility with the deprecated configuration format (before 1.0.0.dev7) was removed.
+- RT collector: compatibility with the deprecated parameter `unzip_attachment` (removed in 2.1.0) was removed.
+
+
 ### Libraries
 
 ### Postgres databases

debian/rules

@@ -37,15 +37,6 @@ override_dh_auto_install: $(BOTDOCS)
  sed -i -e '/#!\/usr\/bin\//d' intelmq/bin/*.py
  sed -i -f debian/sedfile intelmq/etc/* docs/user/intelmqctl.rst docs/user/bots.rst setup.py contrib/logrotate/intelmq contrib/logcheck/logcheck.logfiles
  python3 setup.py install --root=debian/intelmq --prefix=/usr
- # these are already in /usr/bin/
- #rm %{buildroot}/%{python3_sitelib}/intelmq/bots/experts/maxmind_geoip/update-geoip-data
- #rm %{buildroot}/%{python3_sitelib}/intelmq/bots/experts/asn_lookup/update-asn-data
- #rm %{buildroot}/%{python3_sitelib}/intelmq/bots/experts/tor_nodes/update-tor-nodes
- # and rename those in /usr/bin
- mv debian/intelmq/usr/bin/update-geoip-data debian/intelmq/usr/bin/intelmq-update-geoip-data
- mv debian/intelmq/usr/bin/update-asn-data debian/intelmq/usr/bin/intelmq-update-asn-data
- mv debian/intelmq/usr/bin/update-tor-nodes debian/intelmq/usr/bin/intelmq-update-tor-nodes
- mv debian/intelmq/usr/bin/update-rfiprisk-data debian/intelmq/usr/bin/intelmq-update-rfiprisk-data
  # create directories
  mkdir -p debian/intelmq/var/log/intelmq
  mkdir -p debian/intelmq/var/lib/intelmq/bots/file-output

docs/user/configuration-management.rst

@@ -582,9 +582,9 @@ When bots are failing due to bad input data or programming errors, they can dump
  > a modify-expert-queue
  All messages in the opened file will be recovered to the stored or given
  queue and removed from the file.
- - e, Delete entries by IDs
- > e id{,id}
- > e 3,5
+ - d, Delete entries by IDs
+ > d id{,id}
+ > d 3,5
  The entries will be deleted from the dump file.
  - d, Delete file
  > d
@@ -594,10 +594,10 @@ When bots are failing due to bad input data or programming errors, they can dump
  > s 0,4,5
  Show the selected IP in a readable format. It's still a raw format from
  repr, but with newlines for message and traceback.
- - v, Edit by ID
- > v id
- > v 0
- > v 1,2
+ - e, Edit by ID
+ > e id
+ > e 0
+ > e 1,2
  Opens an editor (by calling `sensible-editor`) on the message. The modified message is then saved in the dump.
  - q, Quit
  > q
@@ -616,7 +616,7 @@ When bots are failing due to bad input data or programming errors, they can dump
  Processing dragon-research-group-ssh-parser: 2 dumps
  0: 2015-09-03T13:13:22.159014 InvalidValue: invalid value u'NA' (<type 'unicode'>) for key u'source.asn'
  1: 2015-09-01T14:40:20.973743 InvalidValue: invalid value u'NA' (<type 'unicode'>) for key u'source.asn'
- recover (a)ll, delete (e)ntries, (d)elete file, (q)uit, (s)how by ids, (r)ecover by ids? d
+ (r)ecover by ids, recover (a)ll, delete (e)ntries, (d)elete file, (s)how by ids, (q)uit, edit id (v)? d
  Deleted file /opt/intelmq/var/log/dragon-research-group-ssh-parser.dump
 
 Bots and the intelmqdump tool use file locks to prevent writing to already opened files. Bots are trying to lock the file for up to 60 seconds if the dump file is locked already by another process (intelmqdump) and then give up. Intelmqdump does not wait and instead only shows an error message.

intelmq/bin/intelmqdump.py

@@ -51,9 +51,9 @@
  > a modify-expert-queue
  All messages in the opened file will be recovered to the stored or given
  queue and removed from the file.
-- e, Delete entries by IDs
- > e id{,id}
- > e 3,5
+- d, Delete entries by IDs
+ > d id{,id}
+ > d 3,5
  The entries will be deleted from the dump file.
 - d, Delete file
  > d
@@ -63,10 +63,10 @@
  > s 0,4,5
  Show the selected IP in a readable format. It's still a raw format from
  repr, but with newlines for message and traceback.
-- v, Edit by ID
- > v id
- > v 0
- > v 1,2
+- e, Edit by ID
+ > e id
+ > e 0
+ > e 1,2
  Opens an editor (by calling `sensible-editor`) on the message. The modified message is then saved in the dump.
 - q, Quit
  > q
@@ -77,11 +77,10 @@
 # shortcut: description, takes ids, available for corrupted files
 ACTIONS = {'r': ('(r)ecover by ids', True, False),
  'a': ('recover (a)ll', False, False),
- 'e': ('delete (e)ntries', True, False),
- 'd': ('(d)elete file', False, True),
+ 'd': ('(d)elete file or entries by id', True, False),
  's': ('(s)how by ids', True, False),
  'q': ('(q)uit', False, True),
- 'v': ('edit id (v)', True, False),
+ 'e': ('(e)dit by id', True, False),
  }
 AVAILABLE_IDS = [key for key, value in ACTIONS.items() if value[1]]
 
@@ -273,7 +272,7 @@ def main():
  print('Restricted actions.')
  else:
  # don't display list after 'show', 'recover' & edit commands
- if not (answer and isinstance(answer, list) and answer[0] in ['s', 'r', 'v']):
+ if not (answer and isinstance(answer, list) and answer[0] in ['s', 'r', 'e']):
  content = json.load(handle)
  handle.seek(0)
  content = OrderedDict(sorted(content.items(), key=lambda t: t[0])) # sort by key here, #1280
@@ -325,11 +324,6 @@ def main():
  queue_name = answer[1]
  if answer[0] == 'q':
  break
- elif answer[0] == 'e':
- # Delete entries
- for entry in ids:
- del content[meta[entry][0]]
- save_file(handle, content)
  elif answer[0] == 'r':
  # recover entries
  params = defaults.copy()
@@ -376,10 +370,17 @@ def main():
  print('Deleting empty file {}'.format(fname))
  break
  elif answer[0] == 'd':
- # delete dumpfile
- delete_file = True
- print('Deleting empty file {}'.format(fname))
- break
+ # Delete entries or file
+ if ids:
+ # delete entries
+ for entry in ids:
+ del content[meta[entry][0]]
+ save_file(handle, content)
+ else:
+ # delete dumpfile
+ delete_file = True
+ print('Deleting file {}'.format(fname))
+ break
  elif answer[0] == 's':
  # Show entries by id
  for count, (key, orig_value) in enumerate(content.items()):
@@ -401,7 +402,7 @@ def main():
  if type(value['traceback']) is not list:
  value['traceback'] = value['traceback'].splitlines()
  pprint.pprint(value)
- elif answer[0] == 'v':
+ elif answer[0] == 'e':
  # edit given id
  if not ids:
  print(red('Edit mode needs an id'))

intelmq/bots/collectors/rt/collector_rt.py

@@ -71,18 +71,6 @@ def init(self):
  self._parse_extract_file_parameter('extract_attachment')
  self._parse_extract_file_parameter('extract_download')
 
- if hasattr(self, 'unzip_attachment'):
- self.logger.warning("The parameter 'unzip_attachment' is deprecated and "
- "will be removed in version 3.0 in favor of the "
- "more generic and powerful 'extract_attachment'. "
- "Look at the Bots documentation for more details.")
- if not self.extract_attachment:
- self.extract_attachment = self.unzip_attachment
- else:
- self.logger.warn("Both 'extract_attachment' and the deprecated "
- "'unzip_attachment' parameter are in use. Ignoring "
- "the latter one.")
-
  def process(self):
  RT = rt.Rt(self.uri, self.user,
  self.password)

intelmq/bots/experts/modify/expert.py

@@ -11,7 +11,6 @@
 
 from intelmq.lib.bot import Bot
 from intelmq.lib.utils import load_configuration
-from intelmq.lib.upgrades import modify_expert_convert_config
 
 
 def is_re_pattern(value):
@@ -47,11 +46,6 @@ class ModifyExpertBot(Bot):
 
  def init(self):
  config = load_configuration(self.configuration_path)
- if type(config) is dict:
- self.logger.warning('Support for dict-based configuration will be '
- 'removed in version 3.0. Have a look at the '
- 'NEWS file section 1.0.0.dev7.')
- config = modify_expert_convert_config(config)
 
  if self.case_sensitive:
  self.re_kwargs = {}

intelmq/bots/experts/reverse_dns/expert.py

@@ -33,12 +33,6 @@ class ReverseDnsExpertBot(Bot, CacheMixin):
  redis_cache_port: int = 6379
  redis_cache_ttl: int = 86400
 
- def init(self):
- if not hasattr(self, 'overwrite'):
- self.logger.warning("Parameter 'overwrite' is not given, assuming 'True'. "
- "Please set it explicitly, default will change to "
- "'False' in version 3.0.0'.")
-
  def process(self):
  event = self.receive_message()