Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

This Code Provides a possibility to split large Reports into Chunks #680

Merged
9 commits merged into from
May 10, 2017

Commits on Jul 28, 2016

  1. Add support for splitting large collector inputs into chunks

    As a work-around for limitations in the size of strings allowed by
    Redis, we want to be able to split large inputs into several reports.
    This commit adds basic support code for this together with corresponding
    unit tests.
    
    Part of issue certtools#547
    bernhard-herzog committed Jul 28, 2016
    Configuration menu
    Copy the full SHA
    e0257c8 View commit details
    Browse the repository at this point in the history
  2. Optionally split large inputs in file collector bot

    The file collector bot now has two new parameters, chunk_size and
    chunk_replicate_header, controlling whether the input is to be split
    into chunks and how big those chunks should be and what to do with (CSV)
    header lines.
    
    Part of issue certtools#547
    bernhard-herzog committed Jul 28, 2016
    Configuration menu
    Copy the full SHA
    c704859 View commit details
    Browse the repository at this point in the history
  3. Optionally split large inputs in mail URL collector bot

    The mail URL collector bot now has two new parameters, chunk_size and
    chunk_replicate_header, controlling whether the input is to be split
    into chunks and how big those chunks should be and what to do with (CSV)
    header lines.
    
    Part of issue certtools#547
    bernhard-herzog committed Jul 28, 2016
    Configuration menu
    Copy the full SHA
    db7cd21 View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2016

  1. Configuration menu
    Copy the full SHA
    3a47f9b View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2016

  1. Merge branch 'master' of github.com:Intevation/intelmq into dev-split…

    …-csv-reports
    Dustin Demuth committed Sep 12, 2016
    Configuration menu
    Copy the full SHA
    7ce46bb View commit details
    Browse the repository at this point in the history
  2. Merge branch 'master' of https:/certtools/intelmq into de…

    …v-split-csv-reports
    
    Manually resolved two conflicts. Thei fields feed.name and feed.code are now
    set by the CollectorBots __add_report_fields method.
    
    Conflicts:
    	intelmq/bots/collectors/file/collector_file.py
    	intelmq/bots/collectors/mail/collector_mail_url.py
    Dustin Demuth committed Sep 12, 2016
    Configuration menu
    Copy the full SHA
    bf2d473 View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2017

  1. Merge branch 'master' into dev-split-csv-reports

    Fixed merge conflicts in:
    
    Conflicts:
    	intelmq/bots/collectors/file/collector_file.py
    	intelmq/bots/collectors/mail/collector_mail_url.py
    Dustin Demuth committed Jan 24, 2017
    Configuration menu
    Copy the full SHA
    1f08610 View commit details
    Browse the repository at this point in the history
  2. Renmed Readme for File Collector

    Dustin Demuth committed Jan 24, 2017
    Configuration menu
    Copy the full SHA
    135a9a7 View commit details
    Browse the repository at this point in the history
  3. Typos

    Dustin Demuth committed Jan 24, 2017
    Configuration menu
    Copy the full SHA
    fa7677a View commit details
    Browse the repository at this point in the history