Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New binary file format #6

Merged
merged 13 commits into from
Nov 12, 2018
Merged

New binary file format #6

merged 13 commits into from
Nov 12, 2018

Commits on Nov 12, 2018

  1. Create new sorted set score structure (binary) 

    The new format has an 8-byte header (count) followed by the elements data.
Each element is represented by a 4-byte header and a data section.
The 4-byte header contains the size of the data section.
The data section is the raw data (no serialization).

Example of a score file with 2 elements ("hello" and "world"):

    0x0002   0x05 hello     0x05 world
     count   size data      size data
      2       5   "hello"   5    "world"
    @hltbra
    hltbra committed Nov 12, 2018
    Configuration menu
    Copy the full SHA
    5ff46f8 View commit details
    Browse the repository at this point in the history

  2. Change ZCARD performance test to use multiple scores

    @hltbra
    hltbra committed Nov 12, 2018
    Configuration menu
    Copy the full SHA
    10cd0df View commit details
    Browse the repository at this point in the history

  3. Optimize zcount to use new zset file format

    @hltbra
    hltbra committed Nov 12, 2018
    Configuration menu
    Copy the full SHA
    a4ec792 View commit details
    Browse the repository at this point in the history

  4. Move ZSET encoding/decoding to ZSetEncoder class

    @hltbra
    hltbra committed Nov 12, 2018
    Configuration menu
    Copy the full SHA
    d64afd8 View commit details
    Browse the repository at this point in the history

  5. Fix performance regression after aa12d3 

    The extra `seek()` calls were a performance regression
    @hltbra
    hltbra committed Nov 12, 2018
    Configuration menu
    Copy the full SHA
    1ff832b View commit details
    Browse the repository at this point in the history

  6. Remove unecessary JSON serialization 

    After the binary format for zsets, the JSON serialization
became useless
    @hltbra
    hltbra committed Nov 12, 2018
    Configuration menu
    Copy the full SHA
    c942843 View commit details
    Browse the repository at this point in the history

  7. Always read files in binary mode 

    This is for consistency and to help
in the future Python 3 migration
    @hltbra
    hltbra committed Nov 12, 2018
    Configuration menu
    Copy the full SHA
    4563551 View commit details
    Browse the repository at this point in the history

  8. Avoid unnecessary seek on rewrite_content()

    @hltbra
    hltbra committed Nov 12, 2018
    Configuration menu
    Copy the full SHA
    ba07af7 View commit details
    Browse the repository at this point in the history

  9. Extract method skip_header()

    @hltbra
    hltbra committed Nov 12, 2018
    Configuration menu
    Copy the full SHA
    93e0b0d View commit details
    Browse the repository at this point in the history

  10. Simplify read_element by relying on file header 

    Try to read the number of elements found in the file header
instead of checking if 0 bytes were returned when reading
    @hltbra
    hltbra committed Nov 12, 2018
    Configuration menu
    Copy the full SHA
    bee05af View commit details
    Browse the repository at this point in the history

  11. Convert class methods into instance methods 

    All these methods now share the file object (instance attribute)
    @hltbra
    hltbra committed Nov 12, 2018
    Configuration menu
    Copy the full SHA
    a9ccd48 View commit details
    Browse the repository at this point in the history

  12. Update performance tests to have zset improvement results

    @hltbra
    hltbra committed Nov 12, 2018
    Configuration menu
    Copy the full SHA
    b0e6d58 View commit details
    Browse the repository at this point in the history

  13. Add a string test to exercise unicode chars 

    Before the binary file manipulation, special unicode characters were not handled well
    @hltbra
    hltbra committed Nov 12, 2018
    Configuration menu
    Copy the full SHA
    8113842 View commit details
    Browse the repository at this point in the history