Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix negative score order (backward incompatible!) #35

Merged
merged 1 commit into from
Jun 27, 2019

Conversation

hltbra
Copy link
Contributor

@hltbra hltbra commented Jun 26, 2019

Before this commit, sorted set negative scores were stored after
all positive scores because of their binary representation.
For example:

    >>> struct.pack('>d', -1)
    b'\xbf\xf0\x00\x00\x00\x00\x00\x00'
    >>> struct.pack('>d', +1)
    b'?\xf0\x00\x00\x00\x00\x00\x00'
    >>> struct.pack('>d', -1) < struct.pack('>d', +1)
    False

This commit introduces the FloatCodec class to encode/decode floats into bytes and vice-versa, preserving their numerical order.
For example, the following list is encoded to bytes while keeping and preserves its numerical order:

    >>> floats = [1.5, -2.5, 3.2, 2.0]
    >>> sorted(floats, key=FloatCodec().encode)
    [-2.5, 1.5, 2.0, 3.2]

References:

WARNING: This is a backward incompatible change and requires a major release!


The bit flip is still magical. It seems to be a well-known way to encode floats/doubles but I couldn't find a reference implementation from the IEEE.

@hltbra hltbra requested a review from spulec June 26, 2019 16:13
@hltbra hltbra force-pushed the bugfix/reorder-negative-zset-score branch from d64fe50 to 3621858 Compare June 26, 2019 16:14
Before this commit, sorted set negative scores were stored after
all positive scores because of their binary representation.
For example:
    >>> struct.pack('>d', -1)
    b'\xbf\xf0\x00\x00\x00\x00\x00\x00'
    >>> struct.pack('>d', +1)
    b'?\xf0\x00\x00\x00\x00\x00\x00'
    >>> struct.pack('>d', -1) < struct.pack('>d', +1)
    False

This commit introduces the FloatCodec class to encode/decode floats into bytes and vice-versa, preserving their numerical order.
For example, the following list is encoded to bytes while keeping and preserves its numerical order:
    >>> floats = [1.5, -2.5, 3.2, 2.0]
    >>> sorted(floats, key=FloatCodec().encode)
    [-2.5, 1.5, 2.0, 3.2]

References:
* https://en.wikipedia.org/wiki/Floating-point_arithmetic#IEEE_754_design_rationale
* https://stackoverflow.com/a/43305015/565999
* https://stackoverflow.com/a/12933766/565999
* https://ananthakumaran.in/2018/08/17/order-preserving-serialization.html
* https://ananthakumaran.in/2018/08/17/order-preserving-serialization.html#float

WARNING: This is a backward incompatible change and requires a major release!
@hltbra hltbra force-pushed the bugfix/reorder-negative-zset-score branch from 3621858 to 76a0b36 Compare June 26, 2019 16:19
@hltbra hltbra merged commit 2301f0d into master Jun 27, 2019
@hltbra hltbra deleted the bugfix/reorder-negative-zset-score branch June 27, 2019 13:51
hltbra added a commit that referenced this pull request Jun 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants