cache_hash can give the wrong hash code for deserialized objects #482
Comments
|
I think it would be more elegant to clear the cache – would that be feasible? |
|
Good point - I will just clear the cache |
|
bump? :) |
|
Last week was busier than expected, but I hope to complete this tomorrow |
gabbard
pushed a commit
to gabbard/attrs
that referenced
this issue
Jan 22, 2019
Because the hash code cache field gets serialized and deserialized by Pickle, previously when you deserialize a cache_hash=True attrs object, the hashcode will be the hashcode the object had at serialization-time. However, if your object had fields with hash codes which were not deterministic between interpreter runs, then on a new interpreter run your deserialized object would have a hash code which differs from a newly created identical object. This commit fixes that by clearing the cache on deserialization. It needs to override the __setstate__ method to do so, so this commit also forbids using a custom __setstate__ on a cache_hash=True object. Closes python-attrs#482 .
gabbard
pushed a commit
to gabbard/attrs
that referenced
this issue
Jan 22, 2019
Because the hash code cache field gets serialized and deserialized by Pickle, previously when you deserialize a cache_hash=True attrs object, the hashcode will be the hashcode the object had at serialization-time. However, if your object had fields with hash codes which were not deterministic between interpreter runs, then on a new interpreter run your deserialized object would have a hash code which differs from a newly created identical object. This commit fixes that by clearing the cache on deserialization. It needs to override the __setstate__ method to do so, so this commit also forbids using a custom __setstate__ on a cache_hash=True object. Closes python-attrs#482 .
bors bot
referenced
this issue
in mozilla/normandy
Mar 4, 2019
1784: Scheduled weekly dependency update for week 09 r=peterbe a=pyup-bot ### Update [attrs](https://pypi.org/project/attrs) from **18.2.0** to **19.1.0**. <details> <summary>Changelog</summary> ### 19.1.0 ``` ------------------- Backward-incompatible Changes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Fixed a bug where deserialized objects with ``cache_hash=True`` could have incorrect hash code values. This change breaks classes with ``cache_hash=True`` when a custom ``__setstate__`` is present. An exception will be thrown when applying the ``attrs`` annotation to such a class. This limitation is tracked in issue `494 <https:/python-attrs/attrs/issues/494>`_. `482 <https:/python-attrs/attrs/issues/482>`_ Changes ^^^^^^^ - Add ``is_callable``, ``deep_iterable``, and ``deep_mapping`` validators. * ``is_callable``: validates that a value is callable * ``deep_iterable``: Allows recursion down into an iterable, applying another validator to every member in the iterable as well as applying an optional validator to the iterable itself. * ``deep_mapping``: Allows recursion down into the items in a mapping object, applying a key validator and a value validator to the key and value in every item. Also applies an optional validator to the mapping object itself. You can find them in the ``attr.validators`` package. `425 <https:/python-attrs/attrs/issues/425>`_ - Fixed stub files to prevent errors raised by mypy's ``disallow_any_generics = True`` option. `443 <https:/python-attrs/attrs/issues/443>`_ - Attributes with ``init=False`` now can follow after ``kw_only=True`` attributes. `450 <https:/python-attrs/attrs/issues/450>`_ - ``attrs`` now has first class support for defining exception classes. If you define a class using ``attr.s(auto_exc=True)`` and subclass an exception, the class will behave like a well-behaved exception class including an appropriate ``__str__`` method, and all attributes additionally available in an ``args`` attribute. `500 <https:/python-attrs/attrs/issues/500>`_ - Clarified documentation for hashing to warn that hashable objects should be deeply immutable (in their usage, even if this is not enforced). `503 <https:/python-attrs/attrs/issues/503>`_ ---- ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/attrs - Changelog: https://pyup.io/changelogs/attrs/ - Homepage: https://www.attrs.org/ </details> ### Update [botocore](https://pypi.org/project/botocore) from **1.12.101** to **1.12.106**. <details> <summary>Changelog</summary> ### 1.12.106 ``` ======== * api-change:``ec2``: Update ec2 client to latest version * api-change:``autoscaling-plans``: Update autoscaling-plans client to latest version ``` ### 1.12.105 ``` ======== * api-change:``ssm``: Update ssm client to latest version * api-change:``apigatewayv2``: Update apigatewayv2 client to latest version * api-change:``alexaforbusiness``: Update alexaforbusiness client to latest version * api-change:``application-autoscaling``: Update application-autoscaling client to latest version ``` ### 1.12.104 ``` ======== * api-change:``waf-regional``: Update waf-regional client to latest version * api-change:``waf``: Update waf client to latest version ``` ### 1.12.103 ``` ======== * api-change:``discovery``: Update discovery client to latest version * api-change:``organizations``: Update organizations client to latest version * api-change:``resource-groups``: Update resource-groups client to latest version * api-change:``opsworkscm``: Update opsworkscm client to latest version * api-change:``pinpoint``: Update pinpoint client to latest version * api-change:``mediaconvert``: Update mediaconvert client to latest version * api-change:``cur``: Update cur client to latest version ``` ### 1.12.102 ``` ======== * api-change:``elbv2``: Update elbv2 client to latest version * api-change:``mediastore``: Update mediastore client to latest version * api-change:``ce``: Update ce client to latest version * api-change:``autoscaling``: Update autoscaling client to latest version ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/botocore - Changelog: https://pyup.io/changelogs/botocore/ - Repo: https:/boto/botocore </details> ### Update [cffi](https://pypi.org/project/cffi) from **1.12.1** to **1.12.2**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https:/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/cffi - Changelog: https://pyup.io/changelogs/cffi/ - Docs: http://cffi.readthedocs.org </details> ### Update [idna](https://pypi.org/project/idna) from **2.6** to **2.8**. <details> <summary>Changelog</summary> ### 2.8 ``` ++++++++++++++++ - Update to Unicode 11.0.0. - Provide more specific exceptions for some malformed labels. ``` ### 2.7 ``` ++++++++++++++++ - Update to Unicode 10.0.0. - No longer accepts dot-prefixed domains (e.g. ".example") as valid. This is to be more conformant with the UTS 46 spec. Users should strip dot prefixes from domains before processing. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/idna - Changelog: https://pyup.io/changelogs/idna/ - Repo: https:/kjd/idna </details> ### Update [pyflakes](https://pypi.org/project/pyflakes) from **2.1.0** to **2.1.1**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https:/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pyflakes - Changelog: https://pyup.io/changelogs/pyflakes/ - Repo: https:/PyCQA/pyflakes </details> ### Update [google-api-core](https://pypi.org/project/google-api-core) from **1.7.0** to **1.8.0**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https:/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/google-api-core - Repo: https:/GoogleCloudPlatform/google-cloud-python </details> ### Update [google-cloud-core](https://pypi.org/project/google-cloud-core) from **0.28.1** to **0.29.1**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https:/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/google-cloud-core - Repo: https:/GoogleCloudPlatform/google-cloud-python </details> ### Update [protobuf](https://pypi.org/project/protobuf) from **3.6.1** to **3.7.0**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https:/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/protobuf - Changelog: https://pyup.io/changelogs/protobuf/ - Repo: https:/protocolbuffers/protobuf/releases - Homepage: https://developers.google.com/protocol-buffers/ </details> ### Update [boto3](https://pypi.org/project/boto3) from **1.9.101** to **1.9.106**. <details> <summary>Changelog</summary> ### 1.9.106 ``` ======= * api-change:``ec2``: [``botocore``] Update ec2 client to latest version * api-change:``autoscaling-plans``: [``botocore``] Update autoscaling-plans client to latest version ``` ### 1.9.105 ``` ======= * api-change:``ssm``: [``botocore``] Update ssm client to latest version * api-change:``apigatewayv2``: [``botocore``] Update apigatewayv2 client to latest version * api-change:``alexaforbusiness``: [``botocore``] Update alexaforbusiness client to latest version * api-change:``application-autoscaling``: [``botocore``] Update application-autoscaling client to latest version ``` ### 1.9.104 ``` ======= * api-change:``waf-regional``: [``botocore``] Update waf-regional client to latest version * api-change:``waf``: [``botocore``] Update waf client to latest version ``` ### 1.9.103 ``` ======= * api-change:``discovery``: [``botocore``] Update discovery client to latest version * api-change:``organizations``: [``botocore``] Update organizations client to latest version * api-change:``resource-groups``: [``botocore``] Update resource-groups client to latest version * api-change:``opsworkscm``: [``botocore``] Update opsworkscm client to latest version * api-change:``pinpoint``: [``botocore``] Update pinpoint client to latest version * api-change:``mediaconvert``: [``botocore``] Update mediaconvert client to latest version * api-change:``cur``: [``botocore``] Update cur client to latest version ``` ### 1.9.102 ``` ======= * api-change:``elbv2``: [``botocore``] Update elbv2 client to latest version * api-change:``mediastore``: [``botocore``] Update mediastore client to latest version * api-change:``ce``: [``botocore``] Update ce client to latest version * api-change:``autoscaling``: [``botocore``] Update autoscaling client to latest version ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/boto3 - Changelog: https://pyup.io/changelogs/boto3/ - Repo: https:/boto/boto3 </details> ### Update [django-cors-headers](https://pypi.org/project/django-cors-headers) from **2.4.0** to **2.4.1**. <details> <summary>Changelog</summary> ### 2.4.1 ``` ------------------ * Fix ``DeprecationWarning`` from importing ``collections.abc.Sequence`` on Python 3.7. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/django-cors-headers - Changelog: https://pyup.io/changelogs/django-cors-headers/ - Repo: https:/ottoyiu/django-cors-headers </details> ### Update [django-mozilla-product-details](https://pypi.org/project/django-mozilla-product-details) from **0.13** to **0.13.1**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https:/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/django-mozilla-product-details - Repo: https:/mozilla/django-product-details/ </details> ### Update [djangorestframework](https://pypi.org/project/djangorestframework) from **3.9.1** to **3.9.2**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https:/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/djangorestframework - Changelog: https://pyup.io/changelogs/djangorestframework/ - Homepage: https://www.django-rest-framework.org/ </details> ### Update [jsonschema](https://pypi.org/project/jsonschema) from **3.0.0** to **3.0.1**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https:/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/jsonschema - Changelog: https://pyup.io/changelogs/jsonschema/ - Repo: https:/Julian/jsonschema </details> ### Update [pytest-django](https://pypi.org/project/pytest-django) from **3.4.7** to **3.4.8**. <details> <summary>Changelog</summary> ### 3.4.8 ``` ------------------ Bugfixes ^^^^^^^^ * Fix DB renaming fixture for Multi-DB environment with SQLite (679) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pytest-django - Changelog: https://pyup.io/changelogs/pytest-django/ - Docs: https://pytest-django.readthedocs.io/ </details> ### Update [google-cloud-storage](https://pypi.org/project/google-cloud-storage) from **1.13.0** to **1.14.0**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https:/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/google-cloud-storage - Repo: https:/GoogleCloudPlatform/google-cloud-python </details> Co-authored-by: pyup-bot <[email protected]>
|
I feel slightly better that I am not the only one to screw this up. :-) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I just realized a bug in the code I wrote for hash code caching (#426). Because the hash code cache field gets serialized and deserialized by Pickle, when you deserialize a
cache_hash=Trueattrsobject, the hashcode will be the hashcode the object had at serialization-time. However, if your object has fields with hash codes which are not deterministic between interpreter runs, then on a new interpreter run your deserialized object will have a hash code which differs from a newly created identical object.We can fix this for
pickleby recomputing the hash code in__setstate__. Other serialization libraries which don't respect__setstate__will still have a problem, but I don't think we can do anything about that. If the__setstate__solution sounds acceptable I will implement it next week.The text was updated successfully, but these errors were encountered: