💫 Improve annotation serialisation

[honnibal]

A persistent source of problems in spaCy 1.0 has been the way data is saved and loaded. This issue describes what's changing, and will be updated as implementation proceeds. The changes will finally allow annotations to support the Pickle protocol, making it much easier to use spaCy with Spark and other tools.

How saving a Doc works in 1.x

Annotations are exported as integer IDs, into a numpy array. A custom Huffman-tree implementation is used to store the following fields:

• ORTH –- backing off to characters for unseen words
• SPACY (boolean flag for whether the word has a space after it)
• HEAD (as offset from token)
• TAG (part of speech tag, must have entry in tag map)
• ENT_IOB (IOB format for entity tags – one of 0, 1, 2, 3)
• ENT_TYPE (Must have entry in strings.json)
• DEP (Dependency label – must have entry in strings.json)

To load the serialized bytes into a new Doc object, you need a matching Vocab: to decode the bytes into the correct integer IDs, and then to map the integer IDs to the correct lexemes and tags.

How saving a Doc will work in 2.x

Individual documents will now be serialized as a tuple (attrs, text). Instead of our own Huffman codec, we'll just store the numpy arrays directly –- these compress fine in bulk anyway, especially when merged.

To save and load the document, you'll still need to have a reference to the Vocab object. To assist this, a new Binder class will be introduced, which will allow a group of documents to be saved and loaded together.

The Binder will support Pickle and JSON serialisation. More serialisation protocols can be added in future.

The Binder will support two header styles: a standalone format, and a diff against a model ID. If you serialise the Binder standalone, you'll read out the whole vocab, so this will be large for a small set of documents. However, it will ensure you'll be able to load the documents correctly in future. The diff format will require you to have the appropriate model ID loaded. The Binder will then include in its header any extra information not in the base model, for instance missing strings.

Summary of code changes

• New spacy.tokens.binder module, with spacy.tokens.binder.Binder class
• Support for Pickle protocol in Doc Token, Span objects
• Remove spacy.serialize subpackage

Related issues

• Pickle: Use in Apache Spark / English() object cannot be pickled #125, Pickling relies on temp files, doesn't clean them up #228, Pickle file load error #299, Cannot pandas.read_pickle() with spacy objects #377, Is there any problem to dump DOC object? #460, Pickling/Unpickling a doc does work #606, Pickle support for matcher object #930
• Serializer: (un)serializing a doc object #152, Documents parsed with .pipe() sometimes serialize into invalid byte strings #264, inconsistent sentence boundaries before and after serialization #322, compatibility of serialized Docs across spacy/vocab versions #343, Key error when serializing #350, spark does not utilize all cores when running a spacy parse or dies in serialization code #413, doc.to_bytes() fails with MemoryError: bad allocation #437, KeyError when serializing a doc object after adding a new entity label #514, Issues while loading Textacy corpus from disk. #530, to_bytes and from_bytes changes the token lemma #636, Loading NER model from dump doesn't work #664, Deserialization fails based on whether "nlp" object was used yet #728, Serializing user data in Doc objects #785, Deserialization fails with a new model instance. #927, UnicodeDecodeError in doc.from_bytes #985, High memory usage when deserializing Docs #992, Doc.to_bytes() throws KeyError on label-names for rule-matched spans #1011

[ines]

See the v2.0.0 alpha release notes and #1105 🎉

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.