Some PoS tags not restored on deserialization of doc in specific conditions

[csvance]

Hi, when I serialize and deserialize a Doc with to_bytes() and from_bytes(), the .pos attribute of Token in specific conditions is not being restored. I have specifically observed this with SPACE and PUNCT.

import spacy
from spacy.tokens import Doc

sentence = "This is a sentence.\nAnother line. White   \n Space \n\n  Abuse   "

nlp = spacy.load('en')

original_doc = nlp(sentence)
original_len = len(original_doc)
original_bytes = original_doc.to_bytes()

deserialized_doc = Doc(nlp.vocab).from_bytes(original_bytes)
deserialized_len = len(deserialized_doc)
deserialized_bytes = deserialized_doc.to_bytes()

assert original_bytes == deserialized_bytes
assert original_len == deserialized_len
for idx in range(0, original_len):
    print("Original(%d): %s New(%d): %s" % (
    original_doc[idx].pos, original_doc[idx].pos_, deserialized_doc[idx].pos, deserialized_doc[idx].pos_))
    try:
        assert original_doc[idx].pos == deserialized_doc[idx].pos
    except AssertionError:
        print("Assertation failed.")

Original(89): DET New(89): DET
Original(99): VERB New(99): VERB
Original(89): DET New(89): DET
Original(91): NOUN New(91): NOUN
Original(96): PUNCT New(96): PUNCT
Original(102): SPACE New(0):
Assertation failed.
Original(89): DET New(89): DET
Original(91): NOUN New(91): NOUN
Original(96): PUNCT New(96): PUNCT
Original(95): PROPN New(95): PROPN
Original(102): SPACE New(102): SPACE
Original(95): PROPN New(95): PROPN
Original(102): SPACE New(102): SPACE
Original(95): PROPN New(95): PROPN
Original(102): SPACE New(0):
Assertation failed.

Heres an example with punctuation:

sentence = "Blah — Blahh — Blahhh"

Original(95): PROPN New(95): PROPN
Original(96): PUNCT New(0):
Assertation failed.
Original(95): PROPN New(95): PROPN
Original(96): PUNCT New(0):
Assertation failed.
Original(95): PROPN New(95): PROPN

However sentence = "Blah —— Blahh —— Blahhh" works just fine.

Your Environment

Info about spaCy

• I have tried with both the 2.0.5 release and building from master.
• spaCy version: 2.0.0rc2 (master ff9fc94)
• Platform: Linux-4.10.0-40-generic-x86_64-with-debian-stretch-sid
• Python version: 3.6.3
• Models: en
• Operating System: Ubuntu 16.04.3 LTS

[honnibal]

Sorry for the delay getting to this, and thanks for the nice report. Fixed now.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.