Error in deserializing Spacy doc from disk

[sahelsoft]

How to deserialize Spacy results from disk

I need to run an algorithm on a lot of text files. In order to pre-process them, I use Spacy which has pre-trained models in different languages. Since the pre-processed results are employed in different parts of the algorithm, it is better to save them on disk once and load them many times. However, the Spacy deserialization method makes an error. I wrote a simple code to show the error:

de_nlp=spacy.load("de_core_news_sm",disable=['ner', 'parser'])
de_nlp.add_pipe(de_nlp.create_pipe('sentencizer'))
doc = de_nlp(text_file_content)

for ix, sent in enumerate(doc.sents, 1):
    print("--Sentence number {}: {}".format(ix, sent))
    lemma = [w.lemma_ for w in sent]
    print(f"Lemma ==> {lemma}")

#Serialization and Deserilization 
doc.to_disk("/tmp/test_result.bin")
new_doc = Doc(Vocab()).from_disk("/tmp/test_result.bin")

for ix, sent in enumerate(new_doc.sents, 1):
    print("--Sentence number {}: {}".format(ix, sent))
    lemma = [w.lemma_ for w in sent]
    print(f"Lemma ==> {lemma}")

However, the above example code makes the following error:

Traceback (most recent call last): File "/tmp/test_result.bin", line 14, in <module> for ix, sent in enumerate(new_doc.sents, 1): File "doc.pyx", line 535, in __get__ ValueError: [E030] Sentence boundaries unset. You can add the 'sentencizer' component to the pipeline with: nlp.add_pipe(nlp.create_pipe('sentencizer')) Alternatively, add the dependency parser, or set sentence boundaries by setting doc[i].is_sent_start.

I tried to change the first two line to "de_nlp=spacy.load("de_core_news_sm")", but it still gives some other errors.
I really needs your help. Any information about this topic is appreciated.

My Environment

• Operating System: Windows 10
• Python Version Used: 3.6
• spaCy Version Used: 2.0.18
• Environment Information: PyCharm 2017

[ines]

Thanks for the report! I think what's going on here is that the Doc's is_sentenced property (which indicates whether sentence boundaries have been applied) comes back incorrectly when you load the Doc back in, so spaCy thinks the Doc doesn't have sentence boundaries.

As a quick workaround, you can do the following:

new_doc.is_parsed = True

It's a hack, but it should work. Basically, if the Doc is parsed (or spaCy thinks it is), Doc.is_sentenced will return True as well, so you can trick spaCy a little here.

[sahelsoft]

Thank you. But it doesn't work. It makes some error while printing the content of new_doc.

[andrebm]

Same problem here! it takes 7 minutes to load.. strange indeed!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.