No valid 'lang' when creating blank model with vocab from file

[svlandeg]

How to reproduce the behaviour

Previously, I saved en_core_web_lg to file, creating a vocab subdirectory, and its meta.json which reads (among other things)

"lang":"en",
"name":"core_web_lg"

Now, I'm attempting to use the vocab subdir as source for a blank model:

vocab = Vocab().from_disk(vocab_dir)
nlp = spacy.blank("en", vocab=vocab)
print(nlp("This is a test sentence"))
nlp.to_disk(output_dir)
nlp2 = spacy.load(output_dir)
print(nlp2("This is another test sentence"))

Which fails, giving the error ValueError: [E054] No valid 'lang' setting found in model meta.json.

And indeed, the meta.json of the new nlp object reads

"lang":""

Is this expected behaviour? When I run this in a unit test and replace the second line with

nlp = spacy.blank("en", vocab=en_vocab)

it does work correctly.

Your Environment

• spaCy version: 2.1.6
• Platform: Windows-10-10.0.17763-SP0
• Python version: 3.6.7

[ines]

I think the problem might be related to this:

spaCy/spacy/language.py

Lines 175 to 176 in 23ec07d

	def meta(self):
	self._meta.setdefault("lang", self.vocab.lang)

When you call Vocab().from_disk, it loads in the data, but it does that all on top of a blank Vocab. The language-specific Vocab is created via the create_vocab classmethod of the given Language defaults. The blank English model has that, but it's then overwritten by the vocab you pass in.

I do agree that this is slightly confusing, and I'm not immediately sure what the solution should be. In general, for a use case like this, the recommended best practice would probably be to remove all the pipeline components you don't want, which will give you the desired result: a base model with the vocab.

[svlandeg]

Ok, that makes sense. Thanks for the detailed explanation :-)

I was testing this in the context of serializing the KnowledgeBase for the entity linking functionality. To read the KB back in, you need at least the original vocab object, or otherwise the corresponding nlp object you used to create the KB with.

Because you have the option of loading the KB back in with the vocab only, you can then also create a blank English model with this vocab and create a new entity_linking pipe and have a functional pipeline, at least for demonstration purposes (in practice you'd need ner too, ofcourse).

See here: https:/svlandeg/spaCy/blob/feature/el-docs/examples/training/train_entity_linker.py

This works, but the IO doesn't because of the reasons described above.

But if you think that use-case is not relevant, we can just assume we always have access to the original nlp object with the right vocab...

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.