POS tag of "LANG" assigned to tokens, causing a KeyError

[bdewilde]

How to reproduce the behaviour

Hello! I upgraded to v2.1.5 💫 and ran into an issue POS-tagging a text that wasn't present yesterday in v2.1.4. Specifically, the en_core_web_sm model assigns "LANG" as a POS tag for some tokens, which afaik isn't a valid value. This, in turn, raises a KeyError when calling tok.pos_ on the offending tokens, since parts_of_speech.IDS doesn't have "LANG" as a key.

Diving in, I see that the POS tags are nonsensical:

>>> [(tok, doc.vocab[tok.pos].text) for tok in doc]
...
 (., 'PROPN'),
 (, 'EOL'),
 (In, 'ADJ'),
 (this, 'CCONJ'),
 (case, 'INTJ'),
 (it, 'PART'),
 (was, 'SYM'),
 (several, 'LANG'),
 (feet, 'INTJ'),
 (below, 'ADJ'),
 (it, 'PART'),
 (., 'PROPN'),
 (But, 'CONJ'),
 (a, 'CCONJ'),
 (section, 'INTJ'),
...

I have no idea what's gone wrong. Here's a full example, using a fresh install of both spacy and the model:

In [1]: import spacy

In [2]: nlp = spacy.load("en")

In [3]: doc = nlp("This is an example sentence.")

In [4]: [(tok, doc.vocab[tok.pos].text) for tok in doc]
Out[4]:
[(This, 'CCONJ'),
 (is, 'SYM'),
 (an, 'CCONJ'),
 (example, 'INTJ'),
 (sentence, 'INTJ'),
 (., 'PROPN')]

Your Environment

• spaCy version: 2.1.5
• Platform: Darwin-18.6.0-x86_64-i386-64bit
• Python version: 3.7.0
• Models: es, en, xx

[honnibal]

Damn, we threw the symbols table out of alignment. I don't get why our tests didn't catch this :(. Fix forth-coming, sorry!

[honnibal]

Fixed, and v2.1.6 uploaded. Wheels coming soon. Thanks again for the quick report.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.