Wrong lemmatization using only tokenization (en language)

[ajrader]

Observation

I noticed if I tried to use spacy for tokenization only (disabling 'ner', 'parser' and 'tagger') that i was getting an erroneous mapping for the english word 'spun'. As long as tagger is not disabled, this past tense verb is correctly mapped to 'spin'. But when 'tagger' is disabled then it gets mapped to 'spin-dry' which in my estimation is wrong.

I think the source of this error is due to line 35297 in https:/explosion/spaCy/tree/master/spacy/lang/en/lemmatizer/lookup.py
"spun": "spin-dry"

How to reproduce the behavior

import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp('spun', disable=['tagger'])
for tok in doc0:
print(tok.text, tok.lemma_)

The output of this is
spun spin-dry

I noticed the same behavior when using 'en_core_web_lg' and 'en_core_web_md' because I think this is the default lemmatizer mapping in all 'en' dictionaries.

Your Environment

• Operating System: Windows 10
• Python Version Used: 3.7.3
• spaCy Version Used: 2.1.7
• Environment Information: installed with conda

[svlandeg]

It looks like the look-up lemmatization indeed has two weird entries:

"spun": "spin-dry",
"dry": "spin-dry",

I guess it should be

"dry": "dry",
"spun": "spin",
"spun-dry": "spin-dry",

instead. You could make the edit to lookup.py and create a pull request if you like.

However, please also mind Ines' recent comment about lookup lemmatizations: they should be made redundant for languages where there is better lemmatization available.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.