Tokenizer cache doesn't handle modifications to special cases or token_match correctly

[adrianeboyd]

How to reproduce the behaviour

The github suggested related issues were actually helpful! #1061 seems to have snuck back in. It works in 2.0.18, not in 2.1.0.

Modifications to special cases and token_match don't work if the pipeline has been run at least once due to the tokenizer cache.

import spacy
from spacy.symbols import ORTH

text = '(_SPECIAL_) A/B'

nlp = spacy.load('en_core_web_sm')
nlp.tokenizer.add_special_case('_SPECIAL_', [{ORTH: '_SPECIAL_'}])
nlp.tokenizer.add_special_case('A/B', [{ORTH: 'A/B'}])
print([token.text for token in nlp(text)])
# ['(', '_SPECIAL_', ')', 'A/B']

nlp = spacy.load('en_core_web_sm')
print([token.text for token in nlp(text)])
# ['(', '_', 'SPECIAL', '_', ')', 'A', '/', 'B']
nlp.tokenizer.add_special_case('_SPECIAL_', [{ORTH: '_SPECIAL_'}])
nlp.tokenizer.add_special_case('A/B', [{ORTH: 'A/B'}])
print([token.text for token in nlp(text)])
# ['(', '_', 'SPECIAL', '_', ')', 'A/B']

text = "This is a URL: http://example.com/file.html."

nlp = spacy.load('en_core_web_sm')
nlp.tokenizer.token_match = None
print([token.text for token in nlp(text)])
# ['This', 'is', 'a', 'URL', ':', 'http://example.com', '/', 'file.html', '.']

nlp = spacy.load('en_core_web_sm')
print([token.text for token in nlp(text)])
# ['This', 'is', 'a', 'URL', ':', 'http://example.com/file.html', '.']
nlp.tokenizer.token_match = None
print([token.text for token in nlp(text)])
# ['This', 'is', 'a', 'URL', ':', 'http://example.com/file.html', '.']

Info about spaCy

• spaCy version: 2.1.8
• Platform: Linux-4.19.0-5-amd64-x86_64-with-debian-10.0
• Python version: 3.7.3

[adrianeboyd]

This caching problem has been making me think I was losing my mind while testing special cases and token_match with the tokenizer.

Here's the commit that went missing in from v1->v2 that deals with the cache problem:

4b2e5e5

I think that a solution like this could fix the problem, but I'm not sure it's 100% correct for v2.

When I test this with 2.0.18 it seems to work, but I'm not sure why given the minimal differences in the tokenizer between 2.0.18 and 2.1.8.

[honnibal]

Wow, no idea how that patch went missing! Glad I wrote some notes on that...

So can we just take that commit?

[adrianeboyd]

No, it doesn't quite work, either. I have a new version coming...

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.