EN Tokenizer Error: 'shell' tokenized as 'she', 'll', etc.

[rappdw]

The 1.6.0 tokenizer is incorrectly tokenizing words that have a 'she' prefix.

Examples:
'This sea shell is unique', tokenizes 'shell' as 'she', 'll'
'The shovel is in the shed', tokenizes 'shed' as 'she', 'd'

Your Environment

• Operating System: OSX & Linux
• Python Version Used: 3.6.0
• spaCy Version Used: 1.6.0
• Environment Information:

[ines]

Thanks for the report! The shell error was already fixed in #775, and I just added another test case for shed.

If you install from master, it should be fixed now – we'll also make a bug fix release soon that will include those changes.

[rappdw]

Thanks.

Looking at en.tokenizer_exceptions.EXCLUDE_EXC, there is perhaps one other case that should be added, id. As in "The id and the ego...". Currently 'id' tokenizes as 'i', 'd' and shouldn't in this case.

[ines]

Thanks, good point! Thinking about it, this is actually a tricky one... In general, we prefer to base the default tokenizer exceptions on what's most common. If we come across id in most genres in English, it's more likely to mean "i'd" than the Freudian concept of "id".

To deal with this problem, we've been thinking about adding a new method to the Tokenizer that lets you remove and replace exceptions. This would let you customise the tokenizer for the genre of text you're working with, from simple exception rules like this one to more complex regular expressions for punctuation. (At the moment, the only way to do this is to create a custom Tokenizer instance and override the defaults, which is not ideal.)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.