Should "/" not be a separate token and cause a split?

[kootenpv]

In [32]: [x for x in nlp("I want/need something")]
Out[32]: [I, want/need, something]

Perhaps you're thinking of more common cases where it shouldn't be split? I'd almost think you should make "/" to be a conj.

[ines]

Thanks – and that's weird, I totally would have thought that this was already split correctly. Just added a regression test and played around with this quickly – it might be as simply as just adding the / character here.

But we need to make sure to test this properly and make sure it doesn't break any of the tokenizers that rely on the default punctuation.

[ines]

All worked as expected and the fix will be included in the next point release. The likely reason why / was excluded in the past was because we didn't have a good way of protecting URLs – since we introduced token_match and regex patterns for URLs, this is no problem anymore and / can be included in the infixes.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.