Why would a spaCy blank pipeline tokenize input?

[bradybray]

Hello;

I'm initializing a blank pipe line via nlp = spacy.blank("en") and feeding it text to label some spans. The output DOC from the call to nlp is tokenizing the input sentence rather than returning a string. What would a string contain to make a blank pipeline do this? This is an issue because when I then run char_span on the doc the indexes into the text are returning tokens instead of the individual characters of the string.

Braden.

[honnibal]

A pipeline is defined as a tokenizer and then zero or more processes that modify the Doc in place. The Language.__call__ and Language.pipe() methods take strings as input and return Doc objects, which are tokenized.

[honnibal]

A span is always a sequence of tokens, so it has to start and end on a token boundary. There are some methods to get spans by character offset, but only if the characters fall on token boundaries.

[bradybray]

I'm able to call char_span on most docs. But a few of them are being tokenized then char_span indexes fails as the indexes are now tokens instead of characters in the string.

[bradybray]

I must be confused or am not clear in my understanding. If I process the strings into Docs then they're tokenized. Ok. I understand that. But then why does char_span index strings individually on some strings but only tokens on others? I don't know any other way to explain this.

[honnibal]

I do think there's some sort of confusion, and I'm not understanding. It might sound like a strange suggestion but I find LLMs like Claude or ChatGPT actually pretty good in these situations. Try showing it your code and asking it to explain the output. It knows spaCy pretty well because it's been stable a long time.

[bradybray]

Here is an example of the strings I'm parsing and the issues I'm encountering. This should be a bit more concrete. I'm using NLP on both of these strings:
HP 1y PW NBD w/DMR LJ Ent M608 M611 SVC CNBCP16170 7PS84A MFG Part# U9NG4PE
and
Del UltraSharp 24 Monitor - U2424H Part# 210-BKRR, 814-5380, 814-5381

If I char_span the first string, it gets carved up as expected at the integer boundaries. But the second string does not and returns a NoneType.

Does this help?

[bradybray]

I'm sorry... this is a counting error. I've either made a mistake in the annotations or the annotation is reporting incorrect numbers. The boundary I've set exceeds the length of the string.