Incorrect tokenization of dash punctuation in Spanish

[BrianSladek]

In Spanish text, the conventions for using dashes and em-dashes as punctuation seems to be considerably different than in English. Spacy often does not tokenize the dash or em-dash as a separate token, instead keeping it attached to the closest word.

For example, the Spanish sentence:
—Yo me llamo... –murmuró el niño– Emilio Sánchez Pérez.
English Translation:
"My name is...", murmured the boy, "Emilio Sanchez Perez."

Here, the Spanish dash is used like a comma. The em-dash at the beginning of the sentence is used like a double quote. I believe that the fact that there is no space between the dash and word is throwing off the tokenizer.

The Spanish sentence above is tokenized as:
—Yo
me
llamo
...
–murmuró
el
niño–
Emilio
Sánchez
Pérez
.

I would expect the tokenization to be
—
Yo
me
llamo
...
–
murmuró
el
niño
–
Emilio
Sánchez
Pérez
.

Your Environment

• spaCy version: 2.0.12
• Platform: Darwin-18.0.0-x86_64-i386-64bit
• Python version: 3.7.0
• Models: de, es, en

[ines]

Thanks for the report and explanation – I didn't actually know this about Spanish!

I just opened a PR in #3281 that adds the dashes to the prefixes and suffixes and the change didn't break anything else, so I guess it seems fine 🤷‍♀️

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.