Norwegian Bokmål sentence segmentation not working

[emilmuller]

Doing this:

import spacy
nlp = spacy.load("nb_core_news_sm")
doc = nlp("Hei på deg. Jeg har det fint.")
for sent in doc.sents:
  print(sent.text)

Outputs:

Hei på deg. Jeg har det fint.

:(

There's some mention in #3082 that maybe sbd needs to be added to the pipeline? I'm not sure if it is, how I can do it, or if it should be there by default?

[ines]

The default sentence segmentation happens via the dependency parser – so this seems to have not worked in this example. You can always add the rule-based sentencizer to the pipeline, though, which uses a simpler strategy: https://spacy.io/usage/linguistic-features#sbd-component

[svlandeg]

Also I'd like to point out that the name sbd is now deprecated in favour of the name sentencizer.

@ines : maybe to avoid confusion we should also rename the link in the docs?

[ines]

maybe to avoid confusion we should also rename the link in the docs?

The anchors are the only references I kept to not break backwards compatibility – and rewriting anchor links is a pain 😞

Also, on the topic of the nb model: I can confirm that the current Norwegian parser doesn't seem to segment sentences.

[adrianeboyd]

The training data only consists of individual sentences as documents, so the model seems to be determined to predict only one ROOT per text. I will work on it a bit to create some fake paragraphs so it can be retrained.

[ines]

Just released a new version of the nb_core_news_sm model that should resolve the problem 😃

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.