Missing labels on NER eval_model with classification_report set to True

[yoeldk]

Describe the bug
It seems like the labels shown in the report are taken from the (true) labels in the data to be evaluated on instead of the list of all labels the model was configured with.
If the evaluation is done on a (smaller) dataset that does not contain all possible labels the model was originally trained on (let's call the set of missing labels {Y}), then false positives of labels that belong to {Y} do not appear in the statistics.

To Reproduce
Evaluate NER model with classification_report on a data set that has predicted labels which do not appear in the input data as true positives.

Expected behavior
The expected behavior would be to produce a classification_report for each label in the original list of labels the model was configured with instead of the true labels in the data to be evaluated on.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[yoeldk]

I opened this bug several weeks ago which was tagged as "question", not sure why. Is there a plan to use an alternative to seqeval?
chakki-works/seqeval#39

[ThilinaRajapakse]

Do you know of any good alternatives we could try? I'm happy to consider other options.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.