NER training fails: Process finished with exit code -1073741819 (0xC0000005)

[dkarmon]

I tried to train a new NER model based on the sample code in your site with my data (less than 4K records).
After only a few training iterations the training process suddenly fails with the following error message:

Process finished with exit code -1073741819 (0xC0000005)

I suspected that it had something to do with the batch, size but even if I try to train it example by example the training process fails.

I could not reproduce the error during debug - so maybe it's a memory leak issue? (allocation\releasing)
Please advise

Your Environment

• Operating System: Win10
• Python Version Used: 3.6.4
• spaCy Version Used: 2.0.12

[dkarmon]

I've managed to reproduce the error in debug mode. It seems that the batch size is indeed the root cause as if the batch contains more than 8 items the training process would eventually fail with that error code. However, if I switch to small batches, the training process would be completed.

@honnibal \ @ines please advise

[ghost]

I'm stuck at the exact problem for four days now. Getting that same error especially when I use training dataset of about 200 or more. I've no idea what the cause is. I'm trying to add a new entity type to a Spanish model But I'm stuck. Can someone please, please help. Thank you.

Environment:

spaCy version      2.0.12
Platform           Windows-10-10.0.17134-SP0
Python version     3.6.5
Models             en, es

[honnibal]

@Nandee89 The most likely cause is a batch that has an unusually large number of total words. How many words are in each of your documents, and how big is your batch size?

[ghost]

@honnibal Thank you for responding. I'm training it on a batch size of 799 examples which has a total of 26506 words. Documents/texts are of different sizes.

[ghost]

I also suspected that the dataset is too big..But I thought the larger the data the better model I would get. So I train a model on a smaller data and then try to retrain that model with another 150 batch hoping that the weights get updated. That approach lowered the quality of the model.

[ghost]

https:/explosion/spacy/blob/master/examples/training/train_new_entity_type.py

I'm trying to train it using the script found at the above link but with a Spanish model

[ghost]

It seems that the problem is with using a pre-existing model. I trained a blank Language class with the 799 training examples and I didn't crash at all. I trained two times and worked well in both.

if model is not None:
nlp = spacy.load(model) # load existing spaCy model
print("Loaded model '%s'" % model)
else:
nlp = spacy.blank('es') # create blank Language class
print("Created blank 'es' model")

[dkarmon]

@honnibal I used a batch size of 64 where each instance consists of approximately 15 tokens.

[honnibal]

This problem should now be resolved --- v2.0.17 has a fix for a memory error I think was at fault. You can already try it with pip install spacy==2.0.17.dev1 if you see this before 2.0.17 is out.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.