Update lemma and vector information after splitting a token #4097
Conversation
svlandeg
added
bug
| def test_issue3540(en_vocab): | ||
|
|
||
| words = ["I", "live", "in", "NewYork", "right", "now"] | ||
| tensor = np.asarray([[1.0, 1.1], [2.0, 2.1], [3.0, 3.1], [4.0, 4.1], [5.0, 5.1], [6.0, 6.1]], dtype="f") |
There was a problem hiding this comment.
That's great! 👍 (We should remember this test as a nice example for how to test even pretty complex stuff related to the models without having to actually load or run them.)
|
Hmm, looks like the build failure is a problem with Azure Pipelines:
I'll restart! 🤞 |
|
Hmmm, now this random matcher (?) test keeps failing again randomly on only one configuration... wtf... 🤔 |
|
Where do you see the failing test? I only still see the "Hosted Agent lost communication" error. |
|
@svlandeg See here for all builds: https://dev.azure.com/explosion-ai/Public/_build?definitionId=8&_a=summary Restarted the test again and it now passed 🤷♀️ (just doesn't seem to be reflected in the PR) |
|
Weird 😬 |
|
Patch looks good, thanks! |
|
I'll just go ahead and merge this! There really seems to be nothing wrong with this PR – there's likely something wrong somewhere, though, that makes the tests flaky like this. |
…n#4097) * fixing vector and lemma attributes after retokenizer.split * fixing unit test with mockup tensor * xp instead of numpy
Description
This PR fixes two related bugs that happened after running
retokenizer.split:The
lemmainformation was not updated and was still referring to the oldtoken.textattributes. This is fixed/reset by callingtoken.lemma = 0on the split tokens.The
vectorattributes were left unchanged, causing an out-of-bounds error as described in Issue Problem with vector attribute after calling Retokenizer.split #3540. This is fixed by extending thedoc.tensorand setting the vectors of the split tokens to an array of zeros (not sure what else to do).Types of change
bug fix
Checklist