Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cache results from spacy using pickle #9

Closed
aoldoni opened this issue Sep 18, 2016 · 4 comments
Closed

Cache results from spacy using pickle #9

aoldoni opened this issue Sep 18, 2016 · 4 comments
Assignees
@aoldoni
Labels
enhancement

Comments

@aoldoni
Copy link
Owner

aoldoni commented Sep 18, 2016 •
edited
Loading

  • Attempt to optimise performance by caching result trees.
  • Add parameter to force uncached processing
@aoldoni aoldoni changed the title Cache results from spacy using picle Cache results from spacy using pickle Sep 18, 2016
@aoldoni
Copy link
Owner Author

aoldoni commented Sep 18, 2016 •
edited
Loading

Seems not possible: https:/spacy-io/spaCy/search?q=pickle&type=Issues&utf8=%E2%9C%93
Seeing same error as: explosion/spaCy#377

@aoldoni aoldoni closed this as completed Sep 18, 2016
@aoldoni aoldoni reopened this Sep 18, 2016
@aoldoni aoldoni self-assigned this Sep 18, 2016
@aoldoni
Copy link
Owner Author

aoldoni commented Sep 18, 2016 •
edited
Loading

One way to resolve this is:

  • Parse text using spacy structure.
  • Transform spacy tree into custom Tree structure that supports edge labels (NLTK doesn't).
  • Change code all over to use this custom structure.
  • If, in the future, if extra data from spacy is needed, then add to this custom structure as part of the conversion method.
  • Pickle that structure and reload that structure instead of the spacy one.

Downside is basically reimplementing the spacy structure, but in a "picklable" manner.

@aoldoni
Copy link
Owner Author

aoldoni commented Sep 18, 2016

Attempted in 0f23de3

@aoldoni
Copy link
Owner Author

aoldoni commented Sep 18, 2016

Fixed in a32b85e by implementing TreeNode, a new tree structure that is serializable and mimics Spacy tree structure.

@aoldoni aoldoni closed this as completed Sep 18, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aoldoni aoldoni

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@aoldoni