Reduce size of language data #4140

Rather than a large dict in Python source, the data is now a big json file. This includes a method for loading the json file, falling back to a compressed file, and an update to MANIFEST.in that excludes json in the spacy/lang directory. This focuses on Turkish specifically because it has the most language data in core.

This covers all lemmatizer.py files of a significant size (>500k or so). Small files were left alone. None of the affected files have logic, so this was pretty straightforward. One unusual thing is that the lemma data for Urdu doesn't seem to be used anywhere. That may require further investigation.

These are the languages that use a lemmatizer directory (rather than a single file) and are larger than English. For most of these languages there were many language data files, in which case only the large ones (>500k or so) were converted to json. It may or may not be a good idea to migrate the remaining Python files to json in the future.

The contents of this file were originally just copied from the Python source, but that used single quotes, so it had to be properly converted to json first.

This covers the json.gz files built as part of distribution.

Currently this gzip data on every build; it works, but it should be changed to only gzip when the source file has been updated.

* Make doc.is_sentenced return True if len(doc) < 2. * Make doc.is_nered return True if len(doc) == 0, for consistency. Closes explosion#3934

* more friendly textcat errors with require_model and require_labels * update thinc version with recent bugfix

…erators (explosion#3949) * Add regression test for issue explosion#3541 * Add comment on bugfix * Remove incorrect test * Un-xfail test

* minimal failing example for Issue explosion#3661 * referenced Issue explosion#3661 instead of Issue explosion#3611 * cleanup

Not actually supported in this alignment interpretation

* Fix for issue-3968 * Added contributor agreement * Made suggested changes

- minor formatting fixes - remove STTS tags not used in Tiger - update list of dependency relations to match tiger2dep

Test that the PhraseMatcher can match on overwritten NORM attributes.

* failing unit test for issue 3962 * attempt to fix Issue explosion#3962 * create artificial unit test example * using length instead of self.length * sp * reformat with black * find better ancestor within span and use generic 'dep' * attach to span.root if there is no appropriate ancestor * comment span text * clean up ancestor code * reconstruct dep tree to keep same number of sentences

Norwegian used to borrow from French but that doesn't appear to have been true for a while now, so the comment that was here is no longer relevant.

* Improve error message when model.from_bytes() dies When Thinc's model.from_bytes() is called with a mismatched model, often we get a particularly ungraceful error, e.g. "AttributeError: FunctionLayer has no attribute G" This is because we're trying to load the parameters for something like a LayerNorm layer, and the model architecture has some other layer there instead. This is obviously terrible, especially since the error *type* is wrong. I've changed it to raise a ValueError. The error message is still probably a bit terse, but it's hard to be sure exactly what's gone wrong. * Update spacy/pipeline/pipes.pyx * Update spacy/pipeline/pipes.pyx * Update spacy/pipeline/pipes.pyx * Update spacy/syntax/nn_parser.pyx * Update spacy/syntax/nn_parser.pyx * Update spacy/pipeline/pipes.pyx Co-Authored-By: Matthew Honnibal <[email protected]> * Update spacy/pipeline/pipes.pyx Co-Authored-By: Matthew Honnibal <[email protected]> Co-authored-by: Ines Montani <[email protected]>

* add `words` * update name of entity list to `ner` I think it might be a bit more consistent to have `ner` named `entities` or `ents` (and `ents` is actually set somewhere to `None`, which is a bit confusing), but it looks like renaming it would be a non-trivial decision.

…sion#4022)

…sion#4033) * Support simple training format in nlp.evaluate and add tests * Update docs [ci skip]

…ion#4035) * resolve edge case where no doc has tokens when calling textcat.predict * more explicit value test

* Typo fix for AllenAI url Changed incorrect home page url for AllenAI from appenai.org to allenai.org * Sign contributor agreement * Change date format

The example showed an incorrected import

…sion#4064) * Add links to tokenizer API docs to refer relevant information. * Add suggested changes Co-Authored-By: Ines Montani <[email protected]>

* ensure the language of vocab and nlp stay consistent across serialization * equality with =

* Improve NER per type scoring * include all gold labels in per type scoring, not only when recall > 0 * improve efficiency of per type scoring * Create Scorer tests, initially with NER tests * move regression test explosion#3968 (per type NER scoring) to Scorer tests * add new test for per type NER scoring with imperfect P/R/F and per type P/R/F including a case where R == 0.0

…#4049)

…losion#4067) * Update gold corpus code to properly ingest a directory of jsonlines files In response to: explosion#3975 * Update spacy/gold.pyx Co-Authored-By: Ines Montani <[email protected]>

Makes it consistent with other methods

…loses explosion#4068) * fixed syntax error in declaring variables with python 2.7 in spacy/lang/ko/__init__.py * fixed syntax error in declaring variables with python 2.7 in spacy/lang/ko/__init__.py * Update __init__.py * Create veer-bains.md * Update __init__.py fixed syntax errors in variable datatype assignment when calling spacy.blank("ko") with python 2.7

* Serbian stopwords added. (cyrillic alphabet) * spaCy Contribution agreement included. * Test initialize updated

* Update from master * Re-added Universe readme (explosion#3688) (closes explosion#3680) * Fix typo * Add version tag to `--base-model` argument (closes explosion#3720) * fixing regex matcher examples (explosion#3708) (explosion#3719) * Improve Token.prob and Lexeme.prob docs (resolves explosion#3701) * Fix DependencyParser.predict docs (resolves explosion#3561) * Update languages.json Co-authored-by: Bram Vanroy <[email protected]> Co-authored-by: Aaron Kub <[email protected]>

…d keys explosion#4074 (explosion#4079) * adding enhancement explosion#4074. * modified behavior to strictly require top level dictionary keys - issue explosion#4074 * pass expected keys to error message and add links as expected top level key

…[ci skip]

* Add validate option to EntityRuler * Add validate to EntityRuler, passed to Matcher and PhraseMatcher * Add validate to usage and API docs * Update website/docs/usage/rule-based-matching.md Co-Authored-By: Ines Montani <[email protected]> * Update website/docs/usage/rule-based-matching.md Co-Authored-By: Ines Montani <[email protected]>

…n#4097) * fixing vector and lemma attributes after retokenizer.split * fixing unit test with mockup tensor * xp instead of numpy

* Add entry for Blackstone in universe.json Add an entry for the Blackstone project. Checked JSON is valid. * Create ICLRandD.md * Fix indentation (tabs to spaces) It looks like during validation, the JSON file automatically changed spaces to tabs. This caused the diff to show *everything* as changed, which is obviously not true. This hopefully fixes that. * Try to fix formatting for diff * Fix diff Co-authored-by: Ines Montani <[email protected]>

… skip]

* update lang/zh * update lang/zh

* document token ent_kb_id * document span kb_id * update pipeline documentation * prior and context weights as bool's instead * entitylinker api documentation * drop for both models * finish entitylinker documentation * small fixes * documentation for KB * candidate documentation * links to api pages in code * small fix * frequency examples as counts for consistency * consistent documentation about tensors returned by predict * add entity linking to usage 101 * add entity linking infobox and KB section to 101 * entity-linking in linguistic features * small typo corrections * training example and docs for entity_linker * predefined nlp and kb * revert back to similarity encodings for simplicity (for now) * set prior probabilities to 0 when excluded * code clean up * bugfix: deleting kb ID from tokens when entities were removed * refactor train el example to use either model or vocab * pretrain_kb example for example kb generation * add to training docs for KB + EL example scripts * small fixes * error numbering * ensure the language of vocab and nlp stay consistent across serialization * equality with = * avoid conflict in errors file * add error 151 * final adjustements to the train scripts - consistency * update of goldparse documentation * small corrections * push commit * turn kb_creator into CLI script (wip) * proper parameters for training entity vectors * wikidata pipeline split up into two executable scripts * remove context_width * move wikidata scripts in bin directory, remove old dummy script * refine KB script with logs and preprocessing options * small edits * small improvements to logging of EL CLI script

…plosion#4110) * pytest file for issue4104 established * edited default lookup english lemmatizer for spun; fixes issue 4102 * eliminated parameterization and sorted dictionary dependnency in issue 4104 test * added contributor agreement

Missed this when I added the json.

The way gzipped json is loaded/saved in srsly changed a bit.

If a .json.gz file exists and is newer than the corresponding json file, it's not recompressed.

This only affected files >500kb, which was nouns for both languages and the generic lookup table for English.

It's unclear why, but the Norwegian (nb) tokenizer had empty files for adj/adv/noun/verb lemmas. This may have been a result of copying the structure of the English lemmatizer. This removed the files, but still creates the empty sets in the lemmatizer. That may not actually be necessary.

" furthest" and " skilled" - both prefixed with a space - were in the English lookup table. That seems obviously wrong so I have removed them.

The en tokenizer was including the removed _nouns.py file, so that's removed. The fr tokenizer is unusual in that it has a lemmatizer directory with both __init__.py and lemmatizer.py. lemmatizer.py had not been converted to load the json language data, so that was fixed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce size of language data #4140

Reduce size of language data #4140

Commits on Aug 18, 2019