Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add resume logic to spacy pretrain #3652

Merged
merged 3 commits into from
Jun 12, 2019
Merged

Add resume logic to spacy pretrain #3652

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
85ed0bc
Added ability to resume training
tokestermw Apr 26, 2019
e9142b4
Add to readmee
tokestermw Apr 26, 2019
e57dc66
Remove duplicate entry
tokestermw Apr 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions spacy/cli/pretrain.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
from .._ml import Tok2Vec, flatten, chain, create_default_optimizer
from .._ml import masked_language_model
from .. import util
from .train import _load_pretrained_tok2vec
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move to util.py?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably should, yes. Not strictly necessary for this PR.



@plac.annotations(
Expand All @@ -36,6 +37,12 @@
seed=("Seed for random number generators", "option", "s", float),
n_iter=("Number of iterations to pretrain", "option", "i", int),
n_save_every=("Save model every X batches.", "option", "se", int),
init_tok2vec=(
"Path to pretrained weights for the token-to-vector parts of the models. See 'spacy pretrain'. Experimental.",
"option",
"t2v",
Path,
),
)
def pretrain(
texts_loc,
Expand All @@ -53,6 +60,7 @@ def pretrain(
min_length=5,
seed=0,
n_save_every=None,
init_tok2vec=None,
):
"""
Pre-train the 'token-to-vector' (tok2vec) layer of pipeline components,
Expand All @@ -70,6 +78,9 @@ def pretrain(
errors around this need some improvement.
"""
config = dict(locals())
for key in config:
if isinstance(config[key], Path):
config[key] = str(config[key])
msg = Printer()
util.fix_random_seed(seed)

Expand Down Expand Up @@ -112,6 +123,10 @@ def pretrain(
subword_features=True, # Set to False for Chinese etc
),
)
# Load in pre-trained weights
if init_tok2vec is not None:
components = _load_pretrained_tok2vec(nlp, init_tok2vec)
msg.text("Loaded pretrained tok2vec for: {}".format(components))
optimizer = create_default_optimizer(model.ops)
tracker = ProgressTracker(frequency=10000)
msg.divider("Pre-training tok2vec layer")
Expand Down
1 change: 1 addition & 0 deletions website/docs/api/cli.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,7 @@ $ python -m spacy pretrain [texts_loc] [vectors_model] [output_dir] [--width]
| `--n-iter`, `-i` | option | Number of iterations to pretrain. |
| `--use-vectors`, `-uv` | flag | Whether to use the static vectors as input features. |
| `--n-save_every`, `-se` | option | Save model every X batches. |
| `--init-tok2vec`, `-t2v` <Tag variant="new">2.1</Tag> | option | Path to pretrained weights for the token-to-vector parts of the models. See `spacy pretrain`. Experimental.|
| **CREATES** | weights | The pre-trained weights that can be used to initialize `spacy train`. |

### JSONL format for raw text {#pretrain-jsonl}
Expand Down