XLMRSiamese

Extension of XLM-R with Siamese network and distance-based loss.

Setup

• source activate pytorch_p36
• pip install transformers
• [optional] Install apex

Data

• Prep data: python dataprep.py --parallel-data-dir data/parallel --output-dir data/processed

Pre-train

• python pretrain_distributed.py -i data/processed/train.tsv -o models --batch-size 16 --gpus 8

Train

• python train.py