mT5 data sampling for pre-training

[cyk1337]

Hi @craffel, I have some questions about the data sampling and T5 span corruption. Could you please link me to the corresponding implementation?

• About sampling high/low-resource languages. Have you used one dataloader for each language out of 100+, and doing sampling during pre-training?
• About non-padding span corruption (merge_examples_to_reduce_padding ). Is mT5 trained without padding? If so, is each sampled batch the same language? Otherwise, each concatenated sample may contain different languages, and the span corruption may be across different languages.