Retrain transformer-based models in allennlp_models.pretrained

[epwalsh]

Updates from the new transformers/tokenizers release has broken some of these.

• Fine-grained NER transformer
• RoBERTa SNLI @epwalsh - https://beaker.org/ex/ex_3xaw1xzw6689/tasks/tk_zxz18n8idbn0
• RoBERTa MNLI @epwalsh - https://beaker.org/ex/ex_jbf2biqowfwa/tasks/tk_28u7smfn77o9
• BERT SRL
• RoBERTa SST @AkshitaB

[dirkgr]

I started doing BERT SRL too because I already had the environment for that spun up.

[dirkgr]

@AkshitaB, when you touch the RoBERTa SST model, try it without the cls_is_last_token setting. It's more correct to not have it.

[matt-gardner]

I see BERT SRL checked here, but there's this issue that says that performance actually doesn't match, with a solution that seems to fix it: #4392 (comment). Have you checked performance against the original reported performance? Seems like a simple config file fix to get performance back up, if this is an issue.

I was just looking at the SRL and BERT SRL models, though, and I think we probably can just combine them at this point. I don't think we gain much by using the srl_bert.py code at this point, and it's hard-coded for an earlier version of transformers that makes it so you can't easily update it for roberta, which would surely be better. But, that should be a separate issue.

[dirkgr]

The srl_bert.py code is simpler though. It's a much more transformer-native implementation. If we can get that one to perform as well as srl.py, I might argue for keeping srl_bert.py and removing the other one.

[matt-gardner]

"transformer-native" means "I have to use transformers, and I can't even try using anything else." It kind of goes against the whole point of the abstractions that we have.

[dirkgr]

While that's true, it's hard to argue in print for a solution that's more complicated and different from the standard if it doesn't also improve results.

In principle the srl_bert.py approach should work with any token embedder, but it might not do as well as a custom architecture.

[dirkgr]

We're tracking the remaining issue in #4521.