Can't reproduce SRL result with allennlp==1.0.0

[edchengg]

System (please complete the following information):

OS: Ubuntu 18.04.3 LTS
Python version: 3.7
AllenNLP version: v1.0.0
PyTorch version: 1.5
Allennlp-models: v1.0.0

Question
Hi @DeNeutoy , I try to reproduce the results on the OntoNotes dataset (conll 2012) in the Shi et al., 2019 paper used in the SRL demo. However, I could only get F1 around 0.79.

Command I used:

allennlp evaluate https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.03.24.tar.gz /data/conll-formatted-ontonotes-5.0/conll-formatted-ontonotes-5.0-12/conll-formatted-ontonotes-5.0/data/conll-2012-test/data/english

I also tried to train the model with config file in allennlp-models but only get to F1=79.

I found a related issue #4220 and was able to reproduce the result (86.5) with allennlp==0.9 and an old checkpoint https://s3-us-west-2.amazonaws.com/allennlp/models/bert-base-srl-2019.06.17.tar.gz. But I guess it might be worth reporting the issue since 1.0 is a stable version now.

Any help would be appreciated!

[epwalsh]

Possibly related to #4216

[edchengg]

Possibly related to #4216

I guess there are some problems with pytorch_transformer and transformer lib.

[Riccorl]

I saw in #4457 that the BERT SRL model is trained again. I evaluated it with allennlp==1.1.0rc2.dev20200721 and it still produce F1 below 80. Is it normal to have this performance since the 1.1 is not stable yet or it should work by now?

Command

allennlp evaluate "https://storage.googleapis.com/allennlp-public-models/bert-base-srl-2020.07.14.tar.gz" [data]

[Riccorl]

By changing the indexer in the dataset reader from SingleIdTokenIndexer to PretrainedTransformerIndexer the model seems to work as intended. I cannot complete a full train at the moment but the scores are higher after few epochs than the previous full train.

[edchengg]

By changing the indexer in the dataset reader from SingleIdTokenIndexer to PretrainedTransformerIndexer the model seems to work as intended. I cannot complete a full train at the moment but the scores are higher after few epochs than the previous full train.

Thanks! I will test it asap