CovidQA + MSMARCO doc Replication log

[yuxuan-ji]

CovidQA: All identical except for BM25
MS-MARCO doc: Identical
See comments for detailed replication results

[yuxuan-ji]

CovidQA

Colab Environment

OS: Ubuntu 18.04.3 LTS
Java: openjdk 11.0.8 2020-07-14
Python: Python 3.6.9
Tested with both Tesla P100-PCIE-16GB and Tesla T4

Replication results

Random NL Question: Identical

precision@1	0.0
recall@3	0.0199546485260771
recall@50	0.3247165532879819
recall@1000	1.0
mrr	0.03999734528458418
mrr@10	0.020888672929489253

Random Keyword Query: Identical

precision@1	0.0
recall@3	0.0199546485260771
recall@50	0.3247165532879819
recall@1000	1.0
mrr	0.03999734528458418
mrr@10	0.020888672929489253

BM25 NL Question: Not identical

precision@1	0.15384615384615385
recall@3	0.21865889212827985
recall@50	0.7208778749595076
recall@1000	0.7582928409459021
mrr	0.25329970378011524
mrr@10	0.23344131303314977

BM25 Keyword Query: Not identical

precision@1	0.15384615384615385
recall@3	0.21865889212827985
recall@50	0.7208778749595076
recall@1000	0.7582928409459021
mrr	0.25441237140238665
mrr@10	0.23493413238311195

T5 NL Question: Identical

precision@1	0.2789115646258503
recall@3	0.41854551344347257
recall@50	0.92555879494655
recall@1000	1.0
mrr	0.417982565405279
mrr@10	0.4045405463772811

T5 Keyword Query: Identical

precision@1	0.24489795918367346
recall@3	0.38566569484936825
recall@50	0.9231778425655977
recall@1000	1.0
mrr	0.37988285486956513
mrr@10	0.3671336788683727

Keeping this as draft for now as I wasn't able to replicate BM25 results.

I took a look at what changed since Jul 13 (last time the CovidQA doc was changed) and there doesn't seem to be any significant change to the BM25 ranker (just the Pyserini api change)

[lintool]

@daemon can you comment?

[yuxuan-ji]

MS-MARCO doc

Colab Environment

OS: Ubuntu 18.04.3 LTS
Java: openjdk 11.0.8 2020-07-14
Python: Python 3.6.9
Tested with Tesla T4

Replication results

1st half: Identical

2020-09-09 14:27:13 [INFO] evaluate_document_ranker: Reranking:
100% 25/25 [7:50:24<00:00, 1129.00s/it]
2020-09-09 22:17:38 [INFO] evaluate_document_ranker: precision@1 0.16
2020-09-09 22:17:38 [INFO] evaluate_document_ranker: recall@3    0.44
2020-09-09 22:17:38 [INFO] evaluate_document_ranker: recall@50   0.84
2020-09-09 22:17:38 [INFO] evaluate_document_ranker: recall@1000 0.88
2020-09-09 22:17:38 [INFO] evaluate_document_ranker: mrr         0.33663
2020-09-09 22:17:38 [INFO] evaluate_document_ranker: mrr@10      0.33171

2nd half: Identical

2020-09-09 15:15:08 [INFO] evaluate_document_ranker: Reranking:
100% 25/25 [8:53:00<00:00, 1279.21s/it]
2020-09-10 00:08:09 [INFO] evaluate_document_ranker: precision@1 0.24
2020-09-10 00:08:09 [INFO] evaluate_document_ranker: recall@3    0.32
2020-09-10 00:08:09 [INFO] evaluate_document_ranker: recall@50   0.76
2020-09-10 00:08:09 [INFO] evaluate_document_ranker: recall@1000 0.88
2020-09-10 00:08:09 [INFO] evaluate_document_ranker: mrr         0.31052
2020-09-10 00:08:09 [INFO] evaluate_document_ranker: mrr@10      0.29133

[ronakice]

LGTM! Thanks @yuxuan-ji