Update Replication Log and Refactor Instructions for MS Marco Passage Retreival - Entire Dev Set #116
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Compute Canada environment:
OS: linux CentOS-7
Java 13.0.1
Python 3.6.3
Re-Ranking with monoBERT
2020-11-13 14:36:43 [INFO] evaluate_passage_ranker: Reranking:
100%|██████████| 6980/6980 [63:20:00<00:00, 32.66s/it]
2020-11-16 05:56:44 [INFO] evaluate_passage_ranker: precision@1 0.2533
2020-11-16 05:56:44 [INFO] evaluate_passage_ranker: recall@3 0.45093
2020-11-16 05:56:44 [INFO] evaluate_passage_ranker: recall@50 0.80609
2020-11-16 05:56:44 [INFO] evaluate_passage_ranker: recall@1000 0.86289
2020-11-16 05:56:44 [INFO] evaluate_passage_ranker: mrr 0.38789
2020-11-16 05:56:44 [INFO] evaluate_passage_ranker: mrr@10 0.37922
Re-Ranking with monoT5
2020-11-13 14:46:35 [INFO] evaluate_passage_ranker: Reranking:
100%|██████████| 6980/6980 [26:50:07<00:00, 13.84s/it]
2020-11-14 17:36:42 [INFO] evaluate_passage_ranker: precision@1 0.25129
2020-11-14 17:36:42 [INFO] evaluate_passage_ranker: recall@3 0.45362
2020-11-14 17:36:42 [INFO] evaluate_passage_ranker: recall@50 0.80709
2020-11-14 17:36:42 [INFO] evaluate_passage_ranker: recall@1000 0.86289
2020-11-14 17:36:42 [INFO] evaluate_passage_ranker: mrr 0.38839
2020-11-14 17:36:42 [INFO] evaluate_passage_ranker: mrr@10 0.37986
I have also slightly refactored the instructions for MS Marco Entire. I've fixed some typos and added a note clarifying the installing of tensorflow on CC with the instruction:
pip install tensorflow_gpu