This is our example repository for submission guidelines to the efficiency benchmark.
We focus on WMT
for Seq2Seq MT efficiency evaluation.
- WMT: We provide an interface for
wmt14-en-de
(English to German),wmt14-de-en
(German to English),wmt16-en-ro
(English to Romanian) andwmt16-ro-en
(Romanian to English) tasks.
We plan to add more tasks soon!
submission
├── Dockerfile # Spec for Docker runtime (do not modify)
├── README.md # Instructions
├── auto.py # Template for Seq2Seq Model submissions
├── entrypoint.py # Driver class for usage (see submission format below)
├── mbart.py # MBART model family example (for WMT tasks)
├── opus.py # OPUS model family example (for WMT tasks)
├── requirements.txt # Requirements for runtime. You might need to change this
To install the benchmark code see https://github.com/allenai/efficiency-pentathlon.
https://github.com/haopeng-nlp/submission.git
cd submission
# In the same (new) environment as above. These will be *your* requirements to run.
pip install -r requirements.txt
TASK=wmt14-en-de
SCENARIO=single_stream
MODEL="facebook/mbart-large-50-many-to-many-mmt"
# Everything after -- is your command with specific instructions for the Python script
efficiency-pentathlon run --task $TASK --max_batch_size 50 --scenario $SCENARIO \
-- python entrypoint.py --model $MODEL --$TASK
TASK=wmt14-en-de
MODEL="facebook/mbart-large-50-many-to-many-mmt"
# Everything after -- is your command with specific instructions for the Python script
efficiency-pentathlon submit --task $TASK --max_batch_size 50 \
-- python entrypoint.py --model $MODEL --$TASK