/fairseq

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

Primary LanguagePythonMIT LicenseMIT



Support Ukraine MIT License Latest Release Build Status Documentation Status

Fairseq(-py) is a sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling and other text generation tasks.

We provide reference implementations of various sequence modeling papers:

List of implemented papers

What's New:

Previous updates

Features:

We also provide pre-trained models for translation and language modeling with a convenient torch.hub interface:

en2de = torch.hub.load('pytorch/fairseq', 'transformer.wmt19.en-de.single_model')
en2de.translate('Hello world', beam=5)
# 'Hallo Welt'

No Language Left Behind

No Language Left Behind (NLLB) is a first-of-its-kind, AI breakthrough project that open-sources models capable of delivering high-quality translations directly between any pair of 200+ languages — including low-resource languages like Asturian, Luganda, Urdu and more. It aims to help people communicate with anyone, anywhere, regardless of their language preferences.

To enable the community to leverage and build on top of NLLB, we open source all our evaluation benchmarks(FLORES-200, NLLB-MD, Toxicity-200), LID models and training code, LASER3 encoders, data mining code, MMT training and inference code and our final NLLB-200 models and their smaller distilled versions, for easier use and adoption by the research community.

This code repository contains instructions to get the datasets, optimized training and inference code for MMT models, training code for LASER3 encoders as well as instructions for downloading and using the final large NLLB-200 model and the smaller distilled models. In addition to supporting more than 200x200 translation directions, we also provide reliable evaluations of our model on all possible translation directions on the FLORES-200 benchmark. By open-sourcing our code, models and evaluations, we hope to foster even more research in low-resource languages leading to further improvements in the quality of low-resource translation through contributions from the research community.

Open Sourced Models and Community Integrations

Multilingual Translation Models

Model Name Model Type #params checkpoint metrics
NLLB-200 MoE 54.5B model metrics
NLLB-200 Dense 3.3B model metrics
NLLB-200 Dense 1.3B model metrics
NLLB-200-Distilled Dense 1.3B model metrics
NLLB-200-Distilled Dense 600M model metrics

All models are licensed under CC-BY-NC 4.0 available in Model LICENSE file. We provide FLORES-200 evaluation results for all the models. For more details see the Modeling section README.

Please use wget --trust-server-names <url> to download the provided links in proper file format.

LID Model

LID (Language IDentification) model to predict the language of the input text is available here under CC-BY-NC 4.0 license.

LASER3 Encoder Models

LASER3 models are available at LASER.

HuggingFace Integrations

[Coming Soon]

Installation

Follow installation instructions in INSTALL guide for running training/generation. For general instructions about fairseq and working with the codebase refer to fairseq README. For stopes and LASER follow their README files for installation.

Datasets

NLLB project uses data from three sources : public bitext, mined bitext and data generated using backtranslation. Details of different datasets used and open source links are provided in details here.

Primary Bitext

We provide a download script for public bitext data, and links to download NLLB-Seed data. For more details check here.

Mined Bitext

LASER3 teacher-student training code is open sourced here. LASER3 encoders and mined bitext metadata are open sourced in LASER repository. Global mining pipeline and monolingual data filtering pipelines are released and available in our stopes repository.

Backtranslated Bitext

Follow the instructions here to generate backtranslated data from a pretrained model.

Preparing Datasets for Training

We open source our dataset preparation pipeline for filtering/encoding/binarizing large scale datasets in stopes. Encoding the datasets are done using the new SPM-200 model which was trained on 200+ languages used in the NLLB project. For more details see link.

SPM-200 Artifacts download links
Model link
Dictionary link

Training NLLB Models

We open source all our model training and generation code in this repo. We also share code for finetuning our models on different domains like NLLB-MD. Additionally, we also share the code for online distillation that produced our 1.3B and 600M distilled models. For more details check the Modeling section Readme.

Evaluation and Generation

NLLB project includes release of evaluation datasets like Flores-200, NLLB-MD and Toxicity-200. For instructions to run evaluation see instructions here and for instructions to produce generations from the models follow instructions here.

Flores200 | NLLB-MD | Toxicity-200

Citation

If you use NLLB in your work or any models/datasets/artifacts published in NLLB, please cite :

@article{nllb2022,
  title={No Language Left Behind: Scaling Human-Centered Machine Translation},
  author={{NLLB Team} and Costa-jussà, Marta R. and Cross, James and Çelebi, Onur and Elbayad, Maha and Heafield, Kenneth and Heffernan, Kevin and Kalbassi, Elahe and Lam, Janice and Licht, Daniel and Maillard, Jean and Sun, Anna and Wang, Skyler and Wenzek, Guillaume and Youngblood, Al and Akula, Bapi and Barrault, Loic and Mejia-Gonzalez, Gabriel and Hansanti, Prangthip and Hoffman, John and Jarrett, Semarley and Sadagopan, Kaushik Ram and Rowe, Dirk and Spruit, Shannon and Tran, Chau and Andrews, Pierre and Ayan, Necip Fazil and Bhosale, Shruti and Edunov, Sergey and Fan, Angela and Gao, Cynthia and Goswami, Vedanuj and Guzmán, Francisco and Koehn, Philipp and Mourachko, Alexandre and Ropers, Christophe and Saleem, Safiyyah and Schwenk, Holger and Wang, Jeff},
  year={2022}
}

License

NLLB code and fairseq(-py) is MIT-licensed available in LICENSE file.