Version 1.x
Baseline is a library for reproducible deep learning research and fast model development for NLP. The library provides easily extensible abstractions and implementations for data loading, model development, training and export of deep learning architectures. It also provides implementations for high-performance, deep learning models for various NLP tasks, against which newly developed models can be compared. Deep learning experiments are hard to reproduce, Baseline provides functionalities to track them. The goal is to allow a researcher to focus on model development, delegating the repetitive tasks to the library.
-
baseline-core: An object-oriented Python library for rapid development of deep learning algorithms. The library provides extensible base classes for common components in a deep learning architecture (data loading, model development, training, evaluation, and export) in TensorFlow, PyTorch and DyNet. In addition, it provides strong, deep learning baselines for four fundamental NLP tasks -- Classification, Sequence Tagging, Seq-to-Seq Encoder-Decoders and Language Modeling. Many NLP problems can be seen as variants of these tasks. For example, Part of Speech (POS) Tagging, Named Entity Recognition (NER) and Slot-filling are all Sequence Tagging tasks, Neural Machine Translation (NMT) is typically modeled as an Encoder-Decoder task. An end-user can easily implement a new model and delegate the rest to the library.
-
mead: A library built for fast modeling, experimentation and development. It contains driver programs to run experiments from JSON configuration files to completely control the reader, trainer, model, and hyper-parameters.
-
xpctl: A command-line interface to track experimental results and provide access to a global leaderboard. After running an experiment through mead, the results and the logs are committed to a database. Several commands are provided to show the best experimental results under various constraints.
The workflow for developing a deep learning model using baseline is as follows:
- Map the problem to one of the existing tasks using a
<$task, dataset$>tuple, eg., NER on CoNLL 2003 dataset is a<tagger task, conll>. - Use the existing implementations in
Baselineor extend the base model class to create a new architecture. - Define a configuration file in
meadand run an experiment. - Use
xpctlto compare the result with the previous experiments, commit the results to the leaderboard database and the model files to a persistent storage if desired.
Additionally, the base models provided by the library can be exported from saved checkpoints directly into TensorFlow Serving for deployment in a production environment. The framework can be run within a Docker container to reduce the installation complexity and to isolate experiment configurations and variants. It is actively maintained by a team of core developers and accepts public contributions.
Baseline can be installed as a Python package:
./install_dev.sh baseline
Currently, xpctl depends on baseline but baseline is not available on PyPI so you need to install baseline before you install xpctl:
./install_dev.sh xpctl
Deep Learning Frameworks are evolving quickly, and changes are not always backwards compatible. We recommend recent versions of each framework. Baseline is known to work on most versions of TensorFlow, and is currently being run on versions between 1.5 and 1.12. PyTorch requires a version that is at least 0.4.
If you use the library, please cite the following paper:
@InProceedings{W18-2506,
author = "Pressel, Daniel
and Ray Choudhury, Sagnik
and Lester, Brian
and Zhao, Yanjie
and Barta, Matt",
title = "Baseline: A Library for Rapid Modeling, Experimentation and
Development of Deep Learning Algorithms targeting NLP",
booktitle = "Proceedings of Workshop for NLP Open Source Software (NLP-OSS)",
year = "2018",
publisher = "Association for Computational Linguistics",
pages = "34--40",
location = "Melbourne, Australia",
url = "http://aclweb.org/anthology/W18-2506"
}