forte: A Python repository from mingkaid

Forte is a toolkit for building Natural Language Processing pipelines, featuring cross-task interaction, adaptable data-model interfaces and so forth. Forte was originally developed and is actively contributed by Petuum in collaboration with other institutes. A mirror of this repository is maintained by Petuum Open Source.

Forte provides a platform to assemble state-of-the-art NLP and ML technologies in a highly-composable fashion, including a wide spectrum of tasks ranging from Information Retrieval, Natural Language Understanding to Natural Language Generation.

With Forte, it is extremely simple to build an integrated system that can search documents, analyze, extract information and generate language all in one place. This allows developers to fully utilize the strength of individual module, combine the results from each step, and enables the system to make fully informed decision at the end of the pipeline.

Forte not only makes it easy to integrate with arbitrary 3rd party tools (Check out these examples!), but also brings technology to you by offering a miscellaneous collection of deep learning modules via Texar, and a convenient model-data interface for casting tasks to models.

Core Design Principles

The core design principle of Forte is the abstraction of NLP concepts and machine learning models. It not only separates data, model and tasks but also enables interactions between different components of the pipeline. Based on this principle, we make Forte:

Composable: Forte helps users to decompose a problem into data, models and tasks. The tasks can further be divided into sub-tasks. A complex use case can be solved by composing heterogeneous modules via straightforward python APIs or declarative configuration files. The components (e.g. models or tasks) in the pipeline can be flexibly swapped in and out, as long as the API contracts are matched. This approach greatly improves module reusability, enables fast development and enhances the flexibility of using libraries.
Generalizable and Extensible: Forte not only generalizes well on a wide range of NLP tasks, but also extends easily to new tasks or new domains. In particular, Forte provides the Ontology system that helps users define types according to their specific tasks. Users can declaratively specify the type through simple JSON files and our Code Generation tool will automatically generate ready-to-use python files for your project. Check out our Ontology Generation documentation for more details.
Universal Data Flow: Forte enables a universal data flow that supports seamless data flow between different steps. Central to Forte's composable architecture, a transparent data flow facilitates flexible process interventions and simple pipeline management. Adaptive to generic data formats, Forte is positioned as a perfect tool for data inspection, component swapping and result sharing. This is particularly helpful during team collaborations!


A high level Architecture of Forte showing how ontology and entries work with the pipeline.


Forte stores results in data packs and use the ontology to represent task logic.

Package Overview

forte	an open-source toolkit for NLP
forte.data.readers	a data module for reading different formats of text data like CoNLL, Ontonotes etc
forte.processors	a collection of processors for building NLP pipelines
forte.trainer	a collection of modules for training different NLP tasks
ft.onto.base_ontology	a module containing basic ontologies like Token, Sentence, Document etc

Library API Example

A simple code example that runs Named Entity Recognizer

import yaml

from forte.pipeline import Pipeline
from forte.data.readers import CoNLL03Reader
from forte.processors import CoNLLNERPredictor
from ft.onto.base_ontology import Token, Sentence
from forte.common.configuration import Config


config_data = yaml.safe_load(open("config_data.yml", "r"))
config_model = yaml.safe_load(open("config_model.yml", "r"))

config = Config({}, default_hparams=None)
config.add_hparam('config_data', config_data)
config.add_hparam('config_model', config_model)


pl = Pipeline()
pl.set_reader(CoNLL03Reader())
pl.add(CoNLLNERPredictor(), config=config)

pl.initialize()

for pack in pl.process_dataset(config.config_data.test_path):
    for pred_sentence in pack.get_data(context_type=Sentence, request={Token: {"fields": ["ner"]}}):
        print("============================")
        print(pred_sentence["context"])
        print("The entities are...")
        print(pred_sentence["Token"]["ner"])
        print("============================")

Find more examples here.

Download and Installation

To install the released version from PyPI:

pip install forte

To install from source,

git clone https://github.com/asyml/forte.git
cd forte
pip install .

Getting Started

Examples
Documentation
Currently we are working on some interesting tutorials

Trouble Shooting

If you try to run generate_ontology script but encounter the following

Traceback (most recent call last):
  File "~/anaconda3/bin/generate_ontology", line 33, in <module>
    sys.exit(load_entry_point('forte', 'console_scripts', 'generate_ontology')())
  File "~/anaconda3/bin/generate_ontology", line 22, in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
  File "~/anaconda3/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 418, in distribution
    return Distribution.from_name(package)
  File "~/anaconda3/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 184, in from_name
    raise PackageNotFoundError(name)
importlib_metadata.PackageNotFoundError: forte

This is likely to be caused by multiple conflicting installation, such as installing both from source or from PIP. One way to solve this is to manually remove the script ~/anaconda3/bin/generate_ontology and re-install the package.

Contributing

If you are interested in making enhancement to Forte, please first go over our Code of Conduct and Contribution Guideline

License

Apache License 2.0

mingkaid/forte