ISANLP is a Python 3 library that encompasses several open-source natural language processing tools for English and Russian and provides a framework for running them locally as a single pipeline or in a distributed environment via RPC. It also provides an easy to deploy docker container inemo/isanlp for starting various types of workers.
Current version 0.0.6
Warning since version 0.0.5 compatibility is broken with old containers! When installing new version of the library you need to pull new containers or use 0.0.1 version of the library (old containers are also tagged as 0.0.1).
In the basic use case, the library depends only on grpc library.
pip install grpcio
pip install git+https://github.com/IINemo/isanlp.git
docker run -ti --rm -p 3333:3333 inemo/isanlp
from isanlp.processor_remote import ProcessorRemote
ppl = ProcessorRemote(host='localhost', port=3333, pipeline_name='default')
text_ru = 'Мама мыла раму'
annotations = ppl(text_ru)
print(annotations)
Current:
- NLTK (English tokenizing, sentence splitting, lemmatizing, postagging; Russian tokenizing, sentence splitting).
- MyStem (Russian morphological analysis, postagging, lemmatizing).
- Polyglot (Language detection, English/Russian tokening, sentence splitting, named entity recognition).
- SyntaxNet (English/Russian dependency parsing in Universal dependencies format, postagging, morphological analysis).
- Postag converters.
- MaltParser: dependency parsing (for now without runtime and models).
- MaltParser CoNLL-2008: dependency parsing with a runtime and a model for English as a standalone docker container.
- Deep-SRL parser as a standalone docker container (semantic role labeling for English).
- Russian semantic role labeler (SRL) based on FrameBank and neural network models.
- UDPipe: morphological and syntactic parser (English/Russian).
To be included:
- English/Russian advanced neural network named entity recognition.
- English/Russian sentiment analysis.
- English/Russian anaphora resolution.
The most common usage consists in constructing a pipeline of processors with PipelineCommon
class. For example, the following pipeline processes texts in Russian and solves sequentially the following tasks: tokenization, sentence splitting, postagging, syntax parsing, and named entity recognition.
from isanlp.pipeline_common import PipelineCommon
from isanlp.ru.processor_tokenizer_ru import ProcessorTokenizerRu
from isanlp.processor_sentence_splitter import ProcessorSentenceSplitter
from isanlp.ru.processor_mystem import ProcessorMystem
from isanlp.processor_polyglot import ProcessorPolyglot
from isanlp.processor_syntaxnet_remote import ProcessorSyntaxNetRemote
ppl = PipelineCommon([(ProcessorPolyglot().detect_language,
['text'],
{0: 'lang'}),
(ProcessorTokenizerRu(),
['text'],
{0 : 'tokens'}),
(ProcessorSentenceSplitter(),
['tokens'],
{0 : 'sentences'}),
(ProcessorMystem(),
['tokens', 'sentences'],
{'lemma' : 'lemma',
'postag' : 'postag'}),
(ProcessorSyntaxNetRemote('localhost', 8207),
['tokens', 'sentences'],
{'syntax_dep_tree' : 'syntax_dep_tree'}),
(ProcessorPolyglot(),
['text'],
{'entities' : 'entities'})])
text_ru = 'Мама мыла раму.'
annotations = ppl(text_ru)
The pipeline contains a list of processors -- objects that perform separate language processing tasks. The result of the pipeline execution is a dictionary of "facets" -- different types of annotations extracted from the text by processors. The dictionary of annotations is stored inside the pipeline and filled up by processors. Processors get their input parameters from the annotation dictionary and save results into the dictionary.
The parameters of processors are specified in a tuple during pipeline construction:
PipelineCommon((<ProcObject>(), <list of input parameters>, <dictionary of output results>), ...)
You also should specify the label, which would be used to save your results in a pipeline annotation dictionary. If you do not provide a name for a result annotation it would be dropped from further processing. Processors can overwrite annotations aquired from other processors. To avoid overwriting just drop the result annotations.
Pipelines also can include other pipelines and remote processors:
ppl = PipelineCommon([(ProcessorRemote(host='some host', port=8207),
['text'],
{'tokens' : 'tokens',
'sentences' : 'sentences',
'lemma' : 'lemma',
'postag' : 'postag',
'morph' : 'morph'}),
(ProcessorSyntaxNetRemote(host='other host', port=7678),
['sentences'],
{'syntax_dep_tree' : 'syntax_dep_tree'})])
Conditional execution: TBD:
The pipeline returns the results in a dictionary with keys that were provided in the parameters of processors. If processor returns None, results are ommited. Available keys & values structures:
'lang'
(str) -- ISO code of detected language.'lemma'
(list) -- list of lists of string lemma annotations for each sentence. The first list corresponds to sentences, the second corresponds to annotations inside a sentence.'morph'
(list) -- list of lists of dictionary objects where the keys are the morphological features available for the current word.'postag'
(list) -- list of lists of string POS tags for each sentence.'sentences'
(list) -- list ofisanlp.annotation.Sentence
objects. Each Sentence contains:- begin(int) -- number of the first token in a sentence.
- end(int) -- number of the last token in a sentence + 1.
'syntax_dep_tree'
(list) -- list of lists ofisanlp.annotation.WordSynt
objects. WordSynt objects mean nodes in syntax dependency tree and have two members:- parent(int) -- number of the parent token.
- link_name(str) -- name of the link between this word and it's parent.
'tokens'
(list) -- list ofisanlp.annotation.Token
objects. Each Token contains:- text(str) -- as represented in text.
The library provides a docker container inemo/isanlp that contains all required packages and models to run a natural language pipeline. You can invoke RPC server (gRPC) with a default pipeline using the following command:
docker run --rm -p 3333:3333 inemo/isanlp
You can also run a custom processor instead the container using the simple command:
<some repo>/isanlp /start.py \
-m <custom module> \
-a <pipeline object>
The IsaNLP library provides several core routines, processors, and models that do not need many dependencies. However, the IsaNLP framework is significantly broader. It encompasses many other easy-to-use packages for various types of NLP. Currently they include:
- isanlp_srl_framebank -- the neural-based SRL parser for Russian based on well-known FrameBank corpus. The package provides the python 3 library for final users, trained models (Shelmanov and Devyatkin), a docker container inemo/isanlp_srl_framebank with ready-to-use NLP service, and a playground for training new deep learning models (jupyter notebook).
- isanlp_deep_srl -- the docker container inemo/isanlp_deep_srl with ready-to-use SRL service for English. The parser and models is created by Deep SRL project and adopted for IsaNLP framework.
- isanlp_parser_conll2008 -- the docker container inemo/isanlp_parser_conll2008 with ready-to-use dependency parsing service for English. The parser is based on MaltParser and CONLL-2008/2009 syntactic annotation (semantically rich). The model is developed by Surdenau et al.
- docker-syntaxnet -- the docker containers with Google's SyntaxNet for English, Russian, and Italian. The containers provides easy-to-use gRPC service for dependency parsing, pos-tagging, and morphology analysis.
Publish SRL parser for Russian.published.UDPipe support.Implemented.- Expand documentation.
- Automatic tests.
- Sentiment analysis for English and Russian.
- Anaphora resolution for English and Russian.
- Spacy wrapper.