The ACoLi CoNLL Libraries are an aggregator repository for several tools for processing, manipulating and transforming TSV formats developed at the Applied Computational Linguistics Lab at Goethe University Frankfurt, Germany.
In particular, the ACoLi CoNLL Libraries provide means of merging, transformation and querying of CoNLL and other TSV (CSV) data.
Special features are:
- support for CoNLL-specific extensions of the conventional TSV/CSV format
- applicable to any CoNLL dialect
- machine-readable specification and conversion between more than 20 CoNLL and other TSV/CSV formats, including, but not limited to CoNLL-U, CoNLL-X, CoNLL-2012 (OntoNotes), UniMorph, and SketchEngine.
- out-of-the-box parsing of CoNLL into RDF graphs
The ACoLi CoNLL libraries were originally designed for usage within Java and Shell scripts. Python bindings/replication in preparation. Via Fintan, CoNLL-RDF workflows are provided as Docker containers.
CoNLL-Merge (conll-merge/): Given two or more annotations of the same text, say, abc.conll with 10 columns and xyz.conll with 5 columns, align and concatenate them such that for every word from abc.conll, the corresponding row first has the original annotation from abc.conll, followed by the annotations from xyz.conll, i.e., 14 columns in total (the WORD column from xyz.conll that is used for alignment is suppressed by default).
If the files cannot be aligned, the user can chose from different conflict resolution strategies. By default, CoNLL-Merge will put unaligned words into separate rows and fill up the columns with the placeholder symbol ? to indicate alignment errors. Alternatively, the user can chose to enforce the tokenization of the first file onto the second, to split tokens from all files into their smallest common elements, to use minimal edit distance to find the most likely alignment, or any combination of these strategies.
Note that the alignment is sufficiently robust to be applied off-the-shelf in the following non-standard constellations:
- retokenize a previously annotated text (e.g., undo the tokenization of an annotation tool)
- the same text has been processed by different NLP tools that did some alternations to the primary data (e.g., escape certain symbols)
- the same text is used, but using different encodings (e.g., XML entities vs. UTF-8 characters)
- the same text is used, but tokenized according to different principles, e.g., one version with whitespace tokenization (
don'tgo), one with Penn-style tokenization (don'tgo) - different editions of the same text with differences in wording (e.g., different manuscripts of a historical text)
CoNLL-RDF (conll-rdf/): Given a CoNLL file, break it into sentences and expose every sentence (and, optionally, its context) as an RDF graph. Manipulate it with SPARQL, e.g., enrich it with links to an external knowledge base. Serialize back into a CoNLL format, an RDF representation or other corpus formats.
In combination with CoNLL-Merge, this allows complex tasks of transforming and enriching existing annotations, either in language resources or dynamically as a part of NLP workflows.
We see primary uses of CoNLL-RDF in
- the transformation of the output of NLP tools (shallow/fast transformation with direct replacement operations),
- the enrichment of existing annotations (on-the-fly information integration, e.g., by linking/comparison with an external knowledge graph), but also
- in complex aggregation tasks for the creation of gold data to train NLP tools on (annotation engineering, with up to hundreds of update operations).
Sample applications include, but are not limited to:
- Given corpus data in one CoNLL format, convert it into another CoNLL format (see CoNLL-Transform)
- Given the output of one NLP tool, convert it for consumption of another NLP tool (see our Flexible Integrated Transformation and Annotation Engineering platform [FINTAN] and the NLP workflow management system Teanga)
- Given a WordNet in a TSV format, transform it to OntoLex-Lemon (see our edition of the Open Multilingual WordNet
- Given a dictionary with annotations according to scheme A, use an external ontology to map its annotations to scheme B (see our edition of the Apertium Dictionaries
- Given a corpus with concurrent annotations from PropBank and Universal Dependencies, merge both annotations using CoNLL-Merge, transform the aggregate graph into a new corpus formalism (see our RRG corpus)
- Given a corpus with parts of speech and a lexicon, perform rule-based parsing and add lexical features (see our Middle High German corpus)
Special features missing from generic TSV/CSV parsers is that we support extensions of the CoNLL data model to account for syntactic dependencies (as object properties), chunk annotations (IOBES/BIO schemata), trees (phrase structure syntax), XML markup (e.g., for text structure annotation), and special extensions for semantic role labelling (one additional column per predicate per sentence).
There are plenty of CoNLL visualizers around, however, most of these are limited to a specific CoNLL dialect, e.g., CoNLL-U, and operate either memory-based (Brat) or against a relational backend (CWB). A downside of these that they are hard to adapt for changes in the data model, e.g., the introduction of another annotation layer. As an alternative, we allow to operate against an RDF triple/quad store as a backend.
CQP4RDF (cqp4rdf/): Visualizing and querying CoNLL annotations as graphs.
As the use of RDF technology is still somewhat uncharted territory for corpus linguists, we support a classical corpus query language, CQP, for querying the backend. However, it is possible to perform queries beyond the expressivity of CQP by extending CQP statements with SPARQL fragments.
- CoNLL-Transform (
conll-transform/): lightweight program to autoconfigure CoNLL-RDF for transforming one CoNLL format into another. - XML2CoNLL (
xml2conll/): generic converter from XML format(s) to CoNLL, can be used to feed XML into CoNLL-Merge, CoNLL-RDF or CQP4RDF.
Note that this repo uses the submodule functionality of git and that the aggregator repository is updated occasionally, only, to point to the most recent version. To retrieve the most up-to-date versions, clone this repo using
$> git clone --recurse-submodules --remote-submodules https://github.com/acoli-repo/conll
For updating an existing installation in the directory ./conll/, run
$> cd ./conll/
$> git submodule update --recursive .
Note that these repositories do not have strong interdependencies in the aggregator, but that this has been mostly created to faciliate a quick-and-easy local setup of all components of the ACoLi CoNLL Libraries. For development, we recommend to work within the submodule repositories directly.
For admins: To update all submodules to their latest version in this repository, run
$> cd ./conll/
$> git submodule update --recursive --remote --merge
$> git commit -a -m "updating submodules"
$> git push
For committing changes in a submodule, say ./conll-transform, make sure the submodule is on the correct branch and not in a detached head.
$> cd ./conll-transform
$> git checkout master
After doing local changes in ./conll-transform, push them
$> cd ./conll-transform # unless you're in there anyway
$> git commit -a -m "PLEASE DESCRIBE WHAT YOU DID"
$> git push
See here for more hints.
Please see Related.md for an overview over related tools.
If you use the ACoLi CoNLL Libraries, please refer to our reference publication:
- Chiarcos, C. & Schenk, N. (2018), The ACoLi CoNLL Libraries: Beyond Tab-Separated Values, In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 7-12, 2018, p.571-576.
If you work with individual components, please see the respective sub-folder for attribution and licensing information.