The program is mainly based on the C++ standard library, except for:
- Catch (http://catch-lib.net/) - a unit testing framework for C++
- CLI11 (https://github.com/CLIUtils/CLI11) - framework to parse CLI parameters
- pugixml (https://pugixml.org/) - to parse XML
The dependencies are included in 3rdparty/
and are to no concern of a user.
The code uses features of C++17, including:
- string_view
- experimental/filesystem
It was tested with GCC 7.2.1.
The program does not respond to parameters, it expects the *.tei
files to be placed in ./data
following the following pattern: <category name>_(test|train).tei
(as provided for this task).
For three variants it generates two files per variant, one called <variant name>.train
and one called <variant name>.test
.
Main program, uses tei_directory
to read the input data from files in ./dat
to write the variations.
On construction reads and parses all *.tei
files in the given directory. Extracts category and purpose (test/train) from file name.
writeTrainingFile
and writeTestFile
can be used to write the input files for fasttext for training and testing respectively. These functions take a FeatureExtractor
:
C++ callable that takes a Word
type and returns a std::vector<std::string_view>
(string_view
into the members of Word
). For examples see cli.cpp
. It internally uses token_extractor
and data_file
. The FeatureExtractor
is directly passed to data_file
.
Writes a fast text output file which it represents. It receives TeiFileParsed
(also defined here) and transforms them using a given FeatureExtractor
to lines of the format __label__<label> <joined by space list of extracted tokens>
(fasttext format). See tei_directory
for a description of FeatureExtractor
.
It internally uses token_extractor
to operate on the XML tree structure (provided by pugi::xml_document
). For token_extractor
it does provide it's own DocumentHandler
(to create newlines in the target data file) and wraps FeatureExtractor
into a WordHandler
compatible with token_extractor
.
Implements a pgui::xml_tree_walker
. For XML nodes of the name "TEI" it calls the provided DocumentHandler
(without parameter), for nodes of name "w" it calls the provided WordHandler
with a parameter of type Word
. Word
contains the original word and the XML attributes to the "w" tag (e.g. gender, word type, lemma).