/gentok

A simple tokenizer library created with c11

Primary LanguageCMIT LicenseMIT

GenTok

A simple tokenizer library created with c11.

Installation

If you are on windows platform, use a tool like MSYS2.

Use the below commands to build and release the gentok library.

# To build the debug version of the test program
make

# To build the release version of the test program
make release

# To build the static library
make install

# To clean up the working directory
make clean

License

This library is released with the MIT License.

Note

This library is in its intial stage. It has a long way to go before I use it as a real tokenizer.