Word list generation tools for github.com/indigane/taivutin
Word list generation using English Wiktionary.
- Install Python and Lua interpreters (authored with Python 3.7 and Lua 5.2)
- Download an English Wiktionary dump.
- Select a mirror dumps.wikimedia.org/mirrors
- Download a file called
enwiktionary-YYYYMMDD-pages-meta-current.xml
in your preferred archive format. - Unarchive it somewhere with enough space.
- Run
python parse_wiktionary.py <path-to-the/enwiktionary-YYYYMMDD-pages-meta-current.xml>
- Run
python inflect_wiktionary.py
- Run
lua inflect_wiktionary_generated.lua
- Run
python cleanup_wiktionary.py
- Run
python compress_wiktionary.py
You will end up with a bunch of generated files.
wiktionary_words.txt
will contain a list of partial words and their inflection metadata.
wiktionary_inflections.txt
will contain an index of inflections for words.
The rest of the generated files can be deleted.