postmodern/wordlist.rb

Refactor Wordlist::Parsers into a Lexer

postmodern opened this issue · 2 comments

postmodern commented

Parsers should become a proper String Tokenizer. It should be capable of scrubbing input text of unwanted punctuation, then splitting the input text into individual words.

postmodern commented

Use StringScanner instead of String#split.

postmodern commented

Implemented in aadacf6.