Wordlist

Description

Wordlist is a Ruby library and CLI for reading, combining, mutating, and building wordlists, efficiently.

Features

Supports reading .txt wordlists, and .gz, .bz2, .xz, .zip, and .7z compressed wordlists.
Supports building wordlists from arbitrary text. Also supports .gz, .bz2, .xz, .zip, and .7z compression.
Provides an advanced lexer for parsing text into words.
- Can parse/skip digits, special characters, whole numbers, acronyms.
- Can normalize case, apostrophes, and acronyms.
Supports wordlist operations for combining multiple wordlists together.
Supports wordlist modify or mutating the words in the wordlist on the fly.
Also provides a wordlist command.
Fast-ish

Examples

Reading

Open a wordlist for reading:

wordlist = Wordlist.open("passwords.txt")

Open a compressed wordlist for reading:

wordlist = Wordlist.open("rockyou.txt.gz")

Enumerate through a wordlist:

wordlist.each do |word|
  puts word
end

Create an in-memory list of literal words:

words = Wordlist::Words["foo", "bar", "baz"]

List Operations

Concat two wordlists together:

(wordlist1 + wordlist2).each do |word|
  puts word
end

Union two wordlists together:

(wordlist1 | wordlist2).each do |word|
  puts word
end

Subtract one wordlist from the other:

(wordlist1 - wordlist2).each do |word|
  puts word
end

Combine every word from wordlist1 with the words from wordlist2:

(wordlist1 * wordlist2).each do |word|
  puts word
end

Combine the wordlist with itself multiple times:

(wordslist ** 3).each do |word|
  puts word
end

Filter out duplicates from multiple wordlists:

(wordlist1 + wordlist2 + wordlist3).uniq.each do |word|
  puts word
end

String Manipulation

Convert every word in a wordlist to lowercase:

wordlist.downcase.each do |word|
  puts word
end

Convert every word in a wordlist to UPPERCASE:

wordlist.upcase.each do |word|
  puts word
end

Capitalize every word in a wordlist:

wordlist.capitalize.each do |word|
  puts word
end

Run String#tr on every word in a wordlist:

wordlist.tr('_','-').each do |word|
  puts word
end

Run String#sub on every word in a wordlist:

wordlist.sub("fish","phish").each do |word|
  puts word
end

Run String#gsub on every word in a wordlist:

wordlist.gsub(/\d+/,"").each do |word|
  puts word
end

Performs every possible mutation of each word in a wordlist:

wordlist.mutate(/[oae]/, {'o' => '0', 'a' => '@', 'e' => '3'}).each do |word|
  puts word
end
# dog
# d0g
# firefox
# fir3fox
# firef0x
# fir3f0x
# ...

Enumerates over every possible case variation of every word in a wordlist:

wordlist.mutate_case.each do |word|
  puts word
end
# cat
# Cat
# cAt
# caT
# CAt
# CaT
# cAT
# CAT
# ...

Building a Wordlist

Wordlist::Builder.open('path/to/file.txt.gz') do |builder|
  # ...
end

Add individual words:

builder.add(word)

Adding an Array of words:

builder.append(words)

Parsing text:

builder.parse(text)

Parsing a file's content:

builder.parse_file(path)

Requirements

ruby >= 3.0.0
zcat/gzip (for reading/writing .gz wordlists)
bzcat/bzip2 (for reading/writing .bz2 wordlists)
xzcat/xz (for reading/writing .xz wordlists)
unzip/zip (for reading/writing .zip wordlists)
7za (for reading/writing .7z wordlists)

Install

$ gem install wordlist

gemspec

gem.add_dependency 'wordlist', '~> 1.0'

Gemfile

gem 'wordlist', '~> 1.0'

Synopsis

usage: wordlist { [options] WORDLIST ... | --build WORDLIST [FILE ...] }

Wordlist Reading Options:
    -f {txt|gzip|bz2|xz|zip|7zip},   Sets the desired wordlist format
        --format
        --exec COMMAND               Runs the command with each word from the wordlist.
                                     The string "{}" will be replaced with each word.

Wordlist Operations:
    -U, --union WORDLIST             Unions the wordlist with the other WORDLIST
    -I, --intersect WORDLIST         Intersects the wordlist with the other WORDLIST
    -S, --subtract WORDLIST          Subtracts the words from the WORDLIST
    -p, --product WORDLIST           Combines every word with the other words from WORDLIST
    -P, --power NUM                  Combines every word with the other words from WORDLIST
    -u, --unique                     Filters out duplicate words

Wordlist Modifiers:
    -C, --capitalize                 Capitalize each word
        --uppercase, --upcase        Converts each word to UPPERCASE
        --lowercase, --downcase      Converts each word to lowercase
    -t, --tr CHARS:REPLACE           Translates the characters of each word
    -s, --sub PATTERN:SUB            Replaces PATTERN with SUB in each word
    -g, --gsub PATTERN:SUB           Replaces all PATTERNs with SUB in each word
    -m, --mutate PATTERN:SUB         Performs every possible substitution on each word
    -M, --mutate-case                Switches the case of each letter in each word

Wordlist Building Options:
    -b, --build WORDLIST             Builds a wordlist
    -a, --[no-]append                Appends to the new wordlist instead of overwriting it
    -L, --lang LANG                  The language to expect
        --stop-words WORDS...        Ignores the stop words
        --ignore-words WORDS...      Ignore the words
        --[no-]digits                Allow digits in the middle of words
        --special-chars CHARS        Allows the given special characters inside of words
        --[no-]numbers               Parses whole numbers in addition to words
        --[no-]acronyms              Parses acronyms in addition to words
        --[no-]normalize-case        Converts all words to lowercase
        --[no-]normalize-apostrophes Removes "'s" from words
        --[no-]normalize-acronyms    Removes the dots from acronyms

General Options:
    -V, --version                    Print the version
    -h, --help                       Print the help output

Examples:
    wordlist rockyou.txt.gz
    wordlist passwords_short.txt passwords_long.txt
    wordlist sport_teams.txt -p beers.txt -p digits.txt
    cat *.txt | wordlist --build custom.txt

Reading a wordlist:

$ wordlist rockyou.txt.gz

Reading multiple wordlists:

$ wordlist sport_teams.txt beers.txt

Combining every word from one wordlist with another:

$ wordlist sport_teams.txt -p beers.txt -p all_four_digits.txt
coors0000
coors0001
coors0002
coors0003
...

Combining every word from one wordlist with itself, N times:

$ wordlist words.txt -P 3

Mutating every word in a wordlist:

$ wordlist passwords.txt -m o:0 -m e:3 -m a:@
dog
d0g
firefox
fir3fox
firef0x
fir3f0x
...

Executing a command on each word in the wordlist:

$ wordlist directories.txt --exec "curl -X POST -F 'user=joe&password={}' -o /dev/null -w '%{http_code} {}' https://$TARGET/login"

Building a wordlist from a directory of .txt files:

$ wordlist --build wordlist.txt dir/*.txt

Building a wordlist from STDIN:

$ cat *.txt | wordlist --build wordlist.txt

Benchmarks

                                               user     system      total        real
Wordlist::Builder#parse_text (size=5.4M)   1.943605   0.003809   1.947414 (  1.955960)
Wordlist::File#each (N=1000)               0.000544   0.000000   0.000544 (  0.000559)
Wordlist::File#concat (N=1000)             0.001143   0.000000   0.001143 (  0.001153)
Wordlist::File#subtract (N=1000)           0.001360   0.000000   0.001360 (  0.001375)
Wordlist::File#product (N=1000)            0.536518   0.005959   0.542477 (  0.545536)
Wordlist::File#power (N=1000)              0.000015   0.000001   0.000016 (  0.000014)
Wordlist::File#intersect (N=1000)          0.001389   0.000000   0.001389 (  0.001407)
Wordlist::File#union (N=1000)              0.001310   0.000000   0.001310 (  0.001317)
Wordlist::File#uniq (N=1000)               0.000941   0.000000   0.000941 (  0.000948)
Wordlist::File#tr (N=1000)                 0.000725   0.000000   0.000725 (  0.000736)
Wordlist::File#sub (N=1000)                0.000863   0.000000   0.000863 (  0.000870)
Wordlist::File#gsub (N=1000)               0.001240   0.000000   0.001240 (  0.001249)
Wordlist::File#capittalize (N=1000)        0.000821   0.000000   0.000821 (  0.000828)
Wordlist::File#upcase (N=1000)             0.000760   0.000000   0.000760 (  0.000769)
Wordlist::File#downcase (N=1000)           0.000544   0.000001   0.000545 (  0.000545)
Wordlist::File#mutate (N=1000)             0.004656   0.000000   0.004656 (  0.004692)
Wordlist::File#mutate_case (N=1000)       24.178521   0.000000  24.178521 ( 24.294962)

License

See {file:LICENSE.txt} for details.

postmodern/wordlist.rb