Luau Lexer

A Luau lexer designed for Roblox.

Module structure

The location of each module is also listed at the top of the file
You can make the latest version of this module by downloading/cloning the repo and organizing the modules in Roblox Studio as stated below...

   Lexer
    └─ Patterns
    └─ ScanWords
        └─ Bools
        └─ Globals
        └─ Keywords

Examples of use

Parsing a lua string

Some arbitrary code:

local something = "a string"
print("this is " .. something)

Parsing it:

local source = [[local something = "a string"
print("this is " .. something)]]
local lexer = require(lexer)
local parsed = lexer:Parse(source)

print(parsed)

Outputs:

{
    {
        ["matches"] =  ▶ {...},
        ["raw"] = "local",
        ["type"] = "keyword"
    },
    {
        ["matches"] =  ▶ {...},
        ["raw"] = " ",
        ["type"] = "whitespace"
    },
    {
        ["matches"] =  ▶ {...},
        ["raw"] = "something",
        ["type"] = "word"
    },
...
}
-- Shortened for simplicity (there are gonna be more than 3 tokens, ofc)

What does the output mean?

matches

List of exact strings (or string captures) that were matched with string.match.

raw

In most cases, the same as matches[1]. This just excludes any string captures.

type

The type of "token" each part of the string has.

astriaInight/LuauLexer