/pyscanner

Simple Lexer based on regular expressions to transform a source code file into tokens

Primary LanguagePythonGNU General Public License v2.0GPL-2.0

PyScanner

Simple Lexer used to convert a source code file into tokens Currently using Javascript as an example language (language definition is in lang.py)

Usage: python pyscanner.py path/to/file

TODO:

  • Encapsulate into a function or class
  • Return (lexem, token) list
  • Load language lexic from a text file
  • Documentation