Tokenizer is a simple lexical analyzer written in C. It analyzes the input file and generates two output files: output.txt and symbolTable.txt.
The Tokenizer supports recognizing single-line and multi-line comments, common operators, common identifiers, decimals (without scientific notation), and integers.
To use the Tokenizer, follow these steps:
- Compile the program using the make command:
make
- Run the Tokenizer by providing an input file:
./bin/main <filename>
Replace with the name of your input file. The Tokenizer will analyze the contents of the file and generate the output files output.txt and symbolTable.txt.
You can also directly use the make run command to process the "input.txt" file in the current directory (if it exists).
- Clean up the directory by removing the generated files using the make clean command:
make clean
This command will remove the binary files.
The file compiles successfully under clang version 17. If you are using gcc, you may need to modify the Makefile.