A JS implementation of a tokenizer to split up source code based on tokens
npm install code-tokenizer --save
Tokenizer = require 'code-tokenizer'
tokenizer = new Tokenizer()
tokens = tokenize 'print "Josh"'
# ['print']
tokens = tokenize 'path/to/file.html', 'file'
# ['<div>', 'id', 'class', '</div>']
This is based on the Github Linguist project. This node.js implementation strips any data strings or comments from the data and returns an array of language symbols.
You can use this to do sanity checking of source code, parsing, detection, etc. This is a general purpose library and I'm always looking to hear how you're using it. Happy Coding!
All tests are in the test/tokenizer.coffee file. It tests a variety of cases from different language types. To run the test for yourself, just do npm test.
You can contact me @shamoons or read up on my blog: http://shamoon.me