Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.
Please consider following this project's author, Jon Schlinkert, and consider starring the project to show your ❤️ and support.
Install with npm:
$ npm install --save snapdragon-scannerThis is a simple Lexical Scanner that takes an object of regex patterns, and uses those patterns to process an input string into an array of tokens.
What is the difference between this and snapdragon-lexer?
snapdragon-lexer uses registered handler functions to capture and handle tokens, snapdragon-scanner simply iterates over an object of regular expression patterns to create tokens. You can think of snapdragon-scanner as the "lite" version of snapdragon-lexer.
const Scanner = require('snapdragon-scanner');Create a new Scanner with the given str and optional rules.
Params
input{String}: Input string to scan.options{Object}: (optional) Pass an object of regex patterns onoptions.rules, or use.addRules()or.addRule()after instantiating.
Example
const Scanner = require('snapdragon-scanner');
const scanner = new Scanner('var foo = "bar";', {
rules: {
space: /^ +/,
tab: /^\t+/,
newline: /^\n+/,
text: /^\w+/,
equal: /^=/,
quote: /^["']/,
semi: /^;/,
dot: /^\./
}
});Add a rule to the scanner.
Params
rule{String}match{RegExp}: Match array fromRegExp.exec().
Example
console.log(scanner.token('text', ['foo']);
//=> { rule: 'text', value: 'foo', match: [foo] };Add a rule to the scanner.
Params
rule{String}regex{RegExp}: Regular expression to use when scanning.
Example
scanner.addRule(rule, regex);
// example
scanner.addRule('text', /^\w+/);Add an object of rules to the scanner.
Params
rules{Object}
Example
scanner.addRules({
text: /^\w+/,
slash: /^\//,
dot: /^\./
});Attempts to match scanner.string with the given regex. Also validates the regex to ensure that it starts with ^ since matching should always be against the beginning of the string, and throws if the regex matches an empty string, to avoid catastrophic backtracking.
Params
regex{RegExp}: (required)returns{Array|null}: Returns the match array or null fromRegExp.exec.
Example
const scanner = new Scanner('foo/bar', { text: /^\w+/ });
const match = scanner.match(scanner.rules.get('text'));
console.log(match);
//=> [ 'foo', index: 0, input: 'foo/bar', groups: undefined ]Remove the given length of substring from scanner.string.
Params
len{Number}value{String}: Optionally pass the value being consumed for minor performance improvement.returns{String}: Returns the consumed value
Example
scanner.consume(1);
scanner.consume(1, '*');Push a token onto the scanner.queue array.
Params
token{object}returns{Object}: Returns the token.
Example
console.log(scanner.queue.length); // 0
scanner.enqueue({ rule: 'foo' });
console.log(scanner.queue.length); // 1Shift a token from scanner.queue.
returns{Object}: Returns the first token in thescanner.queue.
Example
console.log(scanner.queue.length); // 0
scanner.enqueue({ rule: 'foo' });
console.log(scanner.queue.length); // 1
scanner.dequeue();
console.log(scanner.queue.length); // 0Iterates over the registered regex patterns until a match is found, then returns a token from the match and regex rule.
returns{Object}: Returns a token withrule,valueandmatchproperties.
Example
const token = scanner.advance();
console.log(token) // { rule: 'text', value: 'foo' }Lookahead n tokens and return the last token. Pushes any intermediate tokens onto scanner.tokens. To lookahead a single token, use .peek().
Params
n{number}returns{Object}
Example
const token = scanner.lookahead(2);Returns a token representing the next match, but without consuming the matched substring (e.g. the cursor position is not advanced).
returns{Object|undefined}: Returns a token, or undefined if no match was found.
Example
const token = scanner.peek();Returns a token representing the next match, but without consuming the matched substring (e.g. the cursor position is not advanced).
returns{Object|undefined}: Returns a token, or undefined if no match was found.
Example
const token = scanner.peek();Returns the next token and advances the cursor position.
returns{Object|undefined}: Returns a token, or undefined if no match was found.
Example
const token = scanner.scan();Scan until the given fn does not return true.
Params
fn{Function}: Must return true to continue scanning.returns{Array}: Returns an array if scanned tokens.
Example
scanner.scanWhile(tok => tok.rule !== 'space');Returns true if the scanner has not consumed any of the input string.
returns{Boolean}
Returns true if scanner.string and scanner.queue are empty.
returns{Boolean}
Scanner tokens are plain JavaScript objects with the following properties:
{
type: String;
value: String
match: Array
}type{String} - The name of the regex that matched the substring.value{String} - The substring that was captured by the regex.match{Array} - The match array from RegExp.exec()
See the changelog.
Contributing
Pull requests and stars are always welcome. For bugs and feature requests, please create an issue.
Running Tests
Running and reviewing unit tests is a great way to get familiarized with a library and its API. You can install dependencies and run tests with the following command:
$ npm install && npm testBuilding docs
(This project's readme.md is generated by verb, please don't edit the readme directly. Any changes to the readme must be made in the .verb.md readme template.)
To generate the readme, run the following command:
$ npm install -g verbose/verb#dev verb-generate-readme && verbYou might also be interested in these projects:
- snapdragon-lexer: Converts a string into an array of tokens, with useful methods for looking ahead and… more | homepage
- snapdragon-node: Snapdragon utility for creating a new AST node in custom code, such as plugins. | homepage
- snapdragon-token: Create a snapdragon token. Used by the snapdragon lexer, but can also be used by… more | homepage
Jon Schlinkert
Copyright © 2018, Jon Schlinkert. Released under the MIT License.
This file was generated by verb-generate-readme, v0.8.0, on November 19, 2018.