Why is this a single regex?

[RoboterHund]

Just wondering.
It works like a charm, but I had to split the regex at the |s.

[lydell]

What's your use case?

[RoboterHund]

My problem is that before finding this package,
I required a tokenizer framework that expected one regex per token type. My bad.

I now realize that the proper way of using this package is something like:

var jsTokens = require ('js-tokens');
var jsString = 'tokenize this';
var match, i, token, type;
var namesLength = jsTokens.names.length;
while ((match = jsTokens.exec (jsString))) {
    for (i = 1; i <= namesLength; i++) {
        if (match [i] !== undefined) {
            token = match [0];
            type = jsTokens.names [i - 1];
            // use token, type
        }
    }
}

[lydell]

Yeah, that’s one way of using it. The only things that I’ve used it for is var tokens = jsString.match(jsTokens) (to get an array of all tokens, without the types, though) and jsString.replace(jsTokens, function(token, whitespace, comment, ...)) { if (whitespace) {return ...} else if (comment) {return ...} else if (...) ... }. But we could change things if you have any suggestions.

[lydell]

Closing due to inactivity. Feel free to reopen.

[lydell]

Check out v0.3.0 and see what you think (if you’re still interested).