How to define two different tokens with Numeric and Floating Numbers

[abhiuser]

Hi,

I was trying to write the grammar and I can came across an issue, Where I am not able to use similar types of two or more tokens.

ex:
const NumericToken = Token(/([0-9]+)/, 'NumericToken');
const FloatingToken = Token(/([-+]?[0-9]*.?[0-9])/, 'FloatToken');

const Result = Node(All(FloatingToken),
([Name]) => ({Name}));
return Any(Result);

This results in error, if i declare FloatingToken first, then it works but then using NumericToken results in error. Can you please help me understand, if there is a way to get this working.

[dmaevsky]

There are several issues with regexes in your example.
One important thing to keep in mind with recursive descent parsers is that they are greedy. It applies to regexes as well as other rules you define in your grammar.
So you have to be careful with the order when you have rules that resemble one another or worse, one is a particular case of the other (hint: the latter won't work).
I guess what you wanted to describe is the following logic:

• 'NumericToken' is an integer
• 'FloatingToken' is a floating point number but NOT an integer
  If that's what you wanted, the following code works:

const Parser = require('rd-parse');

function Grammar(Token, All, Any, Plus, Optional, Node) {
  Token(/\s+/g, 'ignore');   // Ignore whitespace
  Token(/(,)/g, 'verbatim');
  const FloatingToken = Token(/([-+]?[0-9]*\.[0-9]+)/g, 'FloatToken');
  const IntegerToken = Token(/([0-9]+)/g, 'IntegerToken');

  const Number = Any(FloatingToken, IntegerToken);

  return Node(All(Number, ',', Number), numbers => ({ numbers }));
}

var text = '239, 56.6';

var p = new Parser(Grammar);
var ast = p.parse(text);

console.log(JSON.stringify(ast, null, 2));

Note that I have removed the optional flag '?' after the dot in the 'FloatingToken' regex so that it would not match the integers!!!
All the regexes for Tokens HAVE to be global (followed by the g flag). It won't work otherwise. I hope I mentioned it in the documentation.
Enjoy and don't forget to escape '.' in your regexes either :)

[abhiuser]

@dmaevsky - Got it, Thanks for your help :)