Dynamically referencing Parsers via Decorators for TokenizerBuilder

[martinkoslof]

Hello there,

I have created a fully functional tokenizer and parser solution via SuperPower. I love the library and I've learned quite a bit. However, I am stuck on one optimization and before I throw in the towel, I figured I would ask if the creator or anyone else had any ideas.

Right now we use the fluent interface of the TokenizerBuilder and it's .Match(TextParser) builder methods. We have about 30 Match Recoginzers at the moment, all static method or properties, for example:

.Match(Character.EqualTo(','), ...)
.Match(Span.EqualTo("and"), ...)
.Match(ScalarParsers.QuotedGuid, ...)

So as you can see, we use OOB SuperPower Character and Span classes, as well as some custom static parsers within a custom static class as properties. All this works but now, I'm trying to use reflection and some decorators to build the Tokenizer. I am using attributes and I am passing the type and method or property name, array of values, and an ordinal for position (since Recognizers are processed top to bottom). For example, in my Token enum class:

        /// <summary>
        /// Token which represents an closing right side bracket - ]
        /// </summary>
        [Description("]")]
        [Token]
        [TokenScalarMethodRecognizer("EqualTo", 4, new object[] { ']' }, typeof(char), typeof(Character), false)]
        RBracket,

This implies, for this Token, use the Character.EqualTo(']') static method as the TextParser recognizer and it does not require delimiters. It should appear as the 4th recognizer in the top/down order.

However, looking at the source code, TextParser is a delegate receiving TextSpan input. I can not figure out how to wire this up for either Delegate.DynamicInvoke() or using Expression.Lamda. Given I am using Attributes on my Tokens, these are not generic and I can not pass func() or any form of delegate from there. I am more or less trying to do the following:

    var builder = new TokenizerBuilder<MyQueryToken>().Ignore(Span.WhiteSpace);
    var builderType = typeof(TokenizerBuilder<MyQueryToken>);
    var methodInfo = builderType.GetMethod("Match");

   foreach((Token, Attr) in Ordered)
   {
        var parserMethodInfo = Attr.StaticClass.GetMethod(Attr.MethodName);  //such as Character or Span
        var delegate = parserMethodInfo.CreateDelegate(textParserType, null, parserMethodInfo)  //delegate pointer with TextParser<type>

        //given a delegate or Expression.Lamdba invoke or pass the TextParser (just crude example)
        builder = methodInfo.invoke(builder,  new object[]{ delegate.DynamicInvoke(Attr.parameters), Token, 
        Attr.RequiresDelimiters });
  }

Everything I have tried has failed so far. This is just some air code for demonstration. I have tried a few other ways, including Expression trees, and between delegate to method binding errors or non generic/generic type runtime errors, I'm not having any luck. Is this even possible with the current structure of the SuperPower library? Is it possible to dynamically reference Parsers which are the TextParser delegate pointer?

[martinkoslof]

Meh. I found my mistake, I was missing the MakeGenericMethod on the actual Match() or the reflective methodInfo of the TokenizerBuilder. It is working now. Closing this.

[nblumhardt]

Thanks for following up, glad you found it! 👍