support for external lexers and partial input

[mgood7123]

could mpc be used with external lexers (as in an actual proper lexer such as gnu lex/flex), where tokes are fed to the parser, optionally accepting a token type, if so is lookahead supported and if so how far can it look ahead and can it backtrack if a lookahead does not end up matching, and does it support a mpca_lang or similar

for example:

while(token = get_next_token()) parse(token.token, token.type);

given the rule mpc_string("abc"); and given get_next_token() returns the following

token.token = "a"
token.type = TOK_CHAR
token.token = "bc"
token.type = TOK_STR

could this partial input match the rule mpc_string("abc"); without causing the rule to terminate due to parse reading EOF in its input from the string "a" due to "a" by itself not matching the rule mpc_string("abc"); resulting in parse() to error

[orangeduck]

You could maybe do this, although you would need to make sure the parsers you use take your custom token type as input and produce the AST as output. Also the back-tracking wouldn't be able to interact with the token parsing I.E. putting tokens back into the stream but honestly at this point it seems like it would be better to write the parsing function by hand.