Confusion about length of libmarpa token

Some confusion about the length of a libmarpa token may have propagated upward. The libmarpa macros involved are marpa_v_es_id() and marpa_v_token_start_es_id(). Affected might be the g1length descriptor and MARPA_OP_PUSH_G1_LENGTH in R2.xs. In particular, line

Marpa--R2/cpan/xs/R2.xs

Line 1806 in 2ef61bd

length = end_earley_set - start_earley_set + 1;

seems to be wrong.

The solution at this point is probably not to fix R2.xs, which will break everybody depending on the misfeature, but to document that, "for historical reasons", g1length is not what you would expect.

Jeffrey,

Can you explain the confusion in itself and would be the fix if you were to change to code ?

Thanks, JD.

Also in error is this line:

Marpa--R2/cpan/xs/R2.xs

Line 1798 in 2ef61bd

length = end_earley_set - start_earley_set + 1;

@jddurand If I were to change the code, everyone using the g1length action array descriptor for tokens and rules (but not nulling symbols) would see a value that is lesser by one. This would make more sense but would break everything relying on the current behavior.

Fixed as of 317b920. I have added g1len which is equal to the correct length, and deprecated g1length.

Absent feedback to the contrary, I will close this issue after a couple of days.

Per #280 (comment), I am closing this issue.