WIP PPI version causes Marpa to fail its tests (this blocks PPI)

Question

WIP PPI version causes Marpa to fail its tests (this blocks PPI)

wchristian opened this issue 7 years ago · 6 comments

Specifically this one:

D:\cpan\Marpa-R2-4.000000>prove -bv t\use.t
t\use.t ..
1..1
Perl token "THING" was not accepted: '5'
PPI token is PPI::Token::Number: [no file]:11:12, content="5" at t\use.t line 164.
# Looks like your test exited with 255 before it could output anything.
Dubious, test returned 255 (wstat 65280, 0xff00)

The branch with the PPI work is here: https://github.com/adamkennedy/PPI/tree/mmore

The line of Perl code that Marpa seems to disagree with PPI over is:

use v5 xyz 5;

Which i'm not even sure what the expected parse should be, given it's invalid syntax even without strict.

This is currently blocking PPI, so can you please either adapt your tests, or let me know how PPI should be parsing this?

Answer 1 · 2017-05-12T17:37:01.000Z

I'll be able to look at this later today, if that's OK -- it takes me a little bit of time to get back into these old issues. IIRC, yes, the syntax is bizarre, but Perl parses it. The test originally came from a post on the perils of parsing with Bison/yacc, which this made a good example for.

…

On Fri, May 12, 2017 at 10:26 AM, Christian Walde (Mithaldu) < ***@***.***> wrote: Specifically this one: D:\cpan\Marpa-R2-4.000000>prove -bv t\use.t t\use.t .. 1..1 Perl token "THING" was not accepted: '5' PPI token is PPI::Token::Number: [no file]:11:12, content="5" at t\use.t line 164. # Looks like your test exited with 255 before it could output anything. Dubious, test returned 255 (wstat 65280, 0xff00) The branch with the PPI work is here: https://github.com/ adamkennedy/PPI/tree/mmore The line of Perl code that Marpa seems to disagree with PPI over is: use v5 xyz 5; Which i'm not even sure what the expected parse should be, given it's invalid syntax even without strict. This is currently blocking PPI, so can you please either adapt your tests, or let me know how PPI should be parsing this? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#274>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKmoQulOUBQu-A1_mdDYbw4SNJqMN8yks5r5JZdgaJpZM4NZkOa> .

Answer 2 · 2017-05-12T19:29:04.000Z

I'll be looking at it next in about 13 hours, and the day after, so there's no great hurry, and if you can't, oh well, I'll make a Dev release :)

…

On Fri, May 12, 2017, 19:37 Jeffrey Kegler ***@***.***> wrote: I'll be able to look at this later today, if that's OK -- it takes me a little bit of time to get back into these old issues. IIRC, yes, the syntax is bizarre, but Perl parses it. The test originally came from a post on the perils of parsing with Bison/yacc, which this made a good example for. On Fri, May 12, 2017 at 10:26 AM, Christian Walde (Mithaldu) < ***@***.***> wrote: > Specifically this one: > > D:\cpan\Marpa-R2-4.000000>prove -bv t\use.t > t\use.t .. > 1..1 > Perl token "THING" was not accepted: '5' > PPI token is PPI::Token::Number: [no file]:11:12, content="5" at t\use.t line 164. > # Looks like your test exited with 255 before it could output anything. > Dubious, test returned 255 (wstat 65280, 0xff00) > > The branch with the PPI work is here: https://github.com/ > adamkennedy/PPI/tree/mmore > > The line of Perl code that Marpa seems to disagree with PPI over is: > > use v5 xyz 5; > > Which i'm not even sure what the expected parse should be, given it's > invalid syntax even without strict. > > This is currently blocking PPI, so can you please either adapt your tests, > or let me know how PPI should be parsing this? > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#274>, or mute the > thread > < https://github.com/notifications/unsubscribe-auth/AAKmoQulOUBQu-A1_mdDYbw4SNJqMN8yks5r5JZdgaJpZM4NZkOa > > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#274 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKtaxgq033N0UpQ4aX9sRaQLVm-4lNaks5r5Ji-gaJpZM4NZkOa> .

Answer 3 · 2017-05-13T00:33:35.000Z

My best guess is that Marpa::R2 has expectations about how PPI will tokenize its input stream, and these are not begin met. I made a careful study of how Perl actually parses "use" statements, and blogged what I found: http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2011/09/perl-and-parsing-9-use-and-the-ruby-slippers.html http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2011/10/perl-and-parsing-10-use-the-easier-way.html http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2011/10/perl-and-parsing-12-beating-up-on-the-use-statement.html My test suite uses Marpa::R2 to duplicate what Perl does. As a tokenizer, it uses PPI -- but only as a tokenizer -- it does not rely on PPI to figure this out. One way of looking at this is to see my t/use.t test as a carefully worked-out extension to the PPI test suite, covering in detail a very difficult special case. If PPI fails t/use.t, it might follow, that it will also fail other tests in DarkPAN, and possibly in quite mysterious ways. I've added hi-there's to show the tokenization Marpa::R2's t/use.t expects. If my guess is right, that tokenization has changed in your bleeding edge PPI, and a fix would be to restore it to the previous tokenization. My listing with the tokenization will follow shortly. On Fri, May 12, 2017 at 12:29 PM, Christian Walde (Mithaldu) < notifications@github.com> wrote:

…

I'll be looking at it next in about 13 hours, and the day after, so there's no great hurry, and if you can't, oh well, I'll make a Dev release :) On Fri, May 12, 2017, 19:37 Jeffrey Kegler ***@***.***> wrote: > I'll be able to look at this later today, if that's OK -- it takes me a > little bit of time to get back into these old issues. > > IIRC, yes, the syntax is bizarre, but Perl parses it. The test originally > came from a post on the perils of parsing with Bison/yacc, which this made > a good example for. > > > > On Fri, May 12, 2017 at 10:26 AM, Christian Walde (Mithaldu) < > ***@***.***> wrote: > > > Specifically this one: > > > > D:\cpan\Marpa-R2-4.000000>prove -bv t\use.t > > t\use.t .. > > 1..1 > > Perl token "THING" was not accepted: '5' > > PPI token is PPI::Token::Number: [no file]:11:12, content="5" at t\use.t > line 164. > > # Looks like your test exited with 255 before it could output anything. > > Dubious, test returned 255 (wstat 65280, 0xff00) > > > > The branch with the PPI work is here: https://github.com/ > > adamkennedy/PPI/tree/mmore > > > > The line of Perl code that Marpa seems to disagree with PPI over is: > > > > use v5 xyz 5; > > > > Which i'm not even sure what the expected parse should be, given it's > > invalid syntax even without strict. > > > > This is currently blocking PPI, so can you please either adapt your > tests, > > or let me know how PPI should be parsing this? > > > > — > > You are receiving this because you are subscribed to this thread. > > Reply to this email directly, view it on GitHub > > <#274>, or mute the > > thread > > < > https://github.com/notifications/unsubscribe-auth/AAKmoQulOUBQu-A1_ mdDYbw4SNJqMN8yks5r5JZdgaJpZM4NZkOa > > > > . > > > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#274 (comment)- 301139536>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ AAKtaxgq033N0UpQ4aX9sRaQLVm-4lNaks5r5Ji-gaJpZM4NZkOa> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#274 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKmoTR7feW5DvSMU57pS15S2Ru7Z48wks5r5LMBgaJpZM4NZkOa> .

Answer 4 · 2017-05-13T00:39:54.000Z

1..1
PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Number::Version; v5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Number::Float; 5.1
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Number::Version; v5
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Number::Float; 5.1
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number::Version; v5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number::Float; 5.1
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Number::Version; v5
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Number::Float; 5.1
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number::Version; v5
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number::Float; 5.1
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Number::Version; v5
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Operator; ,
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Operator; ,
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Number::Float; 5.1
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Operator; ,
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number::Version; v5
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Operator; ,
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Operator; ,
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number::Float; 5.1
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Operator; ,
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace; 

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number::Float; 5.1
PPI::Token::Whitespace;  
PPI::Token::Symbol; @a
PPI::Token::Structure; ;
PPI::Token::Whitespace;

Answer 5 · 2017-05-13T00:49:42.000Z

Here's the portion of the above that I think is relevant to the issue, if you were correct in singling out the use v5 xyz 5; statement as the point of failure:

PPI::Token::Word; use
PPI::Token::Whitespace;  
PPI::Token::Number::Version; v5
PPI::Token::Whitespace;  
PPI::Token::Word; xyz
PPI::Token::Whitespace;  
PPI::Token::Number; 5
PPI::Token::Structure; ;
PPI::Token::Whitespace;

The error message indicates that the PPI::Token::Number token for the string 5 is not expected at the point where it occurs. This probably means that a false expectation was set up by a change in the token stream earlier. Because of the structure of the grammar, this false expectation is likelier to occur in the portion of the tokenization that I have quoted in this message. In particular, what are the tokens in the bleeding-edge PPI for v5 and xyz?

If they're OK, you might look at the tokenization in prior statements. Bleeding-edge and stable PPI tokenization should match, or else Marpa::R2's t/use.t is going to get out-of-sync.

I hope this is helpful, jeffrey

Answer 6 · 2017-05-13T10:21:33.000Z

Thanks a lot for the information, i've written a test and it's highlighting that changes did indeed break some things. I'll be fixing that on my end to conform to the correct expectations. :)

D:\cpan\PPI>prove -l t\marpa.t
t\marpa.t ..
#   Failed test '"use v5 ;"'
t\marpa.t .. 1/23 #   at t\marpa.t line 11.
#     Structures begin differing at:
#          $got->[4] = 'PPI::Token::Word'
#     $expected->[4] = 'PPI::Token::Number::Version'
# >>> use v5 ; -- "use v5 ;"
# GOT: [  'PPI::Statement::Include',  'use v5 ;',  'PPI::Token::Word',  'use',  'PPI::Token::Word',  'v5',  'PPI::Token::Structure',  ';']
# EXP: [  'PPI::Statement::Include',  'use v5 ;',  'PPI::Token::Word',  'use',  'PPI::Token::Number::Version',  'v5',  'PPI::Token::Structure',  ';']

#   Failed test '"use v5 xyz () ;"'
#   at t\marpa.t line 54.
#     Structures begin differing at:
#          $got->[4] = 'PPI::Token::Word'
#     $expected->[4] = 'PPI::Token::Number::Version'
# >>> use v5 xyz () ; -- "use v5 xyz () ;"
# GOT: [  'PPI::Statement::Include',  'use v5 xyz () ;',  'PPI::Token::Word',  'use',  'PPI::Token::Word',  'v5',  'PPI::Token::Word',  'xyz',  'PPI::Structure::List',  '()',  'PPI::Token::Structure',  '(',  'PPI::Token::Structure',  ')',  'PPI::Token::Structure',  ';']
# EXP: [  'PPI::Statement::Include',  'use v5 xyz () ;',  'PPI::Token::Word',  'use',  'PPI::Token::Number::Version',  'v5',  'PPI::Token::Word',  'xyz',  'PPI::Structure::List',  '()',  'PPI::Token::Structure',  '(',  'PPI::Token::Structure',  ')',  'PPI::Token::Structure',  ';']

#   Failed test '"use v5 xyz 5 ;"'
#   at t\marpa.t line 138.
#     Structures begin differing at:
#          $got->[4] = 'PPI::Token::Word'
#     $expected->[4] = 'PPI::Token::Number::Version'
# >>> use v5 xyz 5 ; -- "use v5 xyz 5 ;"
# GOT: [  'PPI::Statement::Include',  'use v5 xyz 5 ;',  'PPI::Token::Word',  'use',  'PPI::Token::Word',  'v5',  'PPI::Token::Word',  'xyz',  'PPI::Token::Number',  '5',  'PPI::Token::Structure',  ';']
# EXP: [  'PPI::Statement::Include',  'use v5 xyz 5 ;',  'PPI::Token::Word',  'use',  'PPI::Token::Number::Version',  'v5',  'PPI::Token::Word',  'xyz',  'PPI::Token::Number',  '5',  'PPI::Token::Structure',  ';']

#   Failed test '"use v5 xyz 5,5 ;"'
#   at t\marpa.t line 210.
#     Structures begin differing at:
#          $got->[4] = 'PPI::Token::Word'
#     $expected->[4] = 'PPI::Token::Number::Version'
# >>> use v5 xyz 5,5 ; -- "use v5 xyz 5,5 ;"
# GOT: [  'PPI::Statement::Include',  'use v5 xyz 5,5 ;',  'PPI::Token::Word',  'use',  'PPI::Token::Word',  'v5',  'PPI::Token::Word',  'xyz',  'PPI::Token::Number',  '5',  'PPI::Token::Operator',  ',',  'PPI::Token::Number',  '5',  'PPI::Token::Structure',  ';']
# EXP: [  'PPI::Statement::Include',  'use v5 xyz 5,5 ;',  'PPI::Token::Word',  'use',  'PPI::Token::Number::Version',  'v5',  'PPI::Token::Word',  'xyz',  'PPI::Token::Number',  '5',  'PPI::Token::Operator',  ',',  'PPI::Token::Number',  '5',  'PPI::Token::Structure',  ';']
# Looks like you failed 4 tests of 23.
t\marpa.t .. Dubious, test returned 4 (wstat 1024, 0x400)
Failed 4/23 subtests

Test Summary Report
-------------------
t\marpa.t (Wstat: 1024 Tests: 23 Failed: 4)
  Failed tests:  1, 5, 11, 17
  Non-zero exit status: 4
Files=1, Tests=23,  1 wallclock secs ( 0.06 usr +  0.03 sys =  0.09 CPU)
Result: FAIL

D:\cpan\PPI>