Crash: SyntaxError during parsing of scribble file

Question

Crash: SyntaxError during parsing of scribble file

NeverFearTomorrow opened this issue 2 years ago · 8 comments

I am trying to get scribble to instrument my smart contract, but after fixing a number of issues with my scribble assertions, I get the following error:

(base) 11:24:15 ~/nifty/web/sc/token$ ~/scribble/dist/bin/scribble.js --arm -m files contracts/Thing.sol 
Found 70 annotations in 1 different files.
/home/un/scribble/node_modules/solc-typed-ast/dist/types/typeStrings/typeString_parser.js:631
        return new SyntaxError(SyntaxError.buildMessage(expected1, found), expected1, found, location1);
               ^

SyntaxError: Expected "[", "calldata", "constant", "external", "internal", "memory", "nonpayable", "payable", "pure", "returns", "storage", "view", [\n,\r, , ], end of input, or whitespace but "p" found.
    at peg$buildStructuredError (/home/un/scribble/node_modules/solc-typed-ast/dist/types/typeStrings/typeString_parser.js:631:16)
    at peg$parse (/home/un/scribble/node_modules/solc-typed-ast/dist/types/typeStrings/typeString_parser.js:5237:15)
    at getNodeType (/home/un/scribble/node_modules/solc-typed-ast/dist/types/typeStrings/typeString_parser.js:56:30)
    at findExternalCalls (/home/un/scribble/dist/instrumenter/instrument.js:67:65)
    at replaceExternalCallSites (/home/un/scribble/dist/instrumenter/instrument.js:437:28)
    at instrumentContract (/home/un/scribble/dist/instrumenter/instrument.js:293:9)
    at instrumentFiles (/home/un/scribble/dist/bin/scribble.js:236:49)
    at /home/un/scribble/dist/bin/scribble.js:695:13
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  expected: [
    { type: 'other', description: 'whitespace' },
    {
      type: 'class',
      parts: [ '\n', '\r', ' ', ' ' ],
      inverted: false,
      ignoreCase: false
    },
    { type: 'literal', text: 'external', ignoreCase: false },
    { type: 'literal', text: 'internal', ignoreCase: false },
    { type: 'literal', text: 'pure', ignoreCase: false },
    { type: 'literal', text: 'view', ignoreCase: false },
    { type: 'literal', text: 'payable', ignoreCase: false },
    { type: 'literal', text: 'nonpayable', ignoreCase: false },
    { type: 'literal', text: 'constant', ignoreCase: false },
    { type: 'literal', text: 'returns', ignoreCase: false },
    { type: 'literal', text: '[', ignoreCase: false },
    { type: 'literal', text: 'memory', ignoreCase: false },
    { type: 'literal', text: 'storage', ignoreCase: false },
    { type: 'literal', text: 'calldata', ignoreCase: false },
    { type: 'end' }
  ],
  found: 'p',
  location: {
    source: undefined,
    start: { offset: 32, line: 1, column: 33 },
    end: { offset: 33, line: 1, column: 34 }
  }
}

I don't know how to cut a minimal example, because there's no line number into my actual code, and i don't know what part of the code is appearing as "p" to the syntax parser. Does anyone have any advice or knowledge about this type of error? If I can get some hints, maybe I can cut a minimal example.

Of note:

npx hardhat compile works fine
This issue repros with both the npm version of scribble, and built-from-source scribble that i downloaded and built a few days ago
I had already run scribble --arm a few times, and each time it had complained about an issue with one of my scribble assertions, and I would fix the issue and run again. So it was odd to have this kind of fundamental crash after the main passes appear to have successfully completed. Note that the first line of output is Found 70 annotations in 1 different files.

Answer 1 · 2022-09-23T05:42:50.000Z

Hello there. From stacktrace I can see that error is related to parsing typeStrings when Scribble is trying to process one of the call sites (instrument external call, probably). TypeStrings are part of expression AST nodes, that are popular in the source files.

I would suggest to modify source a bit to figure out the problematic node. Here is what you can do:

Find this file and line: https://github.com/ConsenSys/scribble/blob/11fcc1aad5d3398311fd0efddce9f914c12cb714/src/instrumenter/instrument.ts#L132
Add console.log(call.vExpression.print(100)); just before mentioned line (use line 131 for example).
Execute npm run transpile in command line or terminal.
Run Scribble on problematic source sample.
- if you will not see additional output, then you probably run wrong Scribble executable. Then try also running npm link in command line or terminal.

It should print additional information in output. The last unindented AST node before error would probably the reason of the issue. Pay attention to typeString properties. It would be nice is you would be able to share the output.

Thanks for your efforts.

Answer 2 · 2022-09-23T20:20:00.000Z

Ok, thanks for the info! i'll try that at some point and see if i can locate the issue.

Answer 3 · 2022-09-28T19:14:55.000Z

Here's the output: https://pastebin.com/8GLbgqqS

Here's the code for _getTypeRegistryKey, which seems to be the offending function?

343   function _getTypeRegistryKey(address contractAddr, uint256 tokenId) private pure returns (bytes memory) {
344     return abi.encodePacked(contractAddr, tokenId);
345   }

In the paste, the typestring is:
typeString: "function (address,uint256) pure private returns (bytes)"
I see that the other typestrings don't include modifiers pure private ..., that must be where the unexpected p is coming from.
But I don't know why the parser slurped up the modifiers in this case, but didn't slurp up the modifiers for the other functions

Answer 4 · 2022-09-29T05:10:49.000Z

The issue seems to be related to p in private. Type string parser in solc-typed-ast (a base dependency) have following grammar:

https://github.com/ConsenSys/solc-typed-ast/blob/195da24f80e1c8dac8f3edce8d32c239f0cc910c/src/types/typeStrings/typeString_grammar.pegjs#L358-L399

Where FunctionVisibility does not respect private keyword:

https://github.com/ConsenSys/solc-typed-ast/blob/master/src/types/typeStrings/typeString_grammar.pegjs#L358-L360

I will try to reproduce and consider making a fix. We will inform when changes will arrive. Thank you very much for assistance.

Answer 5 · 2022-09-29T21:55:03.000Z

I see. Should I wait for the solc-typed-ast dep to be merged, and update it? Or is there a way to patch in this change to solc-typed-ast in my local system so that I can get it working locally? I have built scribble from source, but I can't remember off the top of my head how i installed solc-typed-ast

Answer 6 · 2022-09-30T04:47:54.000Z

Well... You can either wait for changes to arrive or use instructions below to build on your own:

Clone https://github.com/ConsenSys/solc-typed-ast somewhere nearby Scribble repository.
Go to solc-typed-ast repo directory.
Switch to branch fix-typestring-parser:
```
git checkout fix-typestring-parser
```
Run npm install (and optionally npm link if you want to link it globally and use as a CLI tool) in solc-typed-ast repo directory. It would build parser with current fixes, that should help.
Go to Scribble repo directory.
Change package.json to refer to local solc-typed-ast repo installation as dependency: https://github.com/ConsenSys/scribble/blob/76b964cfe849ef1da56975e6680bab98ab6fe7e8/package.json#L21 (it should refer to local solc-typed-ast repo path)
Run npm install and npm link in Scribble repo directory.
Try to use it on source file.

That should be enough. In case if you decide to rebuild things on your own, please consider informing us if current patch solved your issue.

Regards.

Answer 7 · 2022-10-01T05:03:58.000Z

Great! That seems to have run successfully now. Thanks for the help!

Answer 8 · 2022-10-13T05:19:33.000Z

Should be fixed in scribble version 0.6.15