Failed assertions related to backtracking

[mayshukla]

Failed assertions related to backtracking

[mayshukla]

The parser fails the following assertion in lib/Lex/PPCaching.cpp:

clang-6.0: /home/ryan/Files/variability-research/llvm/tools/clang/lib/Lex/PPCaching.cpp:141: void clang::Preprocessor::AnnotatePreviousCachedTokens(const clang::Token &): Assertion `(BacktrackPositions.empty() || BacktrackPositions.back() <= i) && "The backtrack pos points inside the annotated tokens!"' failed.

This assertion is failed whenever there are tokens that get annotated by the parser that are surrounded by an ifdef outside of a function body. For example:

#ifdef A
decltype(1) i;
#endif

And:

namespace foo {
  typedef int bar;
}
#ifdef A
foo::bar i;
#endif

See clang documentation for description of annotation tokens.

[mayshukla]

Update: the issue is caused by the method void Parser::SplitOrConsume. Although the method StmtResult Parser::ParseStatementOrDeclaration checks if the token is a split token when deciding when to split, the method void Parser::SplitOrConsume only looks at the presence condition of the next token when deciding when to split. I think this is because void Parser::SplitOrConsume was written before the split token was being used as an indicator to split. SplitOrConsume looks at each token and pushes a backtrack position to the stack if the presence condition of the next token indicates that a split is neccessary. The issue here is that SplitOrConsume is not just setting backtrack positions at split tokens, but also at foo, :: and bar. These tokens get "annotated" by the parser. That is, the parser replaces these tokens with more semantically specific tokens when it figures out that foo is a namespace and bar is a type. The parser may replace multiple tokens with a single token. You can't have backtrack positions that point inside annotated tokens because these tokens get replaced by the parser and there might not even be a token at that position any more.

[mayshukla]

Update: to fix the issue, I made SplitOrConsume also check if the token is a split token. If it isn't a split token, the method does not enable a backtrack position at that point.