facebook/fbthrift

OSS build failing: parse/thrifty.yy:65:13: error: ‘struct apache::thrift::compiler::yy::parser::symbol_type’ has no member named ‘token’

donsbot opened this issue · 11 comments

donsbot commented

Introduced in 0ed89d8

vitaut commented

Which OSS build are you referring to? Thrift OSS builds are passing: https://github.com/facebook/fbthrift/actions.

donsbot commented

Maybe we need to regenerate something?

On Linux, fresh checkout:

dons@linode1:~/fbthrift$ uname -msr
Linux 5.10.0-10-amd64 x86_64
dons@linode1:~/fbthrift$ cc --version
cc (Debian 10.2.1-6) 10.2.1 20210110
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

dons@linode1:~/fbthrift$ c++ --version
c++ (Debian 10.2.1-6) 10.2.1 20210110
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Built with:

./build/fbcode_builder/getdeps.py build --src-dir . --no-deps fbthrift

fails with:

2/120] Building CXX object thrift/compiler/CMakeFiles/compiler_base.dir/thrifty.cc.o
FAILED: thrift/compiler/CMakeFiles/compiler_base.dir/thrifty.cc.o
/usr/bin/c++ -DBOOST_ALL_NO_LIB -DBOOST_ATOMIC_DYN_LINK -DBOOST_CONTEXT_DYN_LINK -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_PROGRAM_OPTIONS_DYN_LINK -DBOOST_REGEX_DYN_LINK -DBOOST_SYSTEM_DYN_LINK -DBOOST_THREAD_DYN_LINK -DLOCATION_HH=\"/tmp/fbcode_builder_getdeps-ZhomeZdonsZfbthriftZbuildZfbcode_builder/build/fbthrift/thrift/compiler/location.hh\" -DTHRIFTY_HH=\"/tmp/fbcode_builder_getdeps-ZhomeZdonsZfbthriftZbuildZfbcode_builder/build/fbthrift/thrift/compiler/thrifty.hh\" -DTHRIFT_HAVE_LIBSNAPPY=0 -I/home/dons/fbthrift -I. -O2 -g -DNDEBUG -std=c++17 -MD -MT thrift/compiler/CMakeFiles/compiler_base.dir/thrifty.cc.o -MF thrift/compiler/CMakeFiles/compiler_base.dir/thrifty.cc.o.d -o thrift/compiler/CMakeFiles/compiler_base.dir/thrifty.cc.o -c thrift/compiler/thrifty.cc
parse/thrifty.yy: In function ‘apache::thrift::compiler::yy::parser::symbol_type apache::thrift::compiler::{anonymous}::parse_lex(apache::thrift::compiler::parsing_driver&, YYSTYPE*, YYLTYPE*)’:
parse/thrifty.yy:65:13: error: ‘struct apache::thrift::compiler::yy::parser::symbol_type’ has no member named ‘token’
[4/120] Building CXX object thrift/compiler/CMakeFiles/compiler_base.dir/common.cc.o
ninja: build stopped: subcommand failed.
Command '['/usr/bin/cmake', '--build', '/tmp/fbcode_builder_getdeps-ZhomeZdonsZfbthriftZbuildZfbcode_builder/build/fbthrift', '--target', 'install', '--config', 'Release', '-j', '8']' returned non-zero exit status 1.
!! Failed

And on the Glean CI which does the same get_deps.py build: https://github.com/donsbot/Glean/runs/5763763378?check_suite_focus=true

Command '['/usr/bin/cmake', '--build', '/tmp/fbcode_builder_getdeps-Z__wZGleanZGleanZhsthriftZbuildZfbcode_builder-root/build/fbthrift', '--target', 'install', '--config', 'Release', '-j', '4']' returned non-zero exit status 1.
[49/282] Building CXX object thrift/compiler/CMakeFiles/compiler_base.dir/thrifty.cc.o
!! Failed
FAILED: thrift/compiler/CMakeFiles/compiler_base.dir/thrifty.cc.o 
/usr/bin/c++  -DBOOST_ALL_NO_LIB -DBOOST_ATOMIC_DYN_LINK -DBOOST_CONTEXT_DYN_LINK -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_PROGRAM_OPTIONS_DYN_LINK -DBOOST_REGEX_DYN_LINK -DBOOST_SYSTEM_DYN_LINK -DBOOST_THREAD_DYN_LINK -DLOCATION_HH=\"/tmp/fbcode_builder_getdeps-Z__wZGleanZGleanZhsthriftZbuildZfbcode_builder-root/build/fbthrift/thrift/compiler/location.hh\" -DTHRIFTY_HH=\"/tmp/fbcode_builder_getdeps-Z__wZGleanZGleanZhsthriftZbuildZfbcode_builder-root/build/fbthrift/thrift/compiler/thrifty.hh\" -DTHRIFT_HAVE_LIBSNAPPY=0 -Dcompiler_base_EXPORTS -I/tmp/fbcode_builder_getdeps-Z__wZGleanZGleanZhsthriftZbuildZfbcode_builder-root/repos/github.com-facebook-fbthrift.git -I. -march=corei7 -O2 -g -DNDEBUG -fPIC   -std=c++17 -MD -MT thrift/compiler/CMakeFiles/compiler_base.dir/thrifty.cc.o -MF thrift/compiler/CMakeFiles/compiler_base.dir/thrifty.cc.o.d -o thrift/compiler/CMakeFiles/compiler_base.dir/thrifty.cc.o -c thrift/compiler/thrifty.cc
parse/thrifty.yy: In function ‘apache::thrift::compiler::yy::parser::symbol_type apache::thrift::compiler::{anonymous}::parse_lex(apache::thrift::compiler::parsing_driver&, YYSTYPE*, YYLTYPE*)’:
vitaut commented

Which bison version do you use?

donsbot commented

Vanilla bison from binutils. Debian 10 Linode instance. Glean is using bison from ubuntu focal (20.00), https://packages.ubuntu.com/focal/bison - 3.5.1

I am using Debian 10, bison 3.7.5

dons@linode1:~/fbthrift$ bison --version
bison (GNU Bison) 3.7.5
Written by Robert Corbett and Richard Stallman.

Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
vitaut commented

It seems to me like some incompatibility between bison versions. For example in 3.1 symbol_type defined in thrifty.yy.h inherits from by_type which contains:

      /// The token.
      token_type token () const;

Could you post the generated thrifty.yy.h here?

donsbot commented

Ok. Uninstalled system bison (3.7.5). Installed bison 3.1 from source.

$ bison --version
bison (GNU Bison) 3.1
Written by Robert Corbett and Richard Stallman.
...
./build/fbcode_builder/getdeps.py build --src-dir . --no-deps fbthrift
...
-- Found BISON: /home/dons/bin/bison (found suitable version "3.1", minimum required is "3.0.4")

And it builds ok. So its a difference between these early 3.1.x bison and later versions.

With bison 3.1: https://gist.github.com/donsbot/d10c10af6a800368ecbb7b0e79f6e9a9

./build/fbthrift/thrift/compiler/thrifty.hh

      /// The token.
      token_type token () const;

With bison 3.7.5 (system version), we get https://gist.github.com/donsbot/f340dbf1edd21fe323188a148b53eb34

  • 👍1
donsbot commented

Any chance we can remove the dependency on bison 3.1? I think it will be a bit of a pain to have to downgrade bison in open source. The CI "linux" workflow is using ubuntu-18.04 , which ends of life in 12 months. See eg regular Glean CI on Ubuntu/focal now down. https://github.com/facebookincubator/Glean/runs/5766926047?check_suite_focus=true

It might make sense to start building fbthrift on ubuntu-20 to avoid these little things (also means newer gcc and clang).

vitaut commented

Looking into it.

vitaut commented

I have a tentative fix that replaces yy::parser::symbol_type parse_lex with:

const yy::parser::symbol_type error = yy::parser::make_tok_error(yy::location());

yy::parser::symbol_type parse_lex(parsing_driver& driver, YYSTYPE*, YYLTYPE*) {
  auto token = driver.get_lexer().get_next_token();
  if (token.type_get() == error.type_get()) {
    driver.end_parsing();
  }
  return token;
}

@donsbot, could you check that this resolves the issue?

donsbot commented

Yes, if I make that change, compilation succeeds with bison 3.7.5.

diff --git a/thrift/compiler/parse/thrifty.yy b/thrift/compiler/parse/thrifty.yy
index 77887eb831..c6ac3e5a4e 100644
--- a/thrift/compiler/parse/thrifty.yy
+++ b/thrift/compiler/parse/thrifty.yy
@@ -60,9 +60,11 @@ std::unique_ptr<T> own(T* ptr) {
   return std::unique_ptr<T>(ptr);
 }

+const yy::parser::symbol_type error = yy::parser::make_tok_error(yy::location());
+
 yy::parser::symbol_type parse_lex(parsing_driver& driver, YYSTYPE*, YYLTYPE*) {
   auto token = driver.get_lexer().get_next_token();
-  if (token.token() == yy::parser::token::tok_error) {
+  if (token.type_get() == error.type_get()) {
     driver.end_parsing();
   }
   return token;
  • 👍1
vitaut commented

Thanks for checking and reporting the issue in the first place. The fix is on its way.

  • 👍1