mlc-ai/tokenizers-cpp

Error when compiling tokenizers for AMD with HIP C++ compiler (hipcc)

goliaro opened this issue · 1 comments

goliaro commented

I'm running in the error below when attempting to compile the tokenizers using hipcc. Any tips?

In file included from /root/tokenizers-cpp/sentencepiece/third_party/protobuf-lite/repeated_field.cc:35:
/root/tokenizers-cpp/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:923:13: error: reference to __host__ function '~RepeatedPtrFieldBase' in __host__ __device__ function
  constexpr RepeatedPtrField();
            ^
/root/tokenizers-cpp/sentencepiece/third_party/protobuf-lite/repeated_field.cc:133:48: note: in instantiation of member function 'google::protobuf::RepeatedPtrField<std::basic_string<char>>::RepeatedPtrField' requested here
template class PROTOBUF_EXPORT_TEMPLATE_DEFINE RepeatedPtrField<std::string>;
                                               ^
/root/tokenizers-cpp/sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:582:3: note: '~RepeatedPtrFieldBase' declared here
  ~RepeatedPtrFieldBase() {
  ^
1 error generated when compiling for gfx906.
make[2]: *** [tokenizers/sentencepiece/src/CMakeFiles/sentencepiece-static.dir/build.make:272: tokenizers/sentencepiece/src/CMakeFiles/sentencepiece-static.dir/root/tokenizers-cpp/sentencepiece/third_party/protobuf-lite/repeated_field.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:289: tokenizers/sentencepiece/src/CMakeFiles/sentencepiece-static.dir/all] Error 2
make: *** [Makefile:156: all] Error 2

To reproduce:

git clone --recursive https://github.com/mlc-ai/tokenizers-cpp.git
cd examples
mkdir -p build
cd build
cmake .. -DCMAKE_CXX_COMPILER='/opt/rocm/bin/hipcc' -DCMAKE_CXX_LINKER='/opt/rocm/bin/hipcc'
make -j8

Thanks!

tqchen commented

likely you don't want to use hipcc but instead use gcc to compile this, and hipcc to compile everything else then link together

  • 👍2