awslabs/ml-io

Hit python build error on AL2012

Opened this issue · 1 comments

Hi there, I'm building this on AL2012 with given instruction https://github.com/awslabs/ml-io/blob/master/doc/build.md. And I'm receiving following error:

`_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED1Ev' referenced in section `.text' of ../third-party/lib/libprotobuf.a(strutil.o): defined in discarded section `.gnu.linkonce.t._ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED5Ev' of src/mlio/CMakeFiles/mlio.d[1/3] Linking CXX shared library lib/libmlio.so.0.7.1
FAILED: lib/libmlio.so.0.7.1
: && /usr/local/bin/g++ -fPIC -O3 -DNDEBUG -flto -fno-fat-lto-objects  -Wl,--as-needed -Wl,--build-id=sha1 -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -shared -Wl,-soname,libmlio.so.0 -o lib/libmlio.so.0.7.1 src/mlio/detail/protobuf/CMakeFiles/mlio-protobuf.dir/recordio_protobuf.pb.cc.o src/mlio/CMakeFiles/mlio.dir/data_stores/detail/util.cc.o src/mlio/CMakeFiles/mlio.dir/data_stores/compression.cc.o src/mlio/CMakeFiles/mlio.dir/data_stores/data_store.cc.o src/mlio/CMakeFiles/mlio.dir/data_stores/file.cc.o src/mlio/CMakeFiles/mlio.dir/data_stores/file_list.cc.o src/mlio/CMakeFiles/mlio.dir/data_stores/in_memory_store.cc.o src/mlio/CMakeFiles/mlio.dir/data_stores/s3_object.cc.o src/mlio/CMakeFiles/mlio.dir/data_stores/sagemaker_pipe.cc.o src/mlio/CMakeFiles/mlio.dir/detail/path.cc.o src/mlio/CMakeFiles/mlio.dir/detail/s3_utils.cc.o src/mlio/CMakeFiles/mlio.dir/detail/system_info.cc.o src/mlio/CMakeFiles/mlio.dir/instance_readers/core_instance_reader.cc.o src/mlio/CMakeFiles/mlio.dir/instance_readers/instance_reader.cc.o src/mlio/CMakeFiles/mlio.dir/instance_readers/instance_reader_base.cc.o src/mlio/CMakeFiles/mlio.dir/instance_readers/ranged_instance_reader.cc.o src/mlio/CMakeFiles/mlio.dir/instance_readers/sampled_instance_reader.cc.o src/mlio/CMakeFiles/mlio.dir/instance_readers/sharded_instance_reader.cc.o src/mlio/CMakeFiles/mlio.dir/instance_readers/shuffled_instance_reader.cc.o src/mlio/CMakeFiles/mlio.dir/integ/dlpack.cc.o src/mlio/CMakeFiles/mlio.dir/memory/external_memory_block.cc.o src/mlio/CMakeFiles/mlio.dir/memory/file_backed_memory_block.cc.o src/mlio/CMakeFiles/mlio.dir/memory/file_backed_memory_allocator.cc.o src/mlio/CMakeFiles/mlio.dir/memory/file_mapped_memory_block.cc.o src/mlio/CMakeFiles/mlio.dir/memory/heap_memory_allocator.cc.o src/mlio/CMakeFiles/mlio.dir/memory/heap_memory_block.cc.o src/mlio/CMakeFiles/mlio.dir/memory/memory_allocator.cc.o src/mlio/CMakeFiles/mlio.dir/memory/memory_block.cc.o src/mlio/CMakeFiles/mlio.dir/memory/memory_slice.cc.o src/mlio/CMakeFiles/mlio.dir/memory/util.cc.o src/mlio/CMakeFiles/mlio.dir/record_readers/detail/chunk_reader.cc.o src/mlio/CMakeFiles/mlio.dir/record_readers/detail/default_chunk_reader.cc.o src/mlio/CMakeFiles/mlio.dir/record_readers/detail/in_memory_chunk_reader.cc.o src/mlio/CMakeFiles/mlio.dir/record_readers/detail/recordio_header.cc.o src/mlio/CMakeFiles/mlio.dir/record_readers/detail/text_line.cc.o src/mlio/CMakeFiles/mlio.dir/record_readers/csv_record_reader.cc.o src/mlio/CMakeFiles/mlio.dir/record_readers/parquet_record_reader.cc.o src/mlio/CMakeFiles/mlio.dir/record_readers/record_error.cc.o src/mlio/CMakeFiles/mlio.dir/record_readers/record_reader.cc.o src/mlio/CMakeFiles/mlio.dir/record_readers/record_reader_base.cc.o src/mlio/CMakeFiles/mlio.dir/record_readers/recordio_record_reader.cc.o src/mlio/CMakeFiles/mlio.dir/record_readers/stream_record_reader.cc.o src/mlio/CMakeFiles/mlio.dir/record_readers/text_line_record_reader.cc.o src/mlio/CMakeFiles/mlio.dir/record_readers/text_record_reader.cc.o src/mlio/CMakeFiles/mlio.dir/streams/detail/iconv.cc.o src/mlio/CMakeFiles/mlio.dir/streams/detail/zlib.cc.o src/mlio/CMakeFiles/mlio.dir/streams/file_input_stream.cc.o src/mlio/CMakeFiles/mlio.dir/streams/gzip_inflate_stream.cc.o src/mlio/CMakeFiles/mlio.dir/streams/input_stream_base.cc.o src/mlio/CMakeFiles/mlio.dir/streams/input_stream.cc.o src/mlio/CMakeFiles/mlio.dir/streams/memory_input_stream.cc.o src/mlio/CMakeFiles/mlio.dir/streams/s3_input_stream.cc.o src/mlio/CMakeFiles/mlio.dir/streams/sagemaker_pipe_input_stream.cc.o src/mlio/CMakeFiles/mlio.dir/streams/stream_error.cc.o src/mlio/CMakeFiles/mlio.dir/streams/utf8_input_stream.cc.o src/mlio/CMakeFiles/mlio.dir/util/number.cc.o src/mlio/CMakeFiles/mlio.dir/util/string.cc.o src/mlio/CMakeFiles/mlio.dir/config.cc.o src/mlio/CMakeFiles/mlio.dir/coo_tensor_builder.cc.o src/mlio/CMakeFiles/mlio.dir/cpu_array.cc.o src/mlio/CMakeFiles/mlio.dir/csv_reader.cc.o src/mlio/CMakeFiles/mlio.dir/csv_record_tokenizer.cc.o src/mlio/CMakeFiles/mlio.dir/data_reader_base.cc.o src/mlio/CMakeFiles/mlio.dir/data_reader.cc.o src/mlio/CMakeFiles/mlio.dir/data_reader_error.cc.o src/mlio/CMakeFiles/mlio.dir/data_type.cc.o src/mlio/CMakeFiles/mlio.dir/device_array.cc.o src/mlio/CMakeFiles/mlio.dir/device.cc.o src/mlio/CMakeFiles/mlio.dir/example.cc.o src/mlio/CMakeFiles/mlio.dir/image_reader.cc.o src/mlio/CMakeFiles/mlio.dir/init.cc.o src/mlio/CMakeFiles/mlio.dir/init_aws.cc.o src/mlio/CMakeFiles/mlio.dir/instance.cc.o src/mlio/CMakeFiles/mlio.dir/instance_batch.cc.o src/mlio/CMakeFiles/mlio.dir/instance_batch_reader.cc.o src/mlio/CMakeFiles/mlio.dir/logger.cc.o src/mlio/CMakeFiles/mlio.dir/mlio_error.cc.o src/mlio/CMakeFiles/mlio.dir/not_supported_error.cc.o src/mlio/CMakeFiles/mlio.dir/parallel_data_reader.cc.o src/mlio/CMakeFiles/mlio.dir/parser.cc.o src/mlio/CMakeFiles/mlio.dir/recordio_protobuf_reader.cc.o src/mlio/CMakeFiles/mlio.dir/s3_client.cc.o src/mlio/CMakeFiles/mlio.dir/schema.cc.o src/mlio/CMakeFiles/mlio.dir/tensor.cc.o src/mlio/CMakeFiles/mlio.dir/tensor_visitor.cc.o src/mlio/CMakeFiles/mlio.dir/text_encoding.cc.o src/mlio/CMakeFiles/mlio.dir/text_line_reader.cc.o  -Wl,-rpath,/data/ml-io/build/third-party/lib/intel64/gcc4.7:  ../third-party/lib64/libabsl_strings.a  ../third-party/lib64/libfmt.a  ../third-party/lib64/libstrnatcmp.a  ../third-party/lib/libprotobuf.a  ../third-party/lib/intel64/gcc4.7/libtbb.so.2  /usr/lib64/libz.so  ../third-party/lib64/libabsl_strings_internal.a  ../third-party/lib64/libabsl_throw_delegate.a  ../third-party/lib64/libabsl_base.a  ../third-party/lib64/libabsl_dynamic_annotations.a  ../third-party/lib64/libabsl_log_severity.a  ../third-party/lib64/libabsl_spinlock_wait.a  ../third-party/lib64/libabsl_int128.a  -pthread  -lc && :
`_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED1Ev' referenced in section `.text' of ../third-party/lib/libprotobuf.a(common.o): defined in discarded section `.gnu.linkonce.t._ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED5Ev' of src/mlio/CMakeFiles/mlio.dir/device.cc.o (symbol from plugin)

So we did a further deep dive and found few linking problems building MLIO on AL2012.

  1. There is a linking problem while building natsort. You need to enable this DCMAKE_POSITION_INDEPENDENT_CODE for https://github.com/awslabs/ml-io/blob/master/third-party/CMakeLists.txt#L117

  2. There is linking problem building python core. We have to link libpython3.6m.so in https://github.com/awslabs/ml-io/blob/master/src/mlio-py/mlio/core/CMakeLists.txt#L26

Can you guys confirm this as the fix?