Fatal error condition occurred in /opt/vcpkg/buildtrees/aws-c-common/src/v0.9.10-492b35b91d.clean/source/allocator.c:172: allocator != ((void *)0)
westfly opened this issue · 4 comments
Describe the bug
As #2609 issue describe , We newly Use vcpkg As cpp-library manager while aws-sdk-cpp updated version from 1.7.108 to 1.11.225, the problem still exsit.
We Use the project manged by vcpkg to build basic example and run as expected successful,so We think Our build-env not the core problem.
Expected Behavior
works same as aws-sdk-cpp 1.7.108 building native without vcpkg manager
Current Behavior
#2609
Fatal error condition occurred in /opt/vcpkg/buildtrees/aws-c-common/src/v0.9.10-492b35b91d.clean/source/allocator.c:172: allocator != ((void *)0)
Exiting Application
################################################################################
Stack trace:
################################################################################
/data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(aws_backtrace_print+0x61) [0x7f1ade9b9e81]
/data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(aws_fatal_assert+0x4d) [0x7f1ade9aec0d]
/data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(aws_mem_acquire+0x3e) [0x7f1ade9add0e]
/data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(aws_string_new_from_array+0x25) [0x7f1ade9bafb5]
/data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(aws_json_value_get_from_object+0x2a) [0x7f1ade9b676a]
/data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(aws_endpoints_ruleset_new_from_string+0x76) [0x7f1ade9a7916]
/data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(_ZN3Aws3Crt9Endpoints10RuleEngineC1ERK15aws_byte_cursorS5_P13aws_allocator+0x30) [0x7f1ade8887a0]
/data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(_ZN9mindalpha11S3SDKStreamC1Ev+0xfc) [0x7f1ade13bd1c]
/data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(_ZN9mindalpha12S3FileSystem4OpenERKNS_3URIEPKcb+0x23) [0x7f1ade1304e3]
/data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(_ZN9mindalpha6Stream6CreateEPKcS2_b+0x12e) [0x7f1ade12b6fe]
/data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(_ZN9mindalpha13StreamReadAllERKSs+0x41) [0x7f1ade12c601]
Reproduction Steps
As describe above, the offical sdk example work right. We use awk-sdk-cpp to access s3 bucket implement at s3_sdk_filesys.cpp which call Aws::InitAPI
, We build static library mindalpha_filesystem via cmake
add_library(mindalpha_filesystem STATIC
cpp/mindalpha/io.cpp
cpp/mindalpha/filesys.cpp
cpp/mindalpha/local_filesys.cpp
cpp/mindalpha/s3_sdk_filesys.cpp
cpp/mindalpha/stack_trace_utils.h
cpp/mindalpha/stack_trace_utils.cpp
)
target_link_libraries(mindalpha_filesystem PRIVATE
aws-cpp-sdk-s3
aws-cpp-sdk-core
spdlog::spdlog
)
and add a little test case on mindalpha_filesystem
#include <mindalpha/io.h>
int main(int argc, char const *argv[]) {
if (argc < 1) {
printf("usage: binary s3_path\n");
}
{
std::string path(argv[1]);
auto content = mindalpha::StreamReadAll(path);
printf("%s\t%ld\n", content.data(), content.size());
}
return 0;
}
below is cmake
macro(unittest_helper_lite target cpp_file)
add_executable(${target}
${cpp_file}
)
target_compile_features(${target} INTERFACE cxx_std_17)
target_link_libraries(${target} PRIVATE
mindalpha_filesystem
spdlog::spdlog
dl
util
pthread
${Boost_LIBRARIES} )
endmacro(unittest_helper_lite)
unittest_helper_lite(s3_filesystem tests/cpp/s3_filesystem_test.cpp)
unittest_helper_lite(hello_s3 tests/cpp/hello_s3.cpp)
the coredump as below
./s3_filesystem s3://mob-emr-test/ml-platform/mindalpha/server.conf
Fatal error condition occurred in /opt/vcpkg/buildtrees/aws-c-common/src/v0.9.10-492b35b91d.clean/source/allocator.c:172: allocator != ((void *)0)
Exiting Application
################################################################################
Stack trace:
################################################################################
./s3_filesystem(+0x92efd1) [0x55f3c99bcfd1]
./s3_filesystem(+0x923d5d) [0x55f3c99b1d5d]
./s3_filesystem(+0x922e5e) [0x55f3c99b0e5e]
./s3_filesystem(+0x930105) [0x55f3c99be105]
./s3_filesystem(+0x92b8ba) [0x55f3c99b98ba]
./s3_filesystem(+0x91ca66) [0x55f3c99aaa66]
./s3_filesystem(+0x7fd7a0) [0x55f3c988b7a0]
./s3_filesystem(+0xf519c) [0x55f3c918319c]
./s3_filesystem(+0xe9493) [0x55f3c9177493]
./s3_filesystem(+0xe3cbe) [0x55f3c9171cbe]
./s3_filesystem(+0xe4bc1) [0x55f3c9172bc1]
./s3_filesystem(+0xe086b) [0x55f3c916e86b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f7a5c86a083]
./s3_filesystem(+0xe2a8e) [0x55f3c9170a8e]
Aborted (core dumped)
but when We make Aws::InitAPI
call on main and remove s3_sdk_filesys.cpp call Aws::InitAPI
and Aws::ShutdownAPI
which only calling once, the program works right
#include <mindalpha/io.h>
#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>
#include <aws/core/auth/AWSCredentialsProviderChain.h>
int main(int argc, char const *argv[]) {
if (argc < 1) {
printf("usage: binary s3_path\n");
}
Aws::SDKOptions options;
Aws::InitAPI(options);
{
std::string path(argv[1]);
// mindalpha::InputStream stream(path);
auto content = mindalpha::StreamReadAll(path);
printf("%s\t%ld\n", content.data(), content.size());
}
Aws::ShutdownAPI(options);
return 0;
}
Possible Solution
No response
Additional Information/Context
No response
AWS CPP SDK version used
1.11.225
Compiler and Version used
gcc 9.4
Operating System and version
ubuntu 20.04
Hello,
Thank you for submitting this issue.
Please refer to our basic use: https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/basic-use.html
SDK was initially designed around global context pointers (such as memory allocator, loggers, etc).
In your caae, some of SDK resources (such as endpoint provider within S3 client) leak outside InitAPI { ... } ShutdownAPI scope and fails to de-allocate memory due to memory system being already terminated.
Before it was working thanks to a undefined behavior, but also we had to perform some SDK components refactoring, so the issue related to global context started to pop out.
Best regards,
Sergey
Hello, Thank you for submitting this issue. Please refer to our basic use: https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/basic-use.html
SDK was initially designed around global context pointers (such as memory allocator, loggers, etc). In your caae, some of SDK resources (such as endpoint provider within S3 client) leak outside InitAPI { ... } ShutdownAPI scope and fails to de-allocate memory due to memory system being already terminated.
Before it was working thanks to a undefined behavior, but also we had to perform some SDK components refactoring, so the issue related to global context started to pop out.
Best regards, Sergey
When removing the vcpkg environment, I build aws-sdk-cpp on native ubuntu20.04 via the shell script below
function make_aws_cpp_sdk_deps() {
version=1.7.108
version=1.11.225
version=1.11.33
version=1.8.126
version=1.9.0
git clone --depth 1 -b $version https://github.com/aws/aws-sdk-cpp.git /tmp/aws-sdk-cpp && \
cd /tmp/aws-sdk-cpp && git submodule update --init --recursive && \
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr \
-DBUILD_SHARED_LIBS=OFF -DENABLE_TESTING=OFF -DBUILD_ONLY=s3 \
-D_GLIBCXX_USE_CXX11_ABI=1 \
-H/tmp/aws-sdk-cpp -B/tmp/aws-sdk-cpp/build && \
cmake --build /tmp/aws-sdk-cpp/build --target install && \
rm -rf /tmp/aws-sdk-cpp
}
I found 1.8.126 works right, while 1.9.0 which adds aw-crt-cpp as a submodule coredump at aws::InitAPI
Re: first issue
Are you sure mindalpha::StreamReadAll is actually calling InitAPI internally? Taking a quick look through the mindalpha code im not sure it actually will. In that case it makes sense why adding Init and Shutdown outside of mindalpha fixes the issue. As a side note: mindalpha should really avoid reinitializing client on every api call, that is very expensive.
Re: core dump issue
core dumps in init often occur because build had some stale artifacts and things like std lib implementations are no longer matching. make sure to clean your build directory before rebuilding it
Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.