1.11.337+ Android cURL Build No Longer Compiles
AMZN-Gene opened this issue · 21 comments
Describe the bug
Compiling the AWS SDK for Android no longer works for releases after 1.11.337. The build fails during the cURL build step.
The previous release (1.11.336) still compiles. I think this is because it's missing OpenSSL
Expected Behavior
I should be able to build the SDK successfully for Android
Current Behavior
Compilation fails: notice "Could NOT find OpenSSL"
-- Build files have been written to: D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-build
[1/8] Creating directories for 'CURL'
[2/8] Performing download step (git clone) for 'CURL'
Cloning into 'curl'...
HEAD is now at 44b9b4d4f RELEASE-NOTES: curl 7.52.1
[3/8] No update step for 'CURL'
[4/8] No patch step for 'CURL'
[5/8] Performing configure step for 'CURL'
FAILED: CURL-prefix/src/CURL-stamp/CURL-configure D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-build/CURL-prefix/src/CURL-stamp/CURL-configure
C:\Windows\system32\cmd.exe /C "cd /D D:\3p-package-source\package-system\AWSNativeSDK\temp\build\Debug_Shared\external-build\CURL-prefix\src\CURL-build && "C:\Program Files\CMake\bin\cmake.exe" -C D:/3p-package-source/package-system/AWSNativeSDK/temp/src/tools/android-build/CurlAndroidCrossCompile.cmake -DCMAKE_TOOLCHAIN_FILE=C:/Users/genewalt/AppData/Local/Android/Sdk/ndk/25.2.9519653/build/cmake/android.toolchain.cmake -DCMAKE_ANDROID_NDK_TOOLCHAIN_VERSION=clang -DANDROID_NATIVE_API_LEVEL=21 -DANDROID_ABI=arm64-v8a -DANDROID_TOOLCHAIN=clang -DANDROID_STL=c++_shared -DCMAKE_INSTALL_PREFIX=D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-install/curl "-DCMAKE_CXX_FLAGS=-Wno-unused-private-field -Wno-unused-value -fPIE" "-DCMAKE_C_FLAGS= -Wno-unused-value -fPIE" -DCMAKE_STATIC_LINKER_FLAGS= "-DCMAKE_SHARED_LINKER_FLAGS=-Wl,--build-id=sha1 -Wl,--no-rosegment -Wl,--fatal-warnings -Wl,--no-undefined -Qunused-arguments" "-DCMAKE_EXE_LINKER_FLAGS=-Wl,--build-id=sha1 -Wl,--no-rosegment -Wl,--fatal-warnings -Wl,--no-undefined -Qunused-arguments -fPIE -pie" -DCMAKE_BUILD_TYPE=Debug -DOPENSSL_ROOT_DIR= -DOPENSSL_INCLUDE_DIR= -DCURL_STATICLIB=ON -DBUILD_CURL_EXE=ON -DBUILD_CURL_TESTS=OFF -DCURL_ZLIB=ON -GNinja -S D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/curl -B D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-build/CURL-prefix/src/CURL-build && "C:\Program Files\CMake\bin\cmake.exe" -E touch D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-build/CURL-prefix/src/CURL-stamp/CURL-configure"
loading initial cache file D:/3p-package-source/package-system/AWSNativeSDK/temp/src/tools/android-build/CurlAndroidCrossCompile.cmake
CMake Deprecation Warning at CMakeLists.txt:41 (cmake_minimum_required):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
-- The C compiler identification is Clang 14.0.7
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Users/genewalt/AppData/Local/Android/Sdk/ndk/25.2.9519653/toolchains/llvm/prebuilt/windows-x86_64/bin/clang.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
CMake Warning at CMakeLists.txt:49 (message):
the curl cmake build system is poorly maintained. Be aware
-- curl version=[7.52.1-DEV]
-- Performing Test HAVE_SOCKADDR_IN6_SIN6_ADDR
-- Performing Test HAVE_SOCKADDR_IN6_SIN6_ADDR - Success
-- Performing Test HAVE_SOCKADDR_IN6_SIN6_SCOPE_ID
-- Performing Test HAVE_SOCKADDR_IN6_SIN6_SCOPE_ID - Success
CMake Warning at CMakeLists.txt:224 (message):
Found no *nroff program
-- Looking for dlopen in dl;
-- Looking for dlopen in dl; - found
-- Looking for connect in socket;dl
-- Looking for connect in socket;dl - not found
-- Looking for gethostbyname in c
-- Looking for gethostbyname in c - found
-- Looking for gethostname
-- Looking for gethostname - found
-- Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR)
-- Looking for ldap_init in ldap;dl
-- Looking for ldap_init in ldap;dl - not found
-- Looking for ber_init in lber;dl
-- Looking for ber_init in lber;dl - not found
CMake Warning (dev) at C:/Program Files/CMake/share/cmake-3.29/Modules/CheckIncludeFiles.cmake:116 (message):
Policy CMP0075 is not set: Include file check macros honor
CMAKE_REQUIRED_LIBRARIES. Run "cmake --help-policy CMP0075" for policy
details. Use the cmake_policy command to set the policy and suppress this
warning.
CMAKE_REQUIRED_LIBRARIES is set to:
ssl;crypto
For compatibility with CMake 3.11 and below this check is ignoring it.
Call Stack (most recent call first):
CMake/Macros.cmake:21 (check_include_files)
CMakeLists.txt:377 (check_include_file_concat)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Looking for include file ldap.h
-- Looking for include file ldap.h - not found
-- Looking for include file lber.h
-- Looking for include file lber.h - not found
-- LDAP_H not found CURL_DISABLE_LDAP set ON
-- LDAP needs to be enabled to support LDAPS
-- Looking for idn2_lookup_ul in idn2;dl
-- Looking for idn2_lookup_ul in idn2;dl - not found
-- Looking for dlopen in dl
-- Looking for dlopen in dl - not found
-- Could NOT find LibSSH2 (missing: LIBSSH2_INCLUDE_DIR LIBSSH2_LIBRARY)
-- Performing Test USE_UNIX_SOCKETS
-- Performing Test USE_UNIX_SOCKETS - Failed
-- Looking for include file stdio.h
-- Looking for include file stdio.h - found
-- Looking for include files stdio.h, inttypes.h
-- Looking for include files stdio.h, inttypes.h - found
...
-- Looking for 48 include files stdio.h, ..., netinet/if_ether.h
-- Looking for 48 include files stdio.h, ..., netinet/if_ether.h - found
-- Looking for 50 include files stdio.h, ..., sockio.h
-- Looking for 50 include files stdio.h, ..., sockio.h - not found
-- Looking for 50 include files stdio.h, ..., sys/utsname.h
-- Looking for 50 include files stdio.h, ..., sys/utsname.h - found
-- Check size of size_t
-- Check size of size_t - failed
-- Check size of ssize_t
-- Check size of ssize_t - failed
-- Check size of long long
-- Check size of long long - failed
-- Check size of long
-- Check size of long - failed
-- Check size of short
-- Check size of short - failed
-- Check size of int
-- Check size of int - failed
-- Check size of __int64
-- Check size of __int64 - failed
...
-- Looking for setrlimit - not found
-- Looking for fcntl
-- Looking for fcntl - not found
-- Looking for ioctl
-- Looking for ioctl - not found
-- Looking for setsockopt
-- Looking for setsockopt - not found
-- Looking for inet_pton
-- Looking for inet_pton - not found
-- Looking for sigaction
-- Looking for sigaction - not found
-- Looking for sigsetjmp
-- Looking for sigsetjmp - not found
-- Performing Curl Test HAVE_FCNTL_O_NONBLOCK
-- Performing Curl Test HAVE_FCNTL_O_NONBLOCK - Failed
-- Performing Curl Test HAVE_IOCTLSOCKET
-- Performing Curl Test HAVE_IOCTLSOCKET - Failed
-- Performing Curl Test HAVE_IOCTLSOCKET_CAMEL
-- Performing Curl Test HAVE_IOCTLSOCKET_CAMEL - Failed
-- Performing Curl Test HAVE_IOCTLSOCKET_CAMEL_FIONBIO
-- Performing Curl Test HAVE_IOCTLSOCKET_CAMEL_FIONBIO - Failed
-- Performing Curl Test HAVE_IOCTLSOCKET_FIONBIO
-- Performing Curl Test HAVE_IOCTLSOCKET_FIONBIO - Failed
-- Performing Curl Test HAVE_IOCTL_FIONBIO
-- Performing Curl Test HAVE_IOCTL_FIONBIO - Failed
-- Performing Curl Test HAVE_IOCTL_SIOCGIFADDR
-- Performing Curl Test HAVE_IOCTL_SIOCGIFADDR - Failed
-- Performing Curl Test HAVE_SETSOCKOPT_SO_NONBLOCK
-- Performing Curl Test HAVE_SETSOCKOPT_SO_NONBLOCK - Failed
-- Performing Curl Test TIME_WITH_SYS_TIME
-- Performing Curl Test TIME_WITH_SYS_TIME - Failed
-- Performing Curl Test HAVE_O_NONBLOCK
-- Performing Curl Test HAVE_O_NONBLOCK - Failed
-- Performing Curl Test HAVE_GETHOSTBYADDR_R_5
-- Performing Curl Test HAVE_GETHOSTBYADDR_R_5 - Failed
-- Performing Curl Test HAVE_GETHOSTBYADDR_R_7
-- Performing Curl Test HAVE_GETHOSTBYADDR_R_7 - Failed
-- Performing Curl Test HAVE_GETHOSTBYADDR_R_8
-- Performing Curl Test HAVE_GETHOSTBYADDR_R_8 - Failed
-- Performing Curl Test HAVE_GETHOSTBYADDR_R_5_REENTRANT
-- Performing Curl Test HAVE_GETHOSTBYADDR_R_5_REENTRANT - Failed
-- Performing Curl Test HAVE_GETHOSTBYADDR_R_7_REENTRANT
-- Performing Curl Test HAVE_GETHOSTBYADDR_R_7_REENTRANT - Failed
-- Performing Curl Test HAVE_GETHOSTBYADDR_R_8_REENTRANT
-- Performing Curl Test HAVE_GETHOSTBYADDR_R_8_REENTRANT - Failed
-- Performing Curl Test HAVE_GETHOSTBYNAME_R_3
-- Performing Curl Test HAVE_GETHOSTBYNAME_R_3 - Failed
-- Performing Curl Test HAVE_GETHOSTBYNAME_R_5
-- Performing Curl Test HAVE_GETHOSTBYNAME_R_5 - Failed
-- Performing Curl Test HAVE_GETHOSTBYNAME_R_6
-- Performing Curl Test HAVE_GETHOSTBYNAME_R_6 - Failed
-- Performing Curl Test HAVE_GETHOSTBYNAME_R_3_REENTRANT
-- Performing Curl Test HAVE_GETHOSTBYNAME_R_3_REENTRANT - Failed
-- Performing Curl Test HAVE_GETHOSTBYNAME_R_5_REENTRANT
-- Performing Curl Test HAVE_GETHOSTBYNAME_R_5_REENTRANT - Failed
-- Performing Curl Test HAVE_GETHOSTBYNAME_R_6_REENTRANT
-- Performing Curl Test HAVE_GETHOSTBYNAME_R_6_REENTRANT - Failed
-- Performing Curl Test HAVE_SOCKLEN_T
-- Performing Curl Test HAVE_SOCKLEN_T - Failed
-- Performing Curl Test HAVE_IN_ADDR_T
-- Performing Curl Test HAVE_IN_ADDR_T - Failed
-- Performing Curl Test HAVE_BOOL_T
-- Performing Curl Test HAVE_BOOL_T - Failed
-- Performing Curl Test STDC_HEADERS
-- Performing Curl Test STDC_HEADERS - Failed
-- Performing Curl Test RETSIGTYPE_TEST
-- Performing Curl Test RETSIGTYPE_TEST - Failed
-- Performing Curl Test HAVE_INET_NTOA_R_DECL
-- Performing Curl Test HAVE_INET_NTOA_R_DECL - Failed
-- Performing Curl Test HAVE_INET_NTOA_R_DECL_REENTRANT
-- Performing Curl Test HAVE_INET_NTOA_R_DECL_REENTRANT - Failed
-- Performing Curl Test HAVE_FILE_OFFSET_BITS
-- Performing Curl Test HAVE_FILE_OFFSET_BITS - Failed
-- Check size of off_t
-- Check size of off_t - failed
-- Performing Curl Test HAVE_GLIBC_STRERROR_R
-- Performing Curl Test HAVE_GLIBC_STRERROR_R - Failed
-- Performing Curl Test HAVE_POSIX_STRERROR_R
-- Performing Curl Test HAVE_POSIX_STRERROR_R - Failed
-- Performing Test curl_cv_recv
-- Performing Test curl_cv_recv - Failed
CMake Error at CMake/OtherTests.cmake:86 (message):
Unable to link function recv
Call Stack (most recent call first):
CMakeLists.txt:1012 (include)
-- Configuring incomplete, errors occurred!
ninja: build stopped: subcommand failed.
CMake Error at cmake/build_external.cmake:148 (message):
Failed to build dependency libraries.
Call Stack (most recent call first):
CMakeLists.txt:202 (include)
Reproduction Steps
- Build AWS SDK 1.11.337 for Android using Android NDK 25.2.9519653 (or higher)
cmake 3.29
cmake -S %SRC_PATH% -B %BLD_PATH%\Debug_Shared ^
-G Ninja ^
-DNDK_DIR="%ANDROID_NDK_ROOT%" ^
-DBUILD_SHARED_LIBS=ON ^
-DCMAKE_BUILD_TYPE="Debug ^
-DTARGET_ARCH=ANDROID ^
-DANDROID_NATIVE_API_LEVEL=21 ^
-DANDROID_ABI=arm64-v8a ^
-DCPP_STANDARD=17 ^
-DCMAKE_C_FLAGS="-fPIC" ^
-DCMAKE_CXX_FLAGS="-fPIC" ^
-DBUILD_ONLY="access-management;bedrock-runtime;cognito-identity;cognito-idp;core;devicefarm;dynamodb;gamelift;identity-management;kinesis;lambda;queues;s3;sns;sqs;sts;transfer" ^
-DENABLE_TESTING=OFF ^
-DENABLE_RTTI=ON ^
-DCUSTOM_MEMORY_MANAGEMENT=ON^
-DCMAKE_INSTALL_BINDIR="bin/%BUILD_TYPE%_%LIB_TYPE%" ^
-DCMAKE_INSTALL_LIBDIR="lib/%BUILD_TYPE%_%LIB_TYPE%" ^
-DCMAKE_INSTALL_PREFIX="%BLD_PATH%/%BUILD_TYPE%_%LIB_TYPE%" ^
-DANDROID_BUILD_OPENSSL=ON ^
-DANDROID_BUILD_ZLIB=OFF ^
-DANDROID_BUILD_CURL=ON ^
-DCMAKE_MODULE_PATH="%DOWNLOADED_PACKAGE_FOLDERS%" ^
Possible Solution
No response
Additional Information/Context
This PR removed the ANDROID_BUILD_OPENSSL flag, but the readme still says it's available. Is there a replacement?
AWS CPP SDK version used
1.11.337
Compiler and Version used
clang++ Android (9352603, based on r450784d1) clang version 14.0.7
Operating System and version
Windows 10 Enterprise
This PR removed the ANDROID_BUILD_OPENSSL flag, but the readme still says it's available. Is there a replacement?
that PR removed the dependency on OpenSSL from the SDK all together, that parameter should be removed from documentation. can you give me more context on why you need that parameter there? we now use CRT for all crypto related functionalities.
This PR removed the ANDROID_BUILD_OPENSSL flag, but the readme still says it's available. Is there a replacement?
that PR removed the dependency on OpenSSL from the SDK all together, that parameter should be removed from documentation. can you give me more context on why you need that parameter there? we now use CRT for all crypto related functionalities.
Cool! If OpenSSL is not longer necessary that would be great.
It's possible the "Could NOT find OpenSSL" warning is just a red herring. My build ultimately fails because the curl_cv_recv test fails and stops the process. This only occurs in releases after 337.
If I make my own small project outside of AWS then curl_cv_recv compiles fine.
Even though ANDROID_BUILD_OPENSSL is gone, I discovered AWS SDK checks for a BUILD_OPENSSL=ON variable. This moves the build further along, but still get a compiler error.
ld: error: undefined symbol: SSL_shutdown
>>> referenced by openssl.c:905 (D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/curl/lib/vtls/openssl.c:905)
>>> openssl.c.o:(ossl_close) in archive external-install/curl/lib/libcurl.a
>>> referenced by openssl.c:949 (D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/curl/lib/vtls/openssl.c:949)
>>> openssl.c.o:(Curl_ossl_shutdown) in archive external-install/curl/lib/libcurl.a```
I can see SSL_shutdown is defined inside the %build_folder%\external-install\openssl\include\openssl\ssl.h, so maybe I just need to tell cmake where to look for OpenSSL. Hopefully, like you said, I can avoid using OpenSSL altogether...
The error comes from a CURL test.
-- Performing Test curl_cv_recv - Failed
CMake Error at CMake/OtherTests.cmake:86 (message):
Unable to link function recv
Question: If we're not using OpenSSL do we also need cURL? If we do still need cURL, did cURL get updated to not rely on OpenSSL?
Cool! If OpenSSL is not longer necessary that would be great.
yeah this is what we are trying to do, less dependencies, less trouble....hopefully so lets sort this out.
If we're not using OpenSSL do we also need cURL? If we do still need cURL, did cURL get updated to not rely on OpenSSL?
to elaborate a little bit, openssl was used by the SDK directly for some hashing functions, and we used to directly link against it for cryptographic functions. this is no longer done. this is instead done in the CRT.
you definitely still need curl, we plan on using the crt http client in the future but thats in the future.
curl has always and will always need some sort of openssl to use tls. We do not prescribe how you get your curl dependency. we just link against it.
-- Performing Test curl_cv_recv - Failed
CMake Error at CMake/OtherTests.cmake:86 (message):
Unable to link function recv```
this error looks like it happens in your code, if you are brining curl into your dependency set, how are you bringing in openssl? i dont see that test anywhere in our code.
this error looks like it happens in your code, if you are brining curl into your dependency set, how are you bringing in openssl? i dont see that test anywhere in our code.
Sort of... I'm not explicitly bring in my own curl, I'm using AWS's -DANDROID_BUILD_CURL=ON (the default) which brings it in. Using DANDROID_BUILD_OPENSSL used to handle all that for me, but now it doesn't
Looks like AWS Android tools still uses "BUILD_OPENSSL" to build/install OpenSSL for us. Maybe someone just forgot to add some CMake that CURL uses to find where OpenSSL was installed.
https://github.com/aws/aws-sdk-cpp/blob/main/tools/android-build/CMakeLists.txt#L35-L57
this is just out of curiosity by the way, but why are you relying on us building curl and openssl for you? it would seemingly be easier to just use whatever curl and openssl was on the path? it would seem that BUILD_OPENSSL should "enable" building openssl like the old way, curious you see a issue though, that to me says the old way was accidentally using the wrong openssl.
Give me a bit and i can try to reproduce this, until then I would recommend using a curl or openssl on your path rather than building it alongside the SDK.
Sort of... I'm not explicitly bring in my own curl
where is the test CMake/OtherTests.cmake being run from? what project?
I maintain a game engine which ships with OpenSSL as a 3rdParty, so I need to be careful about ensuring the same OpenSSL is used across all our libraries. I don't have OpenSSL on my path. AWS's tools/android-build/CMakeLists.txt made it really easy to build and use a specific OpenSSL when using Android (specifically my engine uses OpenSSL 1.1.1g)
diff --git a/tools/android-build/CMakeLists.txt b/tools/android-build/CMakeLists.txt
index dedf79cec8..14beffaf14 100644
--- a/tools/android-build/CMakeLists.txt
+++ b/tools/android-build/CMakeLists.txt
@@ -37,7 +37,7 @@ if(BUILD_OPENSSL)
ExternalProject_Add(OPENSSL
SOURCE_DIR ${OPENSSL_SOURCE_DIR}
GIT_REPOSITORY https://github.com/openssl/openssl.git
- GIT_TAG e2e09d9fba1187f8d6aafaa34d4172f56f1ffb72 # 1.1.1g
+ GIT_TAG ca2e0784d2c38edcefd5d68028f4d954bd8faddb # 1.1.1o
UPDATE_COMMAND ""
PATCH_COMMAND cd ${CMAKE_BINARY_DIR} && python ${AWS_NATIVE_SDK_ROOT}/tools/android-build/configure_openssl_cmake.py --source ${AWS_NATIVE_SDK_ROOT} --dest ${OPENSSL_SOURCE_DIR}
CMAKE_ARGS
so im curious at this point too, what fails to compile as CMake/OtherTests.cmake is not in the SDK. does the SDK fail to build or does your application fail to build?
CURL runs these tests which just tries to compile some simple code and it fails.
#undef inline
#include <sys/types.h>
#include <sys/socket.h>
int main(void) {
recv(0, 0, 0, 0);
return 0;
}
It's the same error as before (which I was open to solve by letting AWS handle OpenSSL for me :c)
#1282
Closer...
Replacing -DANDROID_BUILD_OPENSSL=ON with the follow allows CURL to successfully "cmake configure" and begin compiling.
-DBUILD_OPENSSL=1 ^
-DCRYPTO_TARGET_NAME="crypto" ^
-DUSE_OPENSSL=ON ^
The build continues, but fails when compiling to missing openssl symbols. for example:
ld: error: undefined symbol: SSL_shutdown
>>> referenced by openssl.c:905 (D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/curl/lib/vtls/openssl.c:905)
>>> openssl.c.o:(ossl_close) in archive external-install/curl/lib/libcurl.a
>>> referenced by openssl.c:949 (D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/curl/lib/vtls/openssl.c:949)
>>> openssl.c.o:(Curl_ossl_shutdown) in archive external-install/curl/lib/libcurl.a
I'm able to get around this if I simply hand edit <build_folder>\CMakeFiles\aws-cpp-sdk-core.rsp to include libssl.a. Strange the rsp already includes the other library "libcrypto.a" from the same folder.
Hand edit fix:
external-install/openssl/lib/libcrypto.a external-install/openssl/lib/libssl.a
Hand editing isn't a viable option, but there's probably a simple flag I'm missing that would place libssl into the build.
This would be an issue even if I built OpenSSL and passed in OpenSSL myself. If -DANDROID_BUILD_CURL=ON, what's the parameter I need to set to tell it where to find OpenSSL?
I get passed the undefined symbol setting more flags, but run into a missing include dir. Need to inject the proper openSSL include folder into clang's -I parameter. It looks like it's attempting to provide the include path below... but just not quite right.
-ID:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-build/CURL-prefix/src/CURL-build/temp/build/Debug_Shared/external-install/openssl/include
... should just be...
-ID:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-install/openssl/include
So it's trying to build an include path, but it's just wrong.
Error:
C:\Users\genewalt\AppData\Local\Android\Sdk\ndk\25.2.9519653\toolchains\llvm\prebuilt\windows-x86_64\bin\clang.exe --target=aarch64-none-linux-android21 --sysroot=C:/Users/genewalt/AppData/Local/Android/Sdk/ndk/25.2.9519653/toolchains/llvm/prebuilt/windows-x86_64/sysroot -DBUILDING_LIBCURL -DCURL_HIDDEN_SYMBOLS -DHAVE_CONFIG_H -ID:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-build/CURL-prefix/src/CURL-build/include/curl -ID:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/curl/include -ID:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-build/CURL-prefix/src/CURL-build/temp/build/Debug_Shared/external-install/openssl/include -ID:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-build/CURL-prefix/src/CURL-build/lib/../include -ID:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/curl/lib/.. -ID:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/curl/lib/../include -ID:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-build/CURL-prefix/src/CURL-build/lib/.. -ID:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/curl/lib -ID:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-build/CURL-prefix/src/CURL-build/lib -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -isystem temp\build/Debug_Shared/external-install/openssl/include -isystem temp\build/Debug_Shared/external-install/openssl/include/openssl -Wno-unused-value -fPIE -LD:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-install/openssl/lib -fno-limit-debug-info -fPIC -fvisibility=hidden -MD -MT lib/CMakeFiles/libcurl.dir/connect.c.o -MF lib\CMakeFiles\libcurl.dir\connect.c.o.d -o lib/CMakeFiles/libcurl.dir/connect.c.o -c D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/curl/lib/connect.c
clang: warning: argument unused during compilation: '-LD:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-install/openssl/lib' [-Wunused-command-line-argument]
In file included from D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/curl/lib/connect.c:59:
D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/curl/lib/urldata.h:86:10: fatal error: 'openssl/ssl.h' file not found
#include <openssl/ssl.h>
^~~~~~~~~~~~~~~
1 error generated.
So even when providing my own OpenSSL, the CURL step is failing.
There's still that warning: Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR)
I've explicitly set the OPENSSL_ROOT_DIR system variable. (Normally cmake FindOpenSSL will do this by looking inside all the folders within CMAKE_MODULE_PATH variable)
I wonder if it's because we're overwriting OPENSSL_ROOT_DIR to AWS's $OPENSSL_SRC_DIR variable.
https://github.com/aws/aws-sdk-cpp/blob/main/tools/android-build/CMakeLists.txt#L83
OPENSSL_SRC_DIR is the variable AWS sets when building OpenSSL, but if OpenSSL is already built (ie, i'm using my own) I need that OPENSSL_ROOT_DIR variable to be the location of OpenSSL already on my hard drive.
if curl is failing to build with that openssl, you could also just bring in your own curl instead of building it, most of our customers bring their own http and crypto deps. We do want remove all deps except the CRT. removing openssl is acutally the first step to that. Android is the only one that offers a "we build dependencies for you". Im still working on reproducing it.
alright replicated with the dockerfils
# Use nomral ubuntu base
FROM public.ecr.aws/amazonlinux/amazonlinux:2023
#Install g++
RUN yum groupinstall "Development Tools" -y
#Install required dependencies
RUN yum install -y ninja-build cmake3 wget
#Install NDK
RUN wget -q https://dl.google.com/android/repository/android-ndk-r26d-linux.zip && \
unzip -q android-ndk-r26d-linux.zip
#Try to build SDK
RUN git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp && \
cd aws-sdk-cpp && \
mkdir build &&\
cd build &&\
cmake -DCMAKE_BUILD_TYPE="Debug" \
-DNDK_DIR="/android-ndk-r26d" \
-DTARGET_ARCH=ANDROID \
-DANDROID_NATIVE_API_LEVEL=21 \
-DANDROID_ABI=arm64-v8a \
-DCPP_STANDARD=17 \
-DCMAKE_C_FLAGS="-fPIC" \
-DBUILD_ONLY="s3" \
-DENABLE_TESTING=OFF \
-DENABLE_RTTI=ON \
-DCUSTOM_MEMORY_MANAGEMENT=O \
-DANDROID_BUILD_OPENSSL=ON \
-DANDROID_BUILD_ZLIB=OFF \
-DANDROID_BUILD_CURL=ON \
-DCMAKE_CXX_FLAGS="-fPIC" .. &&\
cmake --build . && \
cmake --install .looking into fixes
created a branch fix-android, im seeing some linker errors in my docker build curious if you have any success with it.
although i think that is where you got in your testing, looking into why things arent getting linked
I've been working towards compiling CURL by hand and passing in my own OpenSSL.
The original error was during cmake configuration (not even compiling/linking yet)
-- Performing Test curl_cv_recv - Failed
CMake Error at CMake/OtherTests.cmake:86 (message):
Unable to link function recv
Call Stack (most recent call first):
CMakeLists.txt:1012 (include)
-- Configuring incomplete, errors occurred!
ninja: build stopped: subcommand failed.
CMake Error at cmake/build_external.cmake:148 (message):
Failed to build dependency libraries.
Call Stack (most recent call first):
CMakeLists.txt:202 (include)
If I provided a path to my own OpenSSL I get past the recv error, but hit a new error:
(notice the DOPENSSL paths are all hardcoded)
cmake -DCMAKE_TOOLCHAIN_FILE=C:/Users/genewalt/AppData/Local/Android/Sdk/ndk/25.2.9519653/build/cmake/android.toolchain.cmake -DCMAKE_ANDROID_NDK_TOOLCHAIN_VERSION=clang -DANDROID_NATIVE_API_LEVEL=21 -DANDROID_ABI=arm64-v8a -DANDROID_TOOLCHAIN=clang -DANDROID_STL=c++_shared -DCMAKE_INSTALL_PREFIX=D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-install/curl -DOPENSSL_ROOT_DIR=C:\Users\genewalt\.o3de\3rdParty\packages\OpenSSL-1.1.1o-rev2-android\ -DOPENSSL_INCLUDE_DIR=C:\Users\genewalt\.o3de\3rdParty\packages\OpenSSL-1.1.1o-rev2-android\OpenSSL\include\ -DOPENSSL_CRYPTO_LIBRARY=C:\Users\genewalt\.o3de\3rdParty\packages\OpenSSL-1.1.1o-rev2-android\OpenSSL\lib\libcrypto.a -DOPENSSL_SSL_LIBRARY=C:\Users\genewalt\.o3de\3rdParty\packages\OpenSSL-1.1.1o-rev2-android\OpenSSL\lib\libssl.a "-DCMAKE_CXX_FLAGS=-Wno-unused-private-field -Wno-unused-value -fPIE" "-DCMAKE_C_FLAGS= -Wno-unused-value -fPIE" -DCMAKE_STATIC_LINKER_FLAGS= "-DCMAKE_SHARED_LINKER_FLAGS=-Wl,--build-id=sha1 -Wl,--no-rosegment -Wl,--no-undefined -Qunused-arguments" "-DCMAKE_EXE_LINKER_FLAGS=-Wl,--build-id=sha1 -Wl,--no-rosegment -Wl,--no-undefined -Qunused-arguments -fPIE -pie" -DCMAKE_BUILD_TYPE=Debug -DOPENSSL_ROOT_DIR="C:/Users/genewalt/.o3de/3rdParty/packages/OpenSSL-1.1.1o-rev2-android" -DCURL_STATICLIB=ON -DBUILD_CURL_EXE=ON -DBUILD_CURL_TESTS=OFF -DCURL_ZLIB=ON -GNinja -S D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/curl -B D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-build/CURL-prefix/src/CURL-build
The new error looked liked:
-- Configuring incomplete, errors occurred!
-- Performing Test HAVE_POLL_FINE
CMake Error: try_run() invoked in cross-compiling mode, please set the following cache variables appropriately:
HAVE_POLL_FINE_EXITCODE (advanced)
For details see D:/3p-package-source/package-system/AWSNativeSDK/temp/build/Debug_Shared/external-build/CURL-prefix/src/CURL-build/TryRunResults.cmake
-- Performing Test HAVE_POLL_FINE - Failed
CURL fixed this in January 2019:
curl/curl#3475
I just realized, none of my environment variables are making it to the portion of cmake that's responsible for building CURL: \tools\android-build\CMakeLists.txt
For example, if I add a new variable ${GENE}, adding -DGENE to the AWS cmake build command line wont give access to the android-build CURL CMakeList.
I've added a message right after if(BUILD_CURL) https://github.com/aws/aws-sdk-cpp/blob/main/tools/android-build/CMakeLists.txt#L60
message("GENE: ${GENE}")
This prints "GENE: " $GENE is empty.
Need to figure out how to get variables into that android-build/CMakeLists.txt
EDIT: Nevermind, I found this is set via \cmake\build_external.cmake
alright little update, the branch i have going fix-android does fix the build. im 99 percent sure its because i accidentally removed the lines
if(BUILD_OPENSSL)
target_include_directories(${PROJECT_NAME} PRIVATE "${OPENSSL_INCLUDE_DIR}")
endif()
when I made the PR. Im going to work a little more on that branch to whittle down the "rollback" as im pretty sure those are the only lines that it needs. If you need a working branch right now use that branch at this commit. but I should have a fix ready sooner than later, or rather as quick as my testing feedback loops will allow for it.
created a pull request, you can use/test that branch if you would like to, will give a shout when its been released
This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.