Upload file to IoT Hub using x509 certificate (CA-22)
Closed this issue · 12 comments
Hi,
I have setup the certificates for the provisioning service and provisioned a few devices. It works fine to send/receive message, direct methods, twin, ... all the operations I need to do except uploading a file.
I create a device using sas auth and test with upload sample and it works fine. When I switch back using x509 certificat it doesn't works.
I made the same test using c# with the same certificate and it works for all the operations and upload file!.
I have the same problem with esp-idf 3.3 and v4.0 released 4 days ago.
I made the following change in port/CMakeLists.txt to have the upload works.
@ -45,6 +45,7 @@ set (COMPONENT_SRCS
"${AZURE_IOT_SDK}/c-utility/src/gb_time.c"
"${AZURE_IOT_SDK}/c-utility/src/hmac.c"
"${AZURE_IOT_SDK}/c-utility/src/hmacsha256.c"
"${AZURE_IOT_SDK}/c-utility/adapters/httpapi_compact.c"
"${AZURE_IOT_SDK}/c-utility/src/httpapiex.c"
"${AZURE_IOT_SDK}/c-utility/src/httpapiexsas.c"
"${AZURE_IOT_SDK}/c-utility/src/httpheaders.c"
@ -61,6 +62,7 @@ set (COMPONENT_SRCS
"${AZURE_IOT_SDK}/c-utility/src/vector.c"
"${AZURE_IOT_SDK}/c-utility/src/xio.c"
"${AZURE_IOT_SDK}/c-utility/src/base64.c"
"${AZURE_IOT_SDK}/iothub_client/src/blob.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_device_client_ll.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_client_ll.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_client_core_ll.c"
Thanks,
I (71) boot: Chip Revision: 1
I (71) boot_comm: chip revision: 1, min. bootloader chip revision: 0
I (39) boot: ESP-IDF v4.0-beta1-279-g0a03a55c1-dirty 2nd stage bootloader
this is the last part of the output:
`I (12298) mbedtls: ssl_tls.c:2755 => flush output
I (12298) mbedtls: ssl_tls.c:2774 message length: 69, out_left: 69
I (12308) mbedtls: ssl_tls.c:2779 ssl->f_send() returned 69 (-0xffffffbb)
I (12318) mbedtls: ssl_tls.c:2807 <= flush output
I (12318) mbedtls: ssl_tls.c:3476 <= write record
I (12328) mbedtls: ssl_tls.c:5263 <= send alert message
I (12328) mbedtls: ssl_tls.c:4311 => read record
I (12338) mbedtls: ssl_tls.c:2536 => fetch input
I (12348) mbedtls: ssl_tls.c:2697 in_left: 0, nb_want: 5
I (12348) mbedtls: ssl_tls.c:2721 in_left: 0, nb_want: 5
I (12358) mbedtls: ssl_tls.c:2722 ssl->f_recv(_timeout)() returned -80 (-0x0050)
W (12368) mbedtls: ssl_tls.c:4973 mbedtls_ssl_fetch_input() returned -80 (-0x0050)
W (12368) mbedtls: ssl_tls.c:4344 ssl_get_next_record() returned -80 (-0x0050)
W (12378) mbedtls: ssl_tls.c:8335 mbedtls_ssl_read_record() returned -80 (-0x0050)
E (12388) esp-tls: read error :-80:
Error: Time:Wed Oct 30 18:30:25 2019 File:/Users/fred/ESP/esp-azure/azure-iot-sdk-c/c-utility/adapters/httpapi_compact.c Func:readLine Line:591 Receive timeout. The HTTP request is incomplete
Error: Time:Wed Oct 30 18:30:25 2019 File:/Users/fred/ESP/esp-azure/azure-iot-sdk-c/c-utility/adapters/httpapi_compact.c Func:HTTPAPI_ExecuteRequest Line:1226 Receive header from HTTP failed (result = HTTPAPI_READ_DATA_FAILED)
Error: Time:Wed Oct 30 18:30:25 2019 File:/Users/fred/ESP/esp-azure/azure-iot-sdk-c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Wed Oct 30 18:30:25 2019 File:/Users/fred/ESP/esp-azure/azure-iot-sdk-c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:send_http_request Line:137 unable to HTTPAPIEX_ExecuteRequest
Error: Time:Wed Oct 30 18:30:25 2019 File:/Users/fred/ESP/esp-azure/azure-iot-sdk-c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadToBlob_step1and2 Line:405 unable to HTTPAPIEX_ExecuteRequest
Error: Time:Wed Oct 30 18:30:25 2019 File:/Users/fred/ESP/esp-azure/azure-iot-sdk-c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadMultipleBlocksToBlob_Impl Line:768 error in IoTHubClient_LL_UploadToBlob_step1
hello world failed to upload
I (32498) mbedtls: ssl_tls.c:8270 => read
I (32498) mbedtls: ssl_tls.c:4311 => read record
I (32508) mbedtls: ssl_tls.c:2536 => fetch input
I (32518) mbedtls: ssl_tls.c:2697 in_left: 0, nb_want: 5
I (32518) mbedtls: ssl_tls.c:2721 in_left: 0, nb_want: 5`
Can you post a file upload example? cert or not...
@frederic-thibault , can you share your application code?
Can you post a file upload example? cert or not...
The two following examples works fine with SAS. You must update the file port/CMakeLists.txt from esp-azure repo (include).
for the second example you must be careful with the number of iteration because it generate a xml (put block list) and it crashed when serialize it at the end. It is better to start with the first example has is.
Thanks,
`#
Component Makefile
set (AZURE_IOT_SDK "${CMAKE_CURRENT_LIST_DIR}/../azure-iot-sdk-c")
set (COMPONENT_ADD_INCLUDEDIRS
"inc"
"${AZURE_IOT_SDK}/certs"
"${AZURE_IOT_SDK}/c-utility/inc"
"${AZURE_IOT_SDK}/c-utility/pal/inc"
"${AZURE_IOT_SDK}/c-utility/pal/freertos"
"${AZURE_IOT_SDK}/c-utility/pal/generic"
"${AZURE_IOT_SDK}/iothub_client/inc"
"${AZURE_IOT_SDK}/serializer/inc"
"${AZURE_IOT_SDK}/umqtt/inc"
"${AZURE_IOT_SDK}/umqtt/inc/azure_umqtt_c"
"${AZURE_IOT_SDK}/deps/parson"
"${AZURE_IOT_SDK}/provisioning_client/inc"
"${AZURE_IOT_SDK}/provisioning_client/adapters"
"${AZURE_IOT_SDK}/provisioning_client/deps/utpm/inc"
)
set (COMPONENT_SRCS
"src/agenttime_esp.c"
"src/platform_esp.c"
"src/tlsio_esp_tls.c"
"${AZURE_IOT_SDK}/certs/certs.c"
"${AZURE_IOT_SDK}/c-utility/pal/freertos/lock.c"
"${AZURE_IOT_SDK}/c-utility/pal/dns_async.c"
"${AZURE_IOT_SDK}/c-utility/pal/socket_async.c"
"${AZURE_IOT_SDK}/c-utility/pal/freertos/threadapi.c"
"${AZURE_IOT_SDK}/c-utility/pal/freertos/tickcounter.c"
"${AZURE_IOT_SDK}/c-utility/pal/tlsio_options.c"
"${AZURE_IOT_SDK}/c-utility/src/xlogging.c"
"${AZURE_IOT_SDK}/c-utility/src/singlylinkedlist.c"
"${AZURE_IOT_SDK}/c-utility/src/buffer.c"
"${AZURE_IOT_SDK}/c-utility/src/consolelogger.c"
"${AZURE_IOT_SDK}/c-utility/src/constbuffer.c"
"${AZURE_IOT_SDK}/c-utility/src/constmap.c"
"${AZURE_IOT_SDK}/c-utility/src/crt_abstractions.c"
"${AZURE_IOT_SDK}/c-utility/src/doublylinkedlist.c"
"${AZURE_IOT_SDK}/c-utility/src/gballoc.c"
"${AZURE_IOT_SDK}/c-utility/src/gb_stdio.c"
"${AZURE_IOT_SDK}/c-utility/src/gb_time.c"
"${AZURE_IOT_SDK}/c-utility/src/hmac.c"
"${AZURE_IOT_SDK}/c-utility/src/hmacsha256.c"
"${AZURE_IOT_SDK}/c-utility/adapters/httpapi_compact.c"
"${AZURE_IOT_SDK}/c-utility/src/httpapiex.c"
"${AZURE_IOT_SDK}/c-utility/src/httpapiexsas.c"
"${AZURE_IOT_SDK}/c-utility/src/httpheaders.c"
"${AZURE_IOT_SDK}/c-utility/src/map.c"
"${AZURE_IOT_SDK}/c-utility/src/optionhandler.c"
"${AZURE_IOT_SDK}/c-utility/src/sastoken.c"
"${AZURE_IOT_SDK}/c-utility/src/sha1.c"
"${AZURE_IOT_SDK}/c-utility/src/sha224.c"
"${AZURE_IOT_SDK}/c-utility/src/sha384-512.c"
"${AZURE_IOT_SDK}/c-utility/src/strings.c"
"${AZURE_IOT_SDK}/c-utility/src/string_tokenizer.c"
"${AZURE_IOT_SDK}/c-utility/src/urlencode.c"
"${AZURE_IOT_SDK}/c-utility/src/usha.c"
"${AZURE_IOT_SDK}/c-utility/src/vector.c"
"${AZURE_IOT_SDK}/c-utility/src/xio.c"
"${AZURE_IOT_SDK}/c-utility/src/base64.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_device_client_ll.c"
"${AZURE_IOT_SDK}/iothub_client/src/blob.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_client_ll.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_client_core_ll.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_client_ll_uploadtoblob.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_device_client.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_client.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_client_core.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_client_authorization.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_client_retry_control.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_client_diagnostic.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_message.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothubtransport.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothubtransportmqtt.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothubtransport_mqtt_common.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub_transport_ll_private.c"
"${AZURE_IOT_SDK}/iothub_client/src/version.c"
"${AZURE_IOT_SDK}/umqtt/src/mqtt_client.c"
"${AZURE_IOT_SDK}/umqtt/src/mqtt_codec.c"
"${AZURE_IOT_SDK}/umqtt/src/mqtt_message.c"
"${AZURE_IOT_SDK}/deps/parson/parson.c"
"${AZURE_IOT_SDK}/serializer/src/codefirst.c"
"${AZURE_IOT_SDK}/serializer/src/agenttypesystem.c"
"${AZURE_IOT_SDK}/serializer/src/commanddecoder.c"
"${AZURE_IOT_SDK}/serializer/src/datamarshaller.c"
"${AZURE_IOT_SDK}/serializer/src/datapublisher.c"
"${AZURE_IOT_SDK}/serializer/src/dataserializer.c"
"${AZURE_IOT_SDK}/serializer/src/iotdevice.c"
"${AZURE_IOT_SDK}/serializer/src/jsondecoder.c"
"${AZURE_IOT_SDK}/serializer/src/jsonencoder.c"
"${AZURE_IOT_SDK}/serializer/src/methodreturn.c"
"${AZURE_IOT_SDK}/serializer/src/multitree.c"
"${AZURE_IOT_SDK}/serializer/src/schema.c"
"${AZURE_IOT_SDK}/serializer/src/schemalib.c"
"${AZURE_IOT_SDK}/serializer/src/schemaserializer.c"
"${AZURE_IOT_SDK}/provisioning_client/src/prov_device_client.c"
"${AZURE_IOT_SDK}/provisioning_client/src/prov_transport_mqtt_client.c"
"${AZURE_IOT_SDK}/provisioning_client/src/prov_transport_mqtt_common.c"
"${AZURE_IOT_SDK}/provisioning_client/src/prov_security_factory.c"
"${AZURE_IOT_SDK}/provisioning_client/src/prov_device_ll_client.c"
"${AZURE_IOT_SDK}/provisioning_client/src/iothub_security_factory.c"
"${AZURE_IOT_SDK}/provisioning_client/adapters/hsm_client_data.c"
"${AZURE_IOT_SDK}/provisioning_client/adapters/hsm_client_tpm.c"
"${AZURE_IOT_SDK}/provisioning_client/src/prov_auth_client.c"
"${AZURE_IOT_SDK}/provisioning_client/deps/utpm/src/tpm_codec.c"
"${AZURE_IOT_SDK}/provisioning_client/deps/utpm/src/Marshal.c"
"${AZURE_IOT_SDK}/provisioning_client/deps/utpm/src/tpm_comm_emulator.c"
"${AZURE_IOT_SDK}/provisioning_client/deps/utpm/src/Memory.c"
"${AZURE_IOT_SDK}/provisioning_client/deps/utpm/src/tpm_socket_comm.c"
"${AZURE_IOT_SDK}/iothub_client/src/iothub.c"
"${AZURE_IOT_SDK}/c-utility/src/http_proxy_io.c"
"${AZURE_IOT_SDK}/c-utility/src/base32.c"
)
if (CONFIG_DEVICE_COMMON_NAME)
list (APPEND COMPONENT_SRCS "${AZURE_IOT_SDK}/provisioning_client/src/iothub_auth_client.c")
endif()
set (COMPONENT_SUBMODULES "${AZURE_IOT_SDK}")
set(COMPONENT_PRIV_REQUIRES "mbedtls esp-tls main")
register_component()
component_compile_options (
-Wno-unused-function
-Wno-missing-braces
-Wno-missing-field-initializers
)
component_compile_definitions (
USE_LWIP_SOCKET_FOR_AZURE_IOT
HSM_TYPE_X509
HSM_TYPE_SAS_TOKEN
)
if (CONFIG_DEVICE_COMMON_NAME)
add_definitions(-DUSE_PROV_MODULE)
endif()
`
@frederic-thibault , can you share your application code?
Do you already have a project where you can connect to iot hub using x509 certificate and sent/receive messages?
If yes you can use the following project to test the upload (that s where I started). This project works fine with sas but not with certificate. If you dont have a project with certificate working, I can assist you to create one but I can't give you access to my azure service.
@shahpiyushv Can you make the Upload sample project works with x509 security??
@frederic-thibault , I have never really tried a file upload using either SAS or X509. I will first try with your example.
@shahpiyushv Hi, Do you have an idea of when you will look at this problem. Not being able to upload a file to IoT Hub is a show stopper for the project.
If you are not able to look at it I will have to find another solution.
Thanks.
Info: Waiting for TLS connection
Info: Waiting for TLS connection
Info: Waiting for TLS connection
Info: Waiting for TLS connection
E (19574) esp-tls: read error :-80:
Error: Time:Tue Nov 19 19:50:07 2019 File:D:/ESP/esp-azure/azure-iot-sdk-c/c-utility/adapters/httpapi_compact.c Func:readLine Line:593 Receive timeout. The HTTP request is incomplete
Error: Time:Tue Nov 19 19:50:07 2019 File:D:/ESP/esp-azure/azure-iot-sdk-c/c-utility/adapters/httpapi_compact.c Func:HTTPAPI_ExecuteRequest Line:1230 Receive header from HTTP failed (result = HTTPAPI_READ_DATA_FAILED (10))
Error: Time:Tue Nov 19 19:50:07 2019 File:D:/ESP/esp-azure/azure-iot-sdk-c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:526 unable to recover sending to a working state
Error: Time:Tue Nov 19 19:50:07 2019 File:D:/ESP/esp-azure/azure-iot-sdk-c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:send_http_request Line:137 unable to HTTPAPIEX_ExecuteRequest
Error: Time:Tue Nov 19 19:50:07 2019 File:D:/ESP/esp-azure/azure-iot-sdk-c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadToBlob_step1and2 Line:405 unable to HTTPAPIEX_ExecuteRequest
Error: Time:Tue Nov 19 19:50:07 2019 File:D:/ESP/esp-azure/azure-iot-sdk-c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadMultipleBlocksToBlob_Impl Line:768 error in IoTHubClient_LL_UploadToBlob_step1
hello world failed to upload
@frederic-thibault as I had mentioned earlier, I have never really tried file upload wither with sas or x.509. So, this is kinda new for me. In order to speed up, can you send me the code for the file upload application ported to ESP32 ? The above pointers you shared seem generic examples. It would also help if you can share a pointer for setting up stuff on the Azure IoT portal to enable the file uploading.
I've faced almost the same issue while was trying to upload file using x509 certs. I'd like to clarify one moment. Do you receive "error :-80:" after receiving such handshake?
D (66931) mbedtls: ssl_tls.c:3626 handshake message: msglen = 4, type = 0, hslen = 4
I (66931) mbedtls: ssl_tls.c:4385 <= read record
W (66941) mbedtls: ssl_tls.c:8357 received handshake message
D (66941) mbedtls: ssl_tls.c:8434 refusing renegotiation, sending alert
I (66951) mbedtls: ssl_tls.c:5250 => send alert message
D (66951) mbedtls: ssl_tls.c:5251 send alert level=1 message=100
I (66971) mbedtls: ssl_tls.c:3343 => write record
As you can see, renegotiation is refused. Even with CONFIG_MBEDTLS_SSL_RENEGOTIATION=y. If I try to set disable_renegotiation=MBEDTLS_SSL_RENEGOTIATION_ENABLED from mbedtls_ssl_config, upload works fine for me.
@nrmka , thanks for providing a solution for the issue. Applying this mbedtls_renegotiate.txt patch in esp-idf resolves this issue. I tested this with esp-idf master branch (commit id: 7d75213674b4572f90c68162ad6fe9b16dae65ad). I also had to apply this
upload_blob.txt patch in azure-iot-sdk-c to fix a crash. This seems to have been fixed in the upstream azure-iot-sdk-c already and we will soon update to it.
Sorry @frederic-thibault for this very late reply.
The issue has been fixed by updating the azure-sdk and also pushing out the mbedtls renegotiation fix in esp-tls to esp-idf master and release/4.0branches. Closing this now.