Disconnect with AWS IoT Gracefully
law-ko opened this issue · 6 comments
Hello,
We are trying to disconnect with AWS IoT MQTT broker after the provisioning session is complete. However, it seems like it will crash due to MQTT_ProcessLoop is still running.
We are using xTlsDisconnect() to disconnect.
What is the good way to disconnect correctly and reconnect to AWS IoT again?
Hey @law-ko, thanks for reaching out about this issue as well. Is this by any chance related to your other issue #44?
But to follow the issue you are facing, you're saying that you're connecting to the IoT MQTT broker, and then disconnecting, and then seeing a crash? Do you by any chance have a stack trace showing what the flow of execution for this looks like?
Hi @Skptak ,
We further investigated into this and still believes the MQTTAgent_CommandLoop might be running in the background which leads to a LoadProhibited crash when calling xTlsDisconnect(pxNetworkContext).
Here's our code of trying to disconnect from AWS IoT:
/* If a connection was previously established, close it to free memory. */
if( ( pxNetworkContext != NULL ) && ( pxNetworkContext->pxTls != NULL ) )
{
ESP_LOGW( TAG, "Attempting to close previous TLS connection." );
xTlsDisconnect( pxNetworkContext ); // seems like crash happens here due to pxNetworkContext
vTaskDelay(500 / portTICK_PERIOD_MS);
ESP_LOGI( TAG, "TLS connection was disconnected." );
vTaskDelay(500 / portTICK_PERIOD_MS);
}
ESP_LOGW(TAG, "Before BackoffAlgorithm_InitializeParams");
BackoffAlgorithm_InitializeParams( &xReconnectParams,
configRETRY_BACKOFF_BASE_MS,
configRETRY_MAX_BACKOFF_DELAY_MS,
BACKOFF_ALGORITHM_RETRY_FOREVER );
ESP_LOGW(TAG, "After BackoffAlgorithm_InitializeParams");
do
{
ESP_LOGW( TAG, "Attempting to connect to MQTT broker via xTlsConnect" );
xTlsRet = xTlsConnect( pxNetworkContext );
vTaskDelay(500 / portTICK_PERIOD_MS);
ESP_LOGW( TAG, "After xTlsConnect" );
if( xTlsRet == TLS_TRANSPORT_SUCCESS )
{
eMqttRet = prvCoreMqttAgentConnect( xCleanSession );
if( eMqttRet != MQTTSuccess )
{
ESP_LOGE( TAG,
"MQTT_Status: %s",
MQTT_Status_strerror( eMqttRet ) );
}
}
if( eMqttRet != MQTTSuccess )
{
xTlsDisconnect( pxNetworkContext );
xBackoffRet = prvBackoffForRetry( &xReconnectParams );
}
} while( ( eMqttRet != MQTTSuccess ) && ( xBackoffRet == pdPASS ) );
Below is the crash log:
W (64145) core_mqtt_agent_manager: Attempting to close previous TLS connection.
Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC : 0x402082a6 PS : 0x00060230 A0 : 0x8010e6ec A1 : 0x3f814a60
0x402082a6: esp_tls_conn_write at /Users/esp/esp-idf/components/esp-tls/esp_tls.c:135
A2 : 0x00000000 A3 : 0x3f814adc A4 : 0x00000002 A5 : 0x00000001
A6 : 0xb33fffff A7 : 0xb33fffff A8 : 0x00000700 A9 : 0x3f814a20
A10 : 0x3ffc3ad4 A11 : 0x00060420 A12 : 0x00000004 A13 : 0x00060423
A14 : 0xb33fffff A15 : 0xb33fffff SAR : 0x00000008 EXCCAUSE: 0x0000001c
EXCVADDR: 0x00000728 LBEG : 0x4000c46c LEND : 0x4000c477 LCOUNT : 0x00000000
0x4000c46c: memset in ROM
0x4000c477: memset in ROM
Backtrace: 0x402082a3:0x3f814a60 0x4010e6e9:0x3f814a80 0x40110259:0x3f814aa0 0x401106d5:0x3f814ad0 0x401111b9:0x3f814b00 0x4011164e:0x3f814b80 0x40113096:0x3f814ba0 0x400df239:0x3f814c00 0x400949c2:0x3f814c20
0x402082a3: esp_tls_conn_write at /Users/esp/esp-idf/components/esp-tls/esp_tls.c:135
0x4010e6e9: espTlsTransportSend at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreHTTP/port/network_transport/network_transport.c:79
0x40110259: sendBuffer at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreMQTT/coreMQTT/source/core_mqtt.c:861
0x401106d5: MQTT_Ping at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreMQTT/coreMQTT/source/core_mqtt.c:2900
0x401111b9: handleKeepAlive at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreMQTT/coreMQTT/source/core_mqtt.c:1352
(inlined by) receiveSingleIteration at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreMQTT/coreMQTT/source/core_mqtt.c:1698
0x4011164e: MQTT_ProcessLoop at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreMQTT/coreMQTT/source/core_mqtt.c:3065
0x40113096: processCommand at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreMQTT-Agent/coreMQTT-Agent/source/core_mqtt_agent.c:611
(inlined by) MQTTAgent_CommandLoop at /Users/esp-git/esp-matter/libraries/esp-aws-iot/libraries/coreMQTT-Agent/coreMQTT-Agent/source/core_mqtt_agent.c:1059
0x400df239: prvMQTTAgentTask at /Users/esp-git/esp-matter/examples/outlet/main/aws_iot/networking/mqtt/core_mqtt_agent_manager.c:530
0x400949c2: vPortTaskWrapper at /Users/esp/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:162
And for esp_tls.c at line 135:
ssize_t esp_tls_conn_write(esp_tls_t *tls, const void *data, size_t datalen)
{
return tls->write(tls, (char *)data, datalen); // line 135
}
How should we call xTlsDisconnect gracefully? We do not want to kill MQTTAgent as we are reconnecting afterwards.
Hey @law-ko, thanks for letting me know that you're still seeing this issue!
There are a couple different issues being opened in regards to the mqtt process loop at this point
FreeRTOS/coreMQTT-Agent#104, #63, and #44
The team is taking a look into all of these. Unfortunately, it can be hard to pinpoint exactly what is going wrong since many of these issues can be hard to reproduce. I'm going to be trying to replicate some of these issues over the next few days and will see if I can get back to you about it
Thanks in advance for your patience!
I think your assessment is correct that you are disconnecting the underlying TLS connection without stopping the MQTT Agent.
Assuming that your implementation is similar to core_mqtt_agent_manager.c which uses xNetworkEventGroup to control when the MQTT agent task should call MQTTAgent_CommandLoop, you can stop the MQTT Agent like the following before disconnecting the TLS connection:
/* If a connection was previously established, close it to free memory. */
if( ( pxNetworkContext != NULL ) && ( pxNetworkContext->pxTls != NULL ) )
{
/* Ensure that the MQTT Agent task stops calling MQTTAgent_CommandLoop. */
xEventGroupClearBits( xNetworkEventGroup,
CORE_MQTT_AGENT_CONNECTED_BIT );
xEventGroupSetBits( xNetworkEventGroup,
CORE_MQTT_AGENT_DISCONNECTED_BIT );
/* Close the underlying TLS connection. */
ESP_LOGW( TAG, "Attempting to close previous TLS connection." );
xTlsDisconnect( pxNetworkContext );
ESP_LOGI( TAG, "TLS connection was disconnected." );
}
If your implementation is different, please share your complete implementation.
Hello,
I am closing this issue as there hasn't been any update.
Please feel free to reopen this issue or create a new one.
Thanks