aws-iot-device-client fails with "invalid pointer"

[perezmeyer]

I have build the hardknott version on a VAR-DART-MX8M-PLUS board. Build was straightforward, but after configuring /etc/aws-iot-device-client.json and trying to run the app I get:

$ aws-iot-device-client --config-file /etc/aws-iot-device-client.json 
munmap_chunk(): invalid pointer

Is there anything I can do in order to debug this further?

[thomas-roos]

Hi, we are currently try to reproduce and fix this error. Do you have a special aws-iot-device-client.json or is it the default one?

[perezmeyer]

Hi Thomas! I had to create it by hand using the parameters specified by mi IT colleague.

Here is an anonymized copy:

cat /etc/aws-iot-device-client/aws-iot-device-client.conf 
{
        "endpoint": "a3kyhtb3rxud7g-ats.iot.us-east-1.amazonaws.com",
        "cert": "/home/root/keys/<a code>-certificate.pem.crt",
        "key": "/home/root/keys/<a code>-public.pem.key",
        "root-ca": "/home/root/keys/AmazonRootCA1.pem",
        "thing-name": "dart",
        "logging": {
                "level": "DEBUG",
                "type": "FILE",
                "file": "/var/log/aws-iot-device-client/aws-iot-device-client.log"
        },
        "jobs": {
                "enabled": false,
                "handler-directory": "/root/handler"
        },
        "tunneling": {
                "enabled": false
        },
        "device-defender": {
                "enabled": true,
                "interval": 300
        },
        "fleet-provisioning": {
                "enabled": false,
                "template-parameters": "<replace_with_template_parameters>",
                "template-name": "<replace_with_template_name>",
                "csr-file": "<replace_with_csr_file_path>",
                "device-key": "<replace_with_device_private_key_file_path>"
        },
        "samples": {
                "pub-sub": {
                        "enabled": false,
                        "publish-topic": "/topic/dc/pub",
                        "publish-file": "/var/log/pub.txt",
                        "subscribe-topic": "/topic/dc/sub",
                        "subscribe-file": "/var/log/sub.txt"
                }
        },
        "config-shadow": {
                "enabled": false
        },
        "sample-shadow": {
                "enabled": false,
                "shadow-name": "<replace_with_shadow_name>",
                "shadow-input-file": "<replace_with_shadow_input_file_path>",
                "shadow-output-file": "<replace_with_shadow_output_file_path>"
        }
}

[perezmeyer]

Maybe worth to mention: I tried to debug it but the packaging does not creates a suitable -dbg package. Well the package is there but literally empty (no, there are no hidden files).

[thomas-roos]

Could investigate on your problem in master and hardknott - both are working for me (after having same issues as you):

• go to AWS IoT website
  download package and take the files to /etc/aws-iot-device-client/

• download root ca
  curl https://www.amazontrust.com/repository/AmazonRootCA1.pem > root-CA.crt as well

• Thing name is in start.sh

• IMPORTANT: go to THINGS select Certificate edit policy - for testing * * should be ok otherwise will see disconnect ERROR in logfile

• have a nameserver eg. echo nameserver 8.8.4.4 >> /etc/resolv.conf

• have a minimal configuration like this (default one will cause that segmentation fault - as descriped here between the lines):

root@qemux86-64:/etc/aws-iot-device-client# cat aws-iot-device-client.conf 
{
        "endpoint": "XXX.amazonaws.com",
        "cert": "/etc/aws-iot-device-client/XXX.cert.pem",
        "key": "/etc/aws-iot-device-client/XXX.private.key",
        "root-ca": "/etc/aws-iot-device-client/root-CA.crt",
        "thing-name": "XXX",
        "logging": {
                "level": "DEBUG",
                "type": "FILE",
                "file": "/var/log/aws-iot-device-client.log"
        }
}

proper file/dir permissions:

chmod 700 /etc/aws-iot-device-client/
chmod 644 /etc/aws-iot-device-client/*
chmod 600 /etc/aws-iot-device-client/XXX.private.key

Starting:
aws-iot-device-client --config-file /etc/aws-iot-device-client/aws-iot-device-client.conf --enable-sdk-logging

Logfiles:
tail -f /var/log/aws-iot-device-client.log
tail -f /var/log/aws-iot-device-client/sdk.log

P.S: I plan an upgrade to 1.6. in master soon, I will fix the debug problem there as well. There must be RelWithDebInfo: Release optimizations with debug info as CMAKE_BUILD_TYPE in the recipe

[thomas-roos]

please reopen if this will not fix your problem

[perezmeyer]

Thanks, I'll try ASAP.

[perezmeyer]

Well, things improved, but we are still not there.

root@imx8mp-var-dart:/etc/aws-iot-device-client# /sbin/aws-iot-device-client --config-file /etc/aws-iot-device-client/aws-iot-device-client.conf --enable-sdk-logging --log-level DEBUG
[ 1541.108802] audit: type=1701 audit(1654190267.441:31): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=1160 comm="aws-iot-device-" exe="/sbin/aws-iot-device-client" sig=6 res=1                                                
AWS IoT Device Client must abort execution, reason: Failed to establish MQTT connection due to credential/configuration error
Please check the AWS IoT Device Client logs for more information
Aborted (core dumped)
root@imx8mp-var-dart:/etc/aws-iot-device-client# cat /var/log/aws-iot-device-client.log 
2022-06-02T17:17:47.440Z [WARN]  {FileUtils.cpp}: Permissions to given file/dir path '/etc/aws-iot-device-client/' is not set to recommended value... {Permissions: {desired: 745, actual: 700}}
2022-06-02T17:17:47.440Z [INFO]  {Config.cpp}: Successfully fetched JSON config file: {
        "endpoint": "a3k[foo]-ats.iot.us-east-1.amazonaws.com",
        "cert": "/etc/aws-iot-device-client/[foo]0d40eae007b1e30c44def41b790576e4b0736f68827f4-certificate.pem.crt",
        "key": "/etc/aws-iot-device-client/[foo]44def41b790576e4b0736f68827f4-public.pem.key",
        "root-ca": "/etc/aws-iot-device-client/AmazonRootCA1.pem",
        "thing-name": "dart",
        "logging": {
                "level": "DEBUG",
                "type": "FILE",
                "file": "/var/log/aws-iot-device-client.log"
        }
}

2022-06-02T17:17:47.440Z [DEBUG] {Config.cpp}: Did not find a runtime configuration file, assuming Fleet Provisioning has not run for this device
2022-06-02T17:17:47.440Z [INFO]  {Main.cpp}: Now running AWS IoT Device Client version v1.4.0
2022-06-02T17:17:47.444Z [INFO]  {SharedCrtResourceManager.cpp}: SDK logging is enabled. Check /var/log/aws-iot-device-client/sdk.log for SDK logs.
2022-06-02T17:17:47.444Z [DEBUG] {Retry.cpp}: Retryable function starting, it will retry until success
2022-06-02T17:17:47.445Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Client Configuration initialization failed with error: aws-c-io: AWS_IO_TLS_CTX_ERROR, Failed to create tls context
2022-06-02T17:17:47.445Z [ERROR] {Main.cpp}: *** AWS IOT DEVICE CLIENT FATAL ERROR: Failed to establish the MQTT Client. Please verify your AWS IoT credentials, configuration and/or certificate policy. ***

[thomas-roos]

Hi, great to hear that things are improved. But as I'm not an expert for aws-iot-device-client I would friendly ask you to open a bug/discussion at their github page https://github.com/awslabs/aws-iot-device-client
The only thing that might help is the correct setting of this - but I guess you did that.
IMPORTANT: go to THINGS select Certificate edit policy - for testing * * should be ok otherwise will see disconnect ERROR in logfile

[perezmeyer]

Thanks a lot for your help!