aws-greengrass/aws-greengrass-nucleus

SecureTunneling: Unable to open lockfile: /run/lock/devicecl.lock

Closed this issue · 2 comments

Staff-d commented

Describe the bug
I cannot connect to a secure tunnel via the aws.greengrass.SecureTunneling component, as the component crashes with Unable to open lockfile: /run/lock/devicecl.lock Secure tunnel process completed with exit code: 255.

To Reproduce
1.Install Amazon Linux 2 on a device of your choice (I used a kvm virtual machine).
2.Install the nucleus using automatic provisioning
3.Create a deployment which contains the SecureTunneling component, e.g.

{
   "targetArn":"arn:aws:iot:eu-central-1:[ACC]:thing/[DEVICE]",
   "deploymentName":"testDeployment2",
   "components":{
      "aws.greengrass.SecureTunneling":{
         "componentVersion":"1.0.9",
         "configurationUpdate":{
            "merge":"{\"OS_DIST_INFO\":\"amzn2\"}"
         }
      },
      "aws.greengrass.LogManager":{
         "componentVersion":"2.2.6",
         "configurationUpdate":{
            "merge":"{\"logsUploaderConfiguration\":{\"systemLogsConfiguration\":{\"uploadToCloudWatch\":\"true\",\"minimumLogLevel\":\"INFO\",\"diskSpaceLimit\":\"10\",\"diskSpaceLimitUnit\":\"MB\",\"deleteLogFileAfterCloudUpload\":\"false\"},\"componentLogsConfigurationMap\":{\"com.example.PythonHelloWorld\":{\"minimumLogLevel\":\"INFO\",\"diskSpaceLimit\":\"20\",\"diskSpaceLimitUnit\":\"MB\",\"deleteLogFileAfterCloudUpload\":\"false\"}}},\"periodicUploadIntervalSec\":\"300\"}"
         }
      },
      "aws.greengrass.Cli":{
         "componentVersion":"2.7.0"
      }
   }
}
  1. Create a secure tunnel and select a device as the destination to which the component was deployed. Choose SSH as the service.
  2. Check the logs of the SecureTunneling component on the destination device, e.g. by using sudo cat /greengrass/v2/logs/aws.greengrass.SecureTunneling.log. The logfile will look something like this
Logs 
2022-08-25T14:13:14.727Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [INFO ] 2022-08-25 14:13:14.726 [Thread-1] SecureTunnelingTask - Successfully subscribed to topic: $aws/things/[DEVICE]/tunnels/notify. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.356Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [INFO ] 2022-08-25 14:16:26.356 [Thread-1] SubscribeResponseHandler - Received new tunnel notification message.. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.365 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.365 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "key": "replace_with_private_key_file_location",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.366 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "jobs": {. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.366 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: },. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.365 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "cert": "replace_with_certificate_file_location",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.366 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "file": "/var/log/aws-iot-device-client/aws-iot-device-client.log". {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.366 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "type": "STDOUT",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.364 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2022-08-25T14:16:26.361Z [WARN] {Config.cpp}: Path replace_with_root_ca_file_location to RootCA is invalid. Ignoring... Will attempt to use default trust store.2022-08-25T14:16:26.361Z [WARN] {FileUtils.cpp}: Permissions to given file/dir path '/tmp/' is not set to recommended value... {Permissions: {desired: 745, actual: 777}}. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.365 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "endpoint": "replace_with_endpoint_value",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.366 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "level": "ERROR",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.366 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "logging": {. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.365 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2022-08-25T14:16:26.361Z [INFO] {Config.cpp}: Successfully fetched JSON config file: {. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.365 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "thing-name": "replace_with_thing_name",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.365 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "root-ca": "replace_with_root_ca_file_location",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.374Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.368 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: }. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.374Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.368 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "csr-file": "replace_with_csr-file-path". {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.374Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.368 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "template-name": "replace_with_template_name",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.373Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.367 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "enabled": true. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.373Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.367 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "enabled": false,. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.373Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.367 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "fleet-provisioning": {. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.373Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.367 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: },. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.373Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.367 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "interval": 300. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.366Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.366 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "enabled": false,. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.373Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.367 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "enabled": false,. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.373Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.367 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "device-defender": {. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.373Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.367 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: },. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.367Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.367 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "tunneling": {. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.367Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.366 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: },. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.367Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.366 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: "handler-directory": "replace_with_path_to_handler_dir". {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.374Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.368 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: }. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.374Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.368 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2022-08-25T14:16:26.361Z [ERROR] {Main.cpp}: *** AWS IOT DEVICE CLIENT FATAL ERROR: Error obtaining lockfile: Can not write to lockfile.. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.374Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.368 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2022-08-25T14:16:26.361Z [DEBUG] {Config.cpp}: Did not find a runtime configuration file, assuming Fleet Provisioning has not run for this device. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.374Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2022-08-25 14:16:26.368 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2022-08-25T14:16:26.361Z [ERROR] {LockFile.cpp}: Unable to open lockfile: /run/lock/devicecl.lock. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
Unable to parse log message:
2022-08-25T14:16:26.376Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [INFO ] 2022-08-25 14:16:26.376 [pool-3-thread-1] SubscribeResponseHandler - Secure tunnel process completed with exit code: 255. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
  1. Try to connect to the tunnel using localproxy and ssh. The connection will timeout.

Expected behavior
I expected the ssh connection to be established.

Actual behavior
The ssh connection could not be established.

Environment

  • OS: Amazon Linux 2 LTS 2.0.20220719.0
  • JDK version: java --version openjdk 11.0.13 2021-10-19 LTS OpenJDK Runtime Environment 18.9 (build 11.0.13+8-LTS) OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8-LTS, mixed mode, sharing)
  • Nucleus version: 2.7.0

Additional context
I've already verified that the sudoers file contains the line root ALL=(ALL:ALL) ALL
A temporary but ugly fix is running chmod 777 /run/lock. This works until the next reboot.
These also seems to be an environment variable to change the path where the lock file is written, but I have no idea how to configure that in the context of greengrass.

If this is not the correct repo to file this issue, feel free to point me somewhere else :)

MikeDombo commented

Hi @Staff-d,
Secure tunneling isn't part of this GitHub project. I would suggest that you create a support case through the AWS Console.

MikeDombo commented

The implementation of secure tunneling is in https://github.com/awslabs/aws-iot-device-client

  • 👍1