Console Deploy of custom component fails with 'BROKEN'

Question

Console Deploy of custom component fails with 'BROKEN'

Closed this issue a year ago · 4 comments

Describe the bug
Deployment of simple HelloWorld failing in console with 'BROKEN' message cannot rollback

To Reproduce
Steps to reproduce the behavior. If possible, provide a minimal amount of code that causes the bug.

Expected behavior
A clear and concise description of what you expected to happen.

Actual behavior
The local nucleus logs show some sort of file process issue?

023-08-02T16:22:58.652Z [INFO] (pool-2-thread-45) com.thio.lambda.HelloWorld: shell-runner-start. {scriptName=services.com.thio.lambda.HelloWorld.lifecycle.startup.script, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING, command=["/greengrass/v2/packages/artifacts/aws.greengrass.LambdaLauncher/2.0.10/lambda-..."]}
2023-08-02T16:22:58.696Z [INFO] (pool-2-thread-46) com.thio.lambda.HelloWorld: Setting up overlay tmpfs mounts. {serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING}
2023-08-02T16:22:58.713Z [WARN] (Copier) com.thio.lambda.HelloWorld: stderr. 2023/08/02 16:22:58 unable to setup worker process factories: The following cgroup subsystems are not mounted: devices, memory. {scriptName=services.com.thio.lambda.HelloWorld.lifecycle.startup.script, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING}
2023-08-02T16:22:58.718Z [INFO] (pool-2-thread-46) com.thio.lambda.HelloWorld: Finding mounted cgroups.. {serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING}
2023-08-02T16:22:58.743Z [INFO] (Copier) com.thio.lambda.HelloWorld: Startup script exited. {exitCode=1, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING}
2023-08-02T16:22:58.770Z [INFO] (pool-2-thread-41) com.thio.lambda.HelloWorld: shell-runner-start. {scriptName=services.com.thio.lambda.HelloWorld.lifecycle.shutdown.script, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STOPPING, command=["/greengrass/v2/packages/artifacts/aws.greengrass.LambdaLauncher/2.0.10/lambda-..."]}
2023-08-02T16:22:58.974Z [WARN] (Copier) com.thio.lambda.HelloWorld: stderr. 2023/08/02 16:22:58 could not read process state file /greengrass/v2/work/com.thio.lambda.HelloWorld/work/worker/8emiHLUIa_eiwtH8H0hrQ__uCR01mv-_yw3B56YuqIY/state.json: open /greengrass/v2/work/com.thio.lambda.HelloWorld/work/worker/8emiHLUIa_eiwtH8H0hrQ__uCR01mv-_yw3B56YuqIY/state.json: no such file or directory. {scriptName=services.com.thio.lambda.HelloWorld.lifecycle.shutdown.script, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STOPPING}
2023-08-02T16:22:59.097Z [ERROR] (pool-2-thread-42) com.thio.lambda.HelloWorld: error while removing dir {"path": "/greengrass/v2/work/com.thio.lambda.HelloWorld/work/worker/8emiHLUIa_eiwtH8H0hrQ__uCR01mv-_yw3B56YuqIY", "errorString": "unlinkat /greengrass/v2/work/com.thio.lambda.HelloWorld/work/worker/8emiHLUIa_eiwtH8H0hrQ__uCR01mv-_yw3B56YuqIY/overlays: device or resource busy"}. {serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STOPPING}
2023-08-02T16:22:59.243Z [INFO] (pool-2-thread-43) com.thio.lambda.HelloWorld: shell-runner-start. {scriptName=services.com.thio.lambda.HelloWorld.lifecycle.startup.script, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING, command=["/greengrass/v2/packages/artifacts/aws.greengrass.LambdaLauncher/2.0.10/lambda-..."]}
2023-08-02T16:22:59.287Z [INFO] (pool-2-thread-42) com.thio.lambda.HelloWorld: Setting up overlay tmpfs mounts. {serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING}
2023-08-02T16:22:59.288Z [INFO] (pool-2-thread-42) com.thio.lambda.HelloWorld: Finding mounted cgroups.. {serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING}
2023-08-02T16:22:59.288Z [WARN] (Copier) com.thio.lambda.HelloWorld: stderr. 2023/08/02 16:22:59 unable to setup worker process factories: The following cgroup subsystems are not mounted: devices, memory. {scriptName=services.com.thio.lambda.HelloWorld.lifecycle.startup.script, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING}
2023-08-02T16:22:59.313Z [INFO] (Copier) com.thio.lambda.HelloWorld: Startup script exited. {exitCode=1, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING}
2023-08-02T16:22:59.325Z [INFO] (pool-2-thread-43) com.thio.lambda.HelloWorld: shell-runner-start. {scriptName=services.com.thio.lambda.HelloWorld.lifecycle.shutdown.script, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STOPPING, command=["/greengrass/v2/packages/artifacts/aws.greengrass.LambdaLauncher/2.0.10/lambda-..."]}
2023-08-02T16:22:59.488Z [WARN] (Copier) com.thio.lambda.HelloWorld: stderr. 2023/08/02 16:22:59 could not read process state file /greengrass/v2/work/com.thio.lambda.HelloWorld/work/worker/8emiHLUIa_eiwtH8H0hrQ__uCR01mv-_yw3B56YuqIY/state.json: open /greengrass/v2/work/com.thio.lambda.HelloWorld/work/worker/8emiHLUIa_eiwtH8H0hrQ__uCR01mv-_yw3B56YuqIY/state.json: no such file or directory. {scriptName=services.com.thio.lambda.HelloWorld.lifecycle.shutdown.script, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STOPPING}
2023-08-02T16:22:59.664Z [ERROR] (pool-2-thread-46) com.thio.lambda.HelloWorld: error while removing dir {"path": "/greengrass/v2/work/com.thio.lambda.HelloWorld/work/worker/8emiHLUIa_eiwtH8H0hrQ__uCR01mv-_yw3B56YuqIY", "errorString": "unlinkat /greengrass/v2/work/com.thio.lambda.HelloWorld/work/worker/8emiHLUIa_eiwtH8H0hrQ__uCR01mv-_yw3B56YuqIY/overlays: device or resource busy"}. {serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STOPPING}
2023-08-02T16:22:59.701Z [INFO] (pool-2-thread-43) com.thio.lambda.HelloWorld: shell-runner-start. {scriptName=services.com.thio.lambda.HelloWorld.lifecycle.startup.script, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING, command=["/greengrass/v2/packages/artifacts/aws.greengrass.LambdaLauncher/2.0.10/lambda-..."]}
2023-08-02T16:22:59.742Z [INFO] (pool-2-thread-41) com.thio.lambda.HelloWorld: Setting up overlay tmpfs mounts. {serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING}
2023-08-02T16:22:59.742Z [WARN] (Copier) com.thio.lambda.HelloWorld: stderr. 2023/08/02 16:22:59 unable to setup worker process factories: The following cgroup subsystems are not mounted: devices, memory. {scriptName=services.com.thio.lambda.HelloWorld.lifecycle.startup.script, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING}
2023-08-02T16:22:59.742Z [INFO] (pool-2-thread-41) com.thio.lambda.HelloWorld: Finding mounted cgroups.. {serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING}
2023-08-02T16:22:59.771Z [INFO] (Copier) com.thio.lambda.HelloWorld: Startup script exited. {exitCode=1, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=STARTING}
2023-08-02T16:22:59.773Z [INFO] (pool-2-thread-43) com.thio.lambda.HelloWorld: shell-runner-start. {scriptName=services.com.thio.lambda.HelloWorld.lifecycle.shutdown.script, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=BROKEN, command=["/greengrass/v2/packages/artifacts/aws.greengrass.LambdaLauncher/2.0.10/lambda-..."]}
2023-08-02T16:22:59.889Z [WARN] (Copier) com.thio.lambda.HelloWorld: stderr. 2023/08/02 16:22:59 could not read process state file /greengrass/v2/work/com.thio.lambda.HelloWorld/work/worker/8emiHLUIa_eiwtH8H0hrQ__uCR01mv-_yw3B56YuqIY/state.json: open /greengrass/v2/work/com.thio.lambda.HelloWorld/work/worker/8emiHLUIa_eiwtH8H0hrQ__uCR01mv-_yw3B56YuqIY/state.json: no such file or directory. {scriptName=services.com.thio.lambda.HelloWorld.lifecycle.shutdown.script, serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=BROKEN}
2023-08-02T16:23:00.006Z [ERROR] (pool-2-thread-44) com.thio.lambda.HelloWorld: error while removing dir {"path": "/greengrass/v2/work/com.thio.lambda.HelloWorld/work/worker/8emiHLUIa_eiwtH8H0hrQ__uCR01mv-_yw3B56YuqIY", "errorString": "unlinkat /greengrass/v2/work/com.thio.lambda.HelloWorld/work/worker/8emiHLUIa_eiwtH8H0hrQ__uCR01mv-_yw3B56YuqIY/overlays: device or resource busy"}. {serviceInstance=0, serviceName=com.thio.lambda.HelloWorld, currentState=BROKEN}
~

Environment

OS: [e.g. Ubuntu 20.04]
JDK version:
Nucleus version:

Additional context
Add any other context about the problem here.

E.g. what is the impact of the bug?

Answer 1 · 2023-08-02T16:31:53.000Z

Hi,

The error you are seeing is "The following cgroup subsystems are not mounted: devices, memory."

This error and approaches to fix it is listed in our troubleshooting documentation: https://docs.aws.amazon.com/greengrass/v2/developerguide/troubleshooting.html#lambda-cgroups-not-mounted.

Your device does not have CGroup V1 enabled most likely, it has CGroup V2.

I'd recommend that you use standard Greengrass V2 components rather than a lambda. Here is a tutorial for creating a standard component: https://docs.aws.amazon.com/greengrass/v2/developerguide/defer-component-updates-tutorial.html.

Your options are:

Don't use lambda
Don't use container mode lambda
enable cgroup v1
- https://repost.aws/questions/QUf-RGFJnDQHCadC0-dW7xdA/the-following-cgroup-subsystems-are-not-mounted-devices-memory

Answer 2 · 2023-08-02T19:18:52.000Z

Thank you. I did get this to work finally. In addition to what you mentioned I had to create a ln to node that nucleus expects -- (e.g. /usr/bin/nodejs18.x).

Answer 3 · 2023-08-02T19:21:19.000Z

Correct yes. I'm glad it is working for you. Your lambda name is "helloWorld", so I'm guessing that you're new to Greengrass? If you are new to Greengrass and you are not migrating from Greengrass v1 to v2, then I will reiterate my recommendation that you create a native Greengrass v2 component. Here is a tutorial for creating a standard component: https://docs.aws.amazon.com/greengrass/v2/developerguide/defer-component-updates-tutorial.html.

Closing issue.

Answer 4 · 2023-08-02T19:21:52.000Z

Thank you (and yes I am) :)