Using the `service --restart` command within an Octopus Script step may appear to fail

[rhysparry]

Team

• I've assigned a team label to this issue

What happened?

When using Octopus to run a script that uses Tentacles service --restart command the command may appear to fail, although the Tentacle will still be restarted.

On Windows, this may sometimes succeed or fail with an exit code of -46. Logs will likely indicate that the restart was successful.

On Linux, the script will fail with an exit code of 1 (although -46 may occur in rare circumstances). Logs will likely indicate that the restart was unsuccessful and that systemctl restart Tentacle failed with exit code: 143.

In both instances, the Tentacle has mostly likely restarted successfully.

The -46 exit code represents a tentacle starting and completing a script execution from the last time the tentacle was running. It indicates that the exit code of the script is unknown.

On Linux, with the default systemd configuration installed by Tentacle, restarting the Terminal will send the TERM signal to the tentacle and all child subprocesses, including the currently executing script and the call to systemctl restart Tentacle, which returns 143 to indicate that it has received a TERM signal. Because the script has already initiated the restart, the Tentacle will restart.

Reproduction

Use a "Run a Script" step to restart the tentacle using the command:

tentacle service --restart

Error and Stacktrace

No response

More Information

No response

Workaround

On Linux, it is possible to work around this problem by leveraging the restart capabilities provided by systemd.

• systemd sets an environment variable INVOCATION_ID when it starts a service. By capturing this value before restarting it can be compared to the value after restarting the service.
• Instead of using the Tentacle service --restart command, use kill to send a TERM signal to the Tentacle process. After the Tentacle has terminated, systemd will restart it.

This assumes the default service configuration. Example scripts (keep these in separate steps to ensure they are run correctly):

Get Initial Invocation ID

set_octopusvariable "OriginalInvocationId" $INVOCATION_ID

Terminate the current Tentacle process (note: you will need to update the process id step if you are running multiple instances of Tentacle)

TENTACLE_PID=$(ps aux | grep '/Tentacle\srun' | gawk '{ print $2 }')

if [ -z "$TENTACLE_PID" ]; then
    echo "Failed to find tentacle" >&2
    exit 1
fi

echo "Sending TERM to Process $TENTACLE_PID"

kill -s TERM "$TENTACLE_PID"

Verify the Tentacle is running with a new invocation ID

if [ "$(get_octopusvariable "Octopus.Action[Get Initial Invocation ID].Output.OriginalInvocationId")" != "$INVOCATION_ID" ]; then
	echo "No, it was successful."
else
	echo "Yes, it did fail"
fi