Force fastFail to work for concurrent cases in Jenkins Pipeline

[TerrenceMcGuinness-NOAA]

What new functionality do you need?

Currently in the Jenkins Pipeline fastFail does not work with the concurrent cases that are running because the system will not kill long running jobs in the shell scripts running on remote machines. Additional code to the pipeline needs to be added that can allow these jobs on the remote machine to run in the background with the PIDs stored so they can be killed force the fastFail behavior.

What are the requirements for the new functionality?

• fastFail works for when a case fails
• This feature should be configurable so fastFail can be enforced not by the code manager at will.

Acceptance Criteria

The behavior can be demonstrated and tested and works on restart in a full end-to-end CI self test

Suggest a solution (optional)

• Run run-check_ci.sh in the background with & and capture the PID in a list for each
• In the try catch kill all those jobs and cancel all the batch processes for the entire CI
• Run FINALIZE with the fail state saved and acted on accordingly