Make Spark job names unique

[robert-clegg-tessella]

Submitting a job request to the web app more than once with the same name input causes all but the first request to be rejected (409 error: conflict). This happens even if the first job has finished, and can only be resolved by deleting the first job.

A solution is to append a unique identifier tag to each spark job before passing it onto the Spark Operator.

Notes

• The user must be made aware of the new job name in the submit job response (as well as the driver name)
• Care must be taken that the name is not too long
  • driver/executor information is added to the pod names by the Spark Operator
  • validation on the length of the job name may be needed

Acceptance criteria

When submitting a job, the response notifies the user of the new job name: this will be the submitted job name, plus a unique identifier tag.

Subsequent requests to get the logs of, get the status of, or to delete this job must use the unique job name (using the original job name should result in a 404 Not Found response).

Multiple submissions of the same job name should result in a 200 Ok response, and a different unique job name each time.

Submitting a job name that is very long (>200 characters) should result in a 400 Bad Request response, with a message explaining why.

Test Scenarios

• Test that the user is informed with the new job name with a unique identifier.
• Test that a job name with 200+ characters return 400 status and a user-friendly error message
• Test that a job can be submitted with the same name multiple times NA
• Test that job name with different character casing is treated as the same name NA
• Test that 404 and a user-friendly error message is returned while using an invalid job name. FAIL Whitespace and special characters return 422 not 400 SHOULD BE FIXED WHEN #70 IS DONE - fix verified.
• Test numerate name FAIL e.g 0123 fails to run spark submit and job remains in failed state SHOULD BE FIXED WHEN #70 IS DONE - fix verified.