Argorollout Experiment Duration Should Override Success

[hussainbani]

Issue

When using the argorollout experiment type with Kubernetes Job, there's a parameter duration that specifies the time the experiment should wait before moving forward. However, the current behavior is such that even if the job executed by the experiment succeeds (exits with code 0), the experiment still waits for the specified duration before proceeding. This can lead to unnecessary delays, especially if the job completes quickly.

Expected Behavior

Ideally, if the job executed by the experiment succeeds, the duration parameter should be overridden, and the experiment should move forward immediately instead of waiting for the specified duration to finish.

Current Behavior

Currently, even if the job succeeds, the experiment waits for the duration specified in the parameter before proceeding.

Proposal

The behavior should be modified such that if the job executed by the experiment succeeds, the duration parameter is ignored, and the experiment moves forward immediately.

Additional Notes

While not defining a duration would result in the experiment running indefinitely in case of any looping within the job, having the duration parameter override success can improve the efficiency of experiments, especially for jobs that complete quickly.

Steps to Reproduce

Set up an argorollout experiment using a Kubernetes Job.
Specify a duration parameter.
Ensure the job executed by the experiment succeeds (exits with code 0).
Observe that the experiment still waits for the specified duration before proceeding.