salesforce/orchard

Orchard does not recreate resource when manually killed by external user

realstraw opened this issue · 1 comments

To reproduce:

Create a workflow with resource (e.g. EMR) that have more than 1 as max retry. Activating it, kill the resource through AWS console, orchard will mark workflow failed without retry creating the resource.

Note, the issue is that ResourceInstanceFinished message is used in ResourceMgr to get the final confirmation that resource is eventually cleaned up. In the event of user triggered termination, the ResourceMgr would not know if certain activity is running on it, hence send causing ResourceMgr to terminate itself, where activity is marked as "cenceled"