Add functionality to restrict number of parallel runs for a workflow

[SayantaBhar]

Need to add new capability to restrict number of parallel running orchestration instances based on some configurable variable say 'max_active_runs'

E.g:

We have an orchestrator function called workflowOrchestrator.
WorkflowId1-1234567 is one orchestration instance.
WorkflowId2-7346382 is another orchestration instance.
Then If max_active_runs set for WorkflowId1 is 1 then no more than 1 instance should be able to spin up for that. That essentially means that there should not be another running instance of WorkflowOrchestrator with instanceId
WorkflowId1-< random number >. New instance for WorkflowId1 should eventually spin up only when WorkflowId1-1234567 is completed i.e once current active instances for a workflow is less than max_active_runs then only new instance should spin up.
However, if there is no such limitation on WorkflowId2, then any number of workflow orchestrators should be able to spin up with instanceIds Workflow2-< random number >

[cgillum]

We don't have any such controls at the host level. However, It is possible to implement this manually using singleton orchestrations.

For example, you could write a "WorkflowId1-manager" singleton orchestration which is responsible for scheduling all "WorkflowId1-XXX" workflows as sub-orchestrations. You could use that max_active_runs variable from inside "WorkflowId1-manager" to configure how many sub-orchestrations can run concurrently, and write the logic necessary to schedule new instances when old instances complete, etc.