Azure/aztk

Align cluster and job implementations

jafreck opened this issue · 0 comments

Job and cluster implementations should be aligned to ease transition between the two modes for users. This will also decrease code complexity/duplication and facilitate support.

Currently a cluster is a: Batch pool and job.
A job is a: Batch job schedule with a job specification that has an autopool and predefined Tasks for each application.

Going forward, we should convert clusters to job specifications with autopool. So a cluster will be a job with an empty set of Batch tasks and keep_alive set to True on the autopool specification.

API Changes:

  • job configuration has a cluster configuration and a list of applications

Refactors:

  • cluster is implemented as job with keep_alive set to false and no applications defined
  • add plugin support to jobs