Training and Generalization

[kauray]

As mentioned, Decima is trained by capping the number of incoming jobs to 2000. In Appendix I, Generalizing the neural network model is described where a scaled-down version of workload is used.

As per the neural network architecture, when Decima is trained by limiting the number of jobs to 2000, would the same exact trained neural network be used for deployment in test? How does this scale to a larger number of jobs, ie, how would the neural network structure look like if the number of test jobs are larger?

[hongzimao]

Notice that Decima is evaluated with number of incoming jobs at least 10x that during training. The example training and testing script in README.md also indicates this point: the training has tag --num_stream_dags 200 while testing has tag --num_stream_dags 5000. The way to think about whether 200 or 2000 jobs during training is enough is that we need to make sure the agent experience a "full cycle" of workload, meaning, sees the empty system gets loaded by incoming jobs and then drain the jobs to empty (might repeat this ups and downs for multiple times).