batchsize of initial_random runs is unexpected

[TimoSommer]

Hey,
again me, but this is a different issue so I didn't want to merge it with the other one. I noticed that when setting initial_random e.g. to 20 and batch_size to 5, the size of the very first hyperparameter batch is initial_random, not batch_size. You can see this in the following minimal working example:

from mango import Tuner, scheduler
import numpy as np

def objective(hparam_batches): 
    size = len(hparam_batches)
    print('Length:', size)
    # The first time, len(hparam_batches) doesn't equal batch_size but initial_random!
    return hparam_batches, np.random.randn(size)

configs = {
            'initial_random': 20,
            'num_iteration': 10,
            'batch_size': 5
            }
param_space = {'foo': range(100, 200)}
tuner = Tuner(param_space, objective, configs)
results = tuner.minimize()

I understand that this can be useful behaviour since you can parallelize the random runs without any constraint. However, usually if I set a batch_size of 5 this means that I will have asked my Slurm scheduler for an allocation of 5 GPUs to evaluate my networks in parallel, even for the random runs. Therefore this behaviour comes very unexpected to me. It would be nice if you could make this clearer in the documentation.
Additionally, I would expect it's probably quite common that people would wish the batchsize to be always the same. Therefore, maybe it would be worth to offer a flag to turn this behaviour off and on. But maybe this is not true in general.

Cheers,
Timo

[sandeep-iitr]

Hi Timo,
Thanks for pointing this out.
We will update the documentation to clarify this behavior.

Your other suggestion is also welcome, and we will see if we can make this flag.

regards,
Sandeep

[TimoSommer]

Thank you very much! Great work!