Number of samples to approximate the expectation

[luxius-luminus]

Hello,
When computing the Zen-Score it involves expectations over input x and parameters theta.
In ZeroShotProxy/compute_zen_score.py, the approximation of such expectations is controlled by argument repeat in compute_nas_score, which is the number of samples from (x, theta).
Although the default value of repeat is 32 in ZeroShotProxy/compute_zen_score.py, it is set to be 1 (hard-coded) when calling compute_nas_score in evolution_search.py

To reproduce your results, should I keep using repeat=1? I am not sure if it works to approximate the expectation with only one sample. But using larger size of sample increases the search budget.

[MingLin-home]

Hi luxius,

We use numerical average to approximate the expectation of Zen-score. So the total batch size is usually 16~32. However, this value also depends on the variance of the Zen-score, which then depends on the current model under evaluation.

If not sure, setting repeat=1 and mini_batch_size=16 usually works well.

[luxius-luminus]

I get it. Many thanks.