Entire parameter space won't get explored
Opened this issue · 0 comments
Hello, I've been using carbs with decent success but noticed some parameter spaces were not getting explored very well. Here are my parameters:
params = [
Param(
name="total_timesteps",
space=LinearSpace(min=10_000_000, max=50_000_000, scale=10_000_000, is_integer=True),
search_center=20_000_000,
),
Param(name="learning_rate", space=LogSpace(min=1e-5, max=1e-1), search_center=3e-05),
Param(name="gamma", space=LogitSpace(min=0.8, max=0.9999), search_center=0.995),
Param(name="gae_lambda", space=LogitSpace(min=0.8, max=1.0), search_center=0.98),
Param(
name="update_epochs", space=LinearSpace(min=1, max=15, scale=5, is_integer=True), search_center=5
),
Param(name="clip_coef", space=LogitSpace(min=0.1, max=0.4), search_center=0.2),
Param(name="ent-coef", space=LogSpace(min=1e-5, max=1e-1), search_center=7e-03),
Param(name="vf_coef", space=LogitSpace(min=0.0, max=1.0), search_center=0.5),
Param(name="vf_clip_coef", space=LogitSpace(min=0.1, max=0.4), search_center=0.1),
Param(name="max_grad_norm", space=LinearSpace(min=0, max=5), search_center=1),
Param(
name="batch_size",
space=LinearSpace(min=65_536, max=1_048_576, scale=270_000, is_integer=True),
search_center=98_304,
),
Param(
name="minibatch_size",
space=LinearSpace(min=128, max=65_536, scale=20_000, is_integer=True),
search_center=512,
),
]
I left the initial_search_radius
at the default of 0.3
. I saw in the README and in a warning that I needed to adjust the scale of Linear spaces so I started performing some tests to try and figure out a good scale value. I would initialize carbs with 1 parameter space, set num_random_samples
to 1000, gather 1000 suggestions and take note of the min, max, mean, and standard deviation. I did this with the initial_search_radius
at 1.0
because as I understand it, carbs will increase the search radius as more observations suggestions are made. My thought was to find a scale value that will allow carbs to search the entire Linear space.
This worked OK in practice, for example, the minimum suggestions for the update_epochs
parameter was 2 and the max was 8 after 24 suggestions (12 of which were random). I was surprised to see several Log and Logit parameter spaces not explored very much though. gamma
for example had a min suggested value of 0.9821
and a max of 0.9987
. I'm assuming this has to do with the search center being set so close to the max.
All this to say, I don't entirely understand how scale
is supposed to be set and how it interacts with initial_search_radius
. How should I set scale
so that the min and max values of a space can be reached, in all types of parameter spaces? 'Set it >3' wasn't enough information for me to find good scale values for Linear spaces, and I have no idea how to set the scale for Log and Logit spaces, or if I even need to, perhaps in these cases carbs was working as expected.
Thanks for carbs, despite my confusion with parameter space scaling it's been very useful!