imbue-ai/carbs

Entire parameter space won't get explored

Opened this issue · 0 comments

Hello, I've been using carbs with decent success but noticed some parameter spaces were not getting explored very well. Here are my parameters:

params = [
        Param(
            name="total_timesteps",
            space=LinearSpace(min=10_000_000, max=50_000_000, scale=10_000_000, is_integer=True),
            search_center=20_000_000,
        ),
        Param(name="learning_rate", space=LogSpace(min=1e-5, max=1e-1), search_center=3e-05),
        Param(name="gamma", space=LogitSpace(min=0.8, max=0.9999), search_center=0.995),
        Param(name="gae_lambda", space=LogitSpace(min=0.8, max=1.0), search_center=0.98),
        Param(
            name="update_epochs", space=LinearSpace(min=1, max=15, scale=5, is_integer=True), search_center=5
        ),
        Param(name="clip_coef", space=LogitSpace(min=0.1, max=0.4), search_center=0.2),
        Param(name="ent-coef", space=LogSpace(min=1e-5, max=1e-1), search_center=7e-03),
        Param(name="vf_coef", space=LogitSpace(min=0.0, max=1.0), search_center=0.5),
        Param(name="vf_clip_coef", space=LogitSpace(min=0.1, max=0.4), search_center=0.1),
        Param(name="max_grad_norm", space=LinearSpace(min=0, max=5), search_center=1),
        Param(
            name="batch_size",
            space=LinearSpace(min=65_536, max=1_048_576, scale=270_000, is_integer=True),
            search_center=98_304,
        ),
        Param(
            name="minibatch_size",
            space=LinearSpace(min=128, max=65_536, scale=20_000, is_integer=True),
            search_center=512,
        ),
    ]

I left the initial_search_radius at the default of 0.3. I saw in the README and in a warning that I needed to adjust the scale of Linear spaces so I started performing some tests to try and figure out a good scale value. I would initialize carbs with 1 parameter space, set num_random_samples to 1000, gather 1000 suggestions and take note of the min, max, mean, and standard deviation. I did this with the initial_search_radius at 1.0 because as I understand it, carbs will increase the search radius as more observations suggestions are made. My thought was to find a scale value that will allow carbs to search the entire Linear space.

This worked OK in practice, for example, the minimum suggestions for the update_epochs parameter was 2 and the max was 8 after 24 suggestions (12 of which were random). I was surprised to see several Log and Logit parameter spaces not explored very much though. gamma for example had a min suggested value of 0.9821 and a max of 0.9987. I'm assuming this has to do with the search center being set so close to the max.

All this to say, I don't entirely understand how scale is supposed to be set and how it interacts with initial_search_radius. How should I set scale so that the min and max values of a space can be reached, in all types of parameter spaces? 'Set it >3' wasn't enough information for me to find good scale values for Linear spaces, and I have no idea how to set the scale for Log and Logit spaces, or if I even need to, perhaps in these cases carbs was working as expected.

Thanks for carbs, despite my confusion with parameter space scaling it's been very useful!