Test and document usage with RandomizedSearchCV
Opened this issue · 7 comments
Do we want to overload the use of set_grid, or use set_rv? Perhaps we can draw parameters from distrib if available, and grid otherwise.
How do we deal with multiple grids when make_randomized_search is called? (raise error?)
Can we generalise this interface?
Much the same can be done for skopt.BayesSearchCV except there multiple grids are allowed, and may have a number of iterations attached to them. Not sure how to specify that with the current searchgrid api.
Much the same can be done for
skopt.BayesSearchCVexcept there multiple grids are allowed, and may have a number of iterations attached to them.
Do you think the weighting of different options is useful/needed? IMHO we could solve this by implementing weights for Categorical dimensions (in skopt). Instead of having a list of options.
It would require us being able to have a "tree" like search space, with some dimensions only existing for certain values of other dimensions. With the current setup with a list you can do this for one "level", but more gets hard/impossible.
Have you looked at hyperopt's way of specifying a search space?
Yes, I realise implementing weights in categorical dimensions is an option... as long as that categorical draw entailed an entire search subspace as you say. So really it's about a generative sampling model. Which, yes, is what Hyperopt allows you to do (as well as having arbitrary relationships and constraints between parameters).
I suppose I'd be able to express what I need to in BayesSearchCV if your list of tuple format allowed the second value in the tuple to be a float proportion rather than an integer count of iterations.
Or perhaps I mean that the tuple notation could be used to express the prior distribution over those alternatives.
Dunno, I'm not a fan of the tuple notation. Our code to interpret it is a constant source of confusion and bugs. So I've started telling people to be explicit for anything but the most trivial cases...
Does allowing the grid to be a categorical-like thing fix those issues?
I'd phrase it as having a dimension like so Categorical([a_space, a_different_space, ...]). Is that what you mean? If yes, then I also think this is probably the way to go. Hard bit is figuring out how to make a regressor that is happy with that so we can have shared knowledge between the two sub-spaces.