Hyperactive returns integer parameters as float in best_para when search space has both int and float parameters
0liu opened this issue · 4 comments
Hi Simon,
This is not strictly a bug but more like a suggestion or concern. As I reported in SimonBlanke/Gradient-Free-Optimizers#15, integer parameters in search space are returned as float, which caused TypeError if passing the best para to model for evaluation. It turned out the issue was not in the optimizers, but in Hyperactive converter functions, specifically:
Hyperactive/hyperactive/optimizers/hyper_gradient_trafo.py
Lines 30 to 36 in cb1df3b
This function converts para value list to np.array, which converts all integers to float.
This only happens when search space contains both integer and float parameters.
I browsed the code around and don't see it's very necessary to use numpy array as returns. So it may be better to use plain Python list or nested list to pass parameter values between converters. But you may have other considerations to use numpy array in those functions. I will create a pull request to better clarify my idea and provide some edits, just as my suggestion.
Hello @0liu,
I am grateful for your continuing interest in Hyperactive, but we should focus on identifying the error first. I took the code from issue 15 of GFO and converted it to Hyperactive to validate your concerns about this problem:
from hyperactive import Hyperactive
search_space = dict(my_para=range(100, 200, 10))
def obj_func(para):
score = 1
my_para = para["my_para"]
for i in range(my_para):
pass
return score
hyper = Hyperactive()
hyper.add_search(obj_func, search_space, n_iter=100)
hyper.run()This small test script runs fine. Could you provide a small but complete example to replicate the problem you are encountering?
Hi Simon,
I missed the key information in this issue's description but mentioned it in my PR: This happens when search space contains both integer and float type parameters. In this way, numpy will convert integer parameter value to float. I updated the title and first post.
By your example, if we have two parameters for the obj_func:
from hyperactive import Hyperactive
search_space = dict(my_para=range(100, 200, 10), my_para_2=[0.1, 0.2, 0.3])
def obj_func(para):
score = 1
my_para = para["my_para"]
for i in range(my_para):
pass
return score
hyper = Hyperactive()
hyper.add_search(obj_func, search_space, n_iter=100)
hyper.run()The results are:
Results: 'obj_func'
Best score: 1
Best parameter set:
'my_para' : 130.0
'my_para_2' : 0.3
Best iteration: 0
We can see my_para in the returned best para is 130.0 instead of 130. This is fine for human reading or manual post-processing, but in an automated process, if passing the returned results to evaluate model on validation / test data sets, there could be a problem. That's why I thought it's more like a API design concern instead of a bug.
Hello @0liu,
now I see! Thank you for the explanation. This should be classified as a bug. But I think this bug originates in GFO. I reopened your original issue in GFO, because the error occurs only when the search space contains int and float. We missed this in the original issue.
GFO also has the int to float converted parameter, if you extract the best parameter with opt.best_para. Therefore I think those problems (one within the objective function and one with .best_para) are related and both originate in GFO.
When we fix the problem in GFO and the problem persists in Hyperactive, then we continue our efforts here. Until then we should focus on fixing the problem in GFO.