BOHB loop does not abort when there is an error
poojitharamachandra opened this issue · 1 comments
Hi,
Returning just the loss from 'compute' function of the worker causes error, But the execution does not abort, instead goes into an infinite loop.
I wrote 'return loss' in my code , instead of
return ({'loss': s, # remember: HpBandSter always minimizes!
'info': {'r2_score': r2,'RMSE': accuracy}
})#something like this
the debug file had error :
File "/home/....../lib/python3.8/site-packages/hpbandster/core/base_iteration.py", line 133, in register_result
if (not job.result is None) and np.isfinite(result['loss']):
TypeError: 'float' object is not subscriptable
I saw this error after 12 hours.
When this was corrected, the complete setup ran for just 5 min and gave me results (of course I was using a very small dataset and model)
Thanks
I find this behavior not optimal, too. Any error in the compute
method from the worker is caught at some place and causes the optimization to continue running. I'm wondering if that behavior is desired and implemented on purpose.
In my opinion a better approach would be not to catch exceptions in the compute
method.