automl/SMAC3

Forbidden relations of conditioned hyperparameters

Closed this issue · 3 comments

bbudescu commented

Description

Using a relational constraint on the values that can be taken by two hyperparameters, one of them not being guaranteed to be sampled every time, causes a crash either when adding HPs to the config space, or when instantiating the Facade and sampling the initial configs.

Steps/Code to Reproduce

from ConfigSpace import (ConfigurationSpace, Configuration, Categorical, EqualsCondition, ForbiddenEqualsRelation,
                         ForbiddenAndConjunction, ForbiddenEqualsClause)
from smac import Scenario, HyperparameterOptimizationFacade, MultiFidelityFacade

a = Categorical('a', [2, 5, 10], ordered=True)
enable_a = Categorical('enable_a', [False, True])
cond_a = EqualsCondition(a, enable_a, True)

b = Categorical('b', [5, 10, 15], ordered=True)
forbid_a_b = ForbiddenEqualsRelation(a, b)
# forbid_a = ForbiddenEqualsClause(enable_a, True)
# forbid_a_b = ForbiddenAndConjunction(forbid_a, forbid_a_b)

cs = ConfigurationSpace()
cs.add([a, enable_a, cond_a, b, forbid_a_b])

scenario = Scenario(cs, deterministic=True, n_trials=200)

def train(config: Configuration, seed: int = 0, budget: int = 0) -> float:
    pass

smac = MultiFidelityFacade(scenario, train)

I've found that enabling the extra forbidden clause forbid_a in the commented line above, and replacing the original forbid_a_b by a ForbiddenAndConjunction helps when using HyperparameterOptimizationFacade but not with MultiFidelityFacade

Expected Results

No crash.

Actual Results

File "playground/playground_smac_bug.py", line 32, in <module>
    cs.add([a, enable_a, cond_a, b, forbid_a_b])
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/configuration_space.py", line 351, in add
    self._check_default_configuration()
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/configuration_space.py", line 915, in _check_default_configuration
    return Configuration(self, values=values)
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/configuration.py", line 126, in __init__
    self.check_valid_configuration()
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/configuration.py", line 160, in check_valid_configuration
    check_configuration(
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/util.py", line 611, in check_configuration
    if forbidden.is_forbidden_vector(vector):
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/forbidden.py", line 683, in is_forbidden_vector
    return self.left.to_value(left) == self.right.to_value(right)  # type: ignore
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/hyperparameters/hyperparameter.py", line 358, in to_value
    value: DType = self._transformer.to_value(np.array([vector]))[0]
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/hyperparameters/hp_components.py", line 174, in to_value
    raise ValueError(
ValueError: Got unexpected float value while trying to transform a vector representation into a value in [ 2  5 10].Expected integers but got [nan] (dtype: float64)

Versions

2.2.0

LE: setting default=True on init of enable_a helps avoid the issue happening at ConfigurationSpace.add() when checking the default configuration, and postpones it until a bunch of configs are sampled as vectors for the initial design (during the Facade initialization), where nans cause issues.

bbudescu commented

I forgot to mention that when stepping through the code, the nan values are used as replacements for variables that shouldn't be always sampled, e.g., a is assigned nan when enable_a is sampled as False.

bbudescu commented

Is this the right place to post this issue? Since this crash happens in ConfigurationSpace.add() then, perhaps, it's better to post this issue in the ConfigSpace repo. I was in doubt because after uncommenting the two above disabled lines (i.e., we add an extra condition for a to be enabled) makes the crash occur within the MultiFidelityFacade construction instead.

Here's the output after uncommenting the two lines:

File "playground_smac_bug.py", line 40, in <module>
    sm = MultiFidelityFacade(scenario, train)
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/smac/facade/abstract_facade.py", line 213, in __init__
    self._update_dependencies()
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/smac/facade/abstract_facade.py", line 443, in _update_dependencies
    self._config_selector._set_components(
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/smac/main/config_selector.py", line 104, in _set_components
    self._initial_design_configs = initial_design.select_configurations()
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/smac/initial_design/abstract_initial_design.py", line 135, in select_configurations
    configs += self._select_configurations()
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/smac/initial_design/random_design.py", line 15, in _select_configurations
    configs = self._configspace.sample_configuration(size=self._n_configs)
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/configuration_space.py", line 624, in sample_configuration
    cond_forbidden |= clause.is_forbidden_vector_array(valid_configs)
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/forbidden.py", line 547, in is_forbidden_vector_array
    forbidden_mask &= forbidden.is_forbidden_vector_array(arr)
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/forbidden.py", line 689, in is_forbidden_vector_array
    return self.left.to_value(left) == self.right.to_value(right)  # type: ignore
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/hyperparameters/hyperparameter.py", line 356, in to_value
    return self._transformer.to_value(vector)
  File "/home/bogdan/anaconda3/envs/smac/lib/python3.10/site-packages/ConfigSpace/hyperparameters/hp_components.py", line 174, in to_value
    raise ValueError(
ValueError: Got unexpected float value while trying to transform a vector representation into a value in [ 2  5 10].Expected integers but got [nan  1.  2. nan nan  0.  1.  1.  2. nan  1. nan  1.] (dtype: float64)

LE: added an issue in the right place (automl/ConfigSpace#396). Should I move all contents there?