Copying of ConfigSpace objects does not yield identical objects
Closed this issue · 7 comments
alexandertornede commented
Since we are using this package in https://github.com/automl/DeepCAVE, we noticed that the copy function (
py-pdp-partitioner/pyPDP/utils/utils.py
Line 224 in 0feff79
ConfigSpace objects in the utils has a problem. In particular, the indices of the hyperparameters are wrong in the copy. One can check that by inspecting, for example, the _hyperparameter_idx dict of a the original and the copied ConfigSpace object.
Unless I am missing something, a potential fix is to simply use the deepcopy functionality of the copy module. This is also demonstrated in the sample code to reproduce the problem.
Sample code to reproduce the problem, including a potential fix:
from pyPDP.utils.utils import copy_config_space
import copy
if __name__ == "__main__":
json_string = '{ "hyperparameters": [ { "name": "backbone:activation", "type": "categorical", "choices": [ "relu", "swish", "swish-1", "tanh" ], "default": "relu", "weights": null }, { "name": "backbone:batch_normalization", "type": "categorical", "choices": [ true, false ], "default": false, "weights": null }, { "name": "backbone:dropout", "type": "categorical", "choices": [ true, false ], "default": true, "weights": null }, { "name": "backbone:num_latent_units", "type": "uniform_int", "log": false, "lower": 1, "upper": 8, "default": 4 }, { "name": "backbone:num_layers", "type": "uniform_int", "log": false, "lower": 0, "upper": 3, "default": 2 }, { "name": "backbone:skip_connection", "type": "categorical", "choices": [ true, false ], "default": false, "weights": null }, { "name": "data_loader:batch_size", "type": "uniform_int", "log": false, "lower": 32, "upper": 512, "default": 128 }, { "name": "model:__choice__", "type": "categorical", "choices": [ "dasvdd", "dagmm", "ae", "vae" ], "default": "dasvdd", "weights": null }, { "name": "optimizer:__choice__", "type": "categorical", "choices": [ "AdamWOptimizer", "SGDOptimizer" ], "default": "AdamWOptimizer", "weights": null }, { "name": "scaler:__choice__", "type": "categorical", "choices": [ "MinMaxScaler", "NoScaler", "StandardScaler", "RobustScaler", "L101Scaler" ], "default": "MinMaxScaler", "weights": null }, { "name": "backbone:dropout_rate", "type": "uniform_float", "log": false, "lower": 0.0, "upper": 0.8, "default": 0.5 }, { "name": "backbone:num_units_layer_1", "type": "uniform_int", "log": false, "lower": 8, "upper": 16, "default": 12 }, { "name": "backbone:num_units_layer_2", "type": "uniform_int", "log": false, "lower": 4, "upper": 12, "default": 8 }, { "name": "backbone:num_units_layer_3", "type": "uniform_int", "log": false, "lower": 4, "upper": 12, "default": 8 }, { "name": "model:dagmm:activation", "type": "categorical", "choices": [ "relu", "swish", "swish-1", "tanh" ], "default": "tanh", "weights": null }, { "name": "model:dagmm:gmm_dimensionality", "type": "uniform_int", "log": false, "lower": 1, "upper": 5, "default": 2 }, { "name": "model:dagmm:lambda_1", "type": "uniform_float", "log": false, "lower": 0.01, "upper": 0.5, "default": 0.1 }, { "name": "model:dagmm:lambda_2", "type": "uniform_float", "log": false, "lower": 0.0001, "upper": 0.05, "default": 0.005 }, { "name": "model:dagmm:num_units", "type": "uniform_int", "log": false, "lower": 2, "upper": 20, "default": 10 }, { "name": "model:dasvdd:K", "type": "uniform_float", "log": false, "lower": 0.1, "upper": 1.0, "default": 0.9 }, { "name": "model:dasvdd:T", "type": "uniform_int", "log": false, "lower": 3, "upper": 100, "default": 10 }, { "name": "model:dasvdd:center_optimizer:__choice__", "type": "categorical", "choices": [ "AdamWOptimizer", "SGDOptimizer", "AdagradOptimizer" ], "default": "AdagradOptimizer", "weights": null }, { "name": "optimizer:AdamWOptimizer:beta1", "type": "uniform_float", "log": false, "lower": 0.85, "upper": 0.999, "default": 0.9 }, { "name": "optimizer:AdamWOptimizer:beta2", "type": "uniform_float", "log": false, "lower": 0.9, "upper": 0.9999, "default": 0.999 }, { "name": "optimizer:AdamWOptimizer:lr", "type": "uniform_float", "log": true, "lower": 1e-05, "upper": 0.1, "default": 0.001 }, { "name": "optimizer:AdamWOptimizer:weight_decay", "type": "uniform_float", "log": false, "lower": 0.0, "upper": 0.1, "default": 0.01 }, { "name": "optimizer:SGDOptimizer:lr", "type": "uniform_float", "log": true, "lower": 1e-05, "upper": 0.1, "default": 0.001 }, { "name": "optimizer:SGDOptimizer:momentum", "type": "uniform_float", "log": false, "lower": 0.0, "upper": 0.99, "default": 0.0 }, { "name": "optimizer:SGDOptimizer:weight_decay", "type": "uniform_float", "log": false, "lower": 0.0, "upper": 0.1, "default": 0.0 }, { "name": "model:dasvdd:center_optimizer:AdagradOptimizer:lr", "type": "uniform_float", "log": false, "lower": 1e-05, "upper": 2.0, "default": 1.0 }, { "name": "model:dasvdd:center_optimizer:AdagradOptimizer:lr_decay", "type": "uniform_float", "log": false, "lower": 0.0, "upper": 0.99, "default": 0.01 }, { "name": "model:dasvdd:center_optimizer:AdagradOptimizer:weight_decay", "type": "uniform_float", "log": false, "lower": 0.0, "upper": 0.1, "default": 0.0 }, { "name": "model:dasvdd:center_optimizer:AdamWOptimizer:beta1", "type": "uniform_float", "log": false, "lower": 0.85, "upper": 0.999, "default": 0.9 }, { "name": "model:dasvdd:center_optimizer:AdamWOptimizer:beta2", "type": "uniform_float", "log": false, "lower": 0.9, "upper": 0.9999, "default": 0.999 }, { "name": "model:dasvdd:center_optimizer:AdamWOptimizer:lr", "type": "uniform_float", "log": true, "lower": 1e-05, "upper": 0.1, "default": 0.001 }, { "name": "model:dasvdd:center_optimizer:AdamWOptimizer:weight_decay", "type": "uniform_float", "log": false, "lower": 0.0, "upper": 0.1, "default": 0.01 }, { "name": "model:dasvdd:center_optimizer:SGDOptimizer:lr", "type": "uniform_float", "log": true, "lower": 1e-05, "upper": 0.1, "default": 0.001 }, { "name": "model:dasvdd:center_optimizer:SGDOptimizer:momentum", "type": "uniform_float", "log": false, "lower": 0.0, "upper": 0.99, "default": 0.0 }, { "name": "model:dasvdd:center_optimizer:SGDOptimizer:weight_decay", "type": "uniform_float", "log": false, "lower": 0.0, "upper": 0.1, "default": 0.0 } ], "conditions": [ { "child": "backbone:dropout_rate", "parent": "backbone:dropout", "type": "EQ", "value": true }, { "child": "backbone:num_units_layer_1", "parent": "backbone:num_layers", "type": "GT", "value": 0 }, { "child": "backbone:num_units_layer_2", "parent": "backbone:num_layers", "type": "GT", "value": 1 }, { "child": "backbone:num_units_layer_3", "parent": "backbone:num_layers", "type": "GT", "value": 2 }, { "child": "model:dagmm:activation", "parent": "model:__choice__", "type": "EQ", "value": "dagmm" }, { "child": "model:dagmm:gmm_dimensionality", "parent": "model:__choice__", "type": "EQ", "value": "dagmm" }, { "child": "model:dagmm:lambda_1", "parent": "model:__choice__", "type": "EQ", "value": "dagmm" }, { "child": "model:dagmm:lambda_2", "parent": "model:__choice__", "type": "EQ", "value": "dagmm" }, { "child": "model:dagmm:num_units", "parent": "model:__choice__", "type": "EQ", "value": "dagmm" }, { "child": "model:dasvdd:K", "parent": "model:__choice__", "type": "EQ", "value": "dasvdd" }, { "child": "model:dasvdd:T", "parent": "model:__choice__", "type": "EQ", "value": "dasvdd" }, { "child": "model:dasvdd:center_optimizer:__choice__", "parent": "model:__choice__", "type": "EQ", "value": "dasvdd" }, { "child": "optimizer:AdamWOptimizer:beta1", "parent": "optimizer:__choice__", "type": "EQ", "value": "AdamWOptimizer" }, { "child": "optimizer:AdamWOptimizer:beta2", "parent": "optimizer:__choice__", "type": "EQ", "value": "AdamWOptimizer" }, { "child": "optimizer:AdamWOptimizer:lr", "parent": "optimizer:__choice__", "type": "EQ", "value": "AdamWOptimizer" }, { "child": "optimizer:AdamWOptimizer:weight_decay", "parent": "optimizer:__choice__", "type": "EQ", "value": "AdamWOptimizer" }, { "child": "optimizer:SGDOptimizer:lr", "parent": "optimizer:__choice__", "type": "EQ", "value": "SGDOptimizer" }, { "child": "optimizer:SGDOptimizer:momentum", "parent": "optimizer:__choice__", "type": "EQ", "value": "SGDOptimizer" }, { "child": "optimizer:SGDOptimizer:weight_decay", "parent": "optimizer:__choice__", "type": "EQ", "value": "SGDOptimizer" }, { "child": "model:dasvdd:center_optimizer:AdagradOptimizer:lr", "parent": "model:dasvdd:center_optimizer:__choice__", "type": "EQ", "value": "AdagradOptimizer" }, { "child": "model:dasvdd:center_optimizer:AdagradOptimizer:lr_decay", "parent": "model:dasvdd:center_optimizer:__choice__", "type": "EQ", "value": "AdagradOptimizer" }, { "child": "model:dasvdd:center_optimizer:AdagradOptimizer:weight_decay", "parent": "model:dasvdd:center_optimizer:__choice__", "type": "EQ", "value": "AdagradOptimizer" }, { "child": "model:dasvdd:center_optimizer:AdamWOptimizer:beta1", "parent": "model:dasvdd:center_optimizer:__choice__", "type": "EQ", "value": "AdamWOptimizer" }, { "child": "model:dasvdd:center_optimizer:AdamWOptimizer:beta2", "parent": "model:dasvdd:center_optimizer:__choice__", "type": "EQ", "value": "AdamWOptimizer" }, { "child": "model:dasvdd:center_optimizer:AdamWOptimizer:lr", "parent": "model:dasvdd:center_optimizer:__choice__", "type": "EQ", "value": "AdamWOptimizer" }, { "child": "model:dasvdd:center_optimizer:AdamWOptimizer:weight_decay", "parent": "model:dasvdd:center_optimizer:__choice__", "type": "EQ", "value": "AdamWOptimizer" }, { "child": "model:dasvdd:center_optimizer:SGDOptimizer:lr", "parent": "model:dasvdd:center_optimizer:__choice__", "type": "EQ", "value": "SGDOptimizer" }, { "child": "model:dasvdd:center_optimizer:SGDOptimizer:momentum", "parent": "model:dasvdd:center_optimizer:__choice__", "type": "EQ", "value": "SGDOptimizer" }, { "child": "model:dasvdd:center_optimizer:SGDOptimizer:weight_decay", "parent": "model:dasvdd:center_optimizer:__choice__", "type": "EQ", "value": "SGDOptimizer" } ], "forbiddens": [], "python_module_version": "0.6.1", "json_format_version": 0.4}'
cs = cs_json.read(json_string)
cs_copy_wrong = copy_config_space(cs, seed=1)
cs_copy_correct = copy.deepcopy(cs)
dwoiwode commented
Thank you for mentioning this! I will review this in the following weeks. The seeding might be induce a problem there when using deepcopy, but this might be resolved using cs.seed(...) instead of providing the seed in the constructor
sarah-segel commented
Thank you for fixing this! Unfortunately, the increase of the ConfigSpace version requirement to 0.7.1 is problematic for us, as there is a problem with the backward-compatibility of json files from earlier versions. Would it thus be possible to change the version requirement back to to 0.6.1?
dwoiwode commented
According to the Changelog of ConfigSpace, nothing changed in this version. It also looks like v0.7.2 already fixes this problem with jsons? Sadly this version is not available on pypi yet, so you might want to ask whether this version gets bumped and uploaded? I also had this problem, which was the reason for the second commit 41d8c09.
Does this work for you?
sarah-segel commented
Unfortunately, v0.7.1 does not work for us (as loading runs created with e.g. v0.6.1 is not possible) and as you said, v0.7.2 is not available on pypi yet. I will ask the maintainers for the time schedule of bumping v0.7.2 though, and will get back to you then :)
dwoiwode commented
I think it might have been forgotten to update the version on pypi, as it is available from GitHub since June last year. In the mean time I guess it should work to update the version using
pip install git+https://github.com/automl/ConfigSpace.git
sarah-segel commented
So there will be a new ConfigSpace version v0.8.0 coming in the next weeks. Until then, I am afraid we cannot use the pyPDPPartitioner v0.1.9 version though (ConfigSpace v0.7.2 will not be released and there are also some problems for us when running with that version).
sarah-segel commented
FYI: With the release of the new backward-compatible ConfigSpace version, this is solved for us now :)