Exception originated from within Sacred. KeyError: 'Invalid key "DogmaticDict.__setitem__". Config-keys cannot contain "." or start with "$"'

[hyfwyy]

Hey,

I have some issues with the originated from within Sacred when i debug my code.

ex = sacred.Experiment('Global (train)', ingredients=[data_ingredient, model_ingredient],interactive=True)
SETTINGS.CAPTURE_MODE = 'sys'
ex.captured_out_filter = apply_backspaces_and_linefeeds
@ex.automain
def main(epochs, cpu, cudnn_flag, temp_dir, seed, no_bias_decay):
       '...'

When i debug until the line of @ex.automain, the error was raised as follows.

Exception originated from within Sacred.
Traceback (most recent calls):
File "/root/anaconda3/lib/python3.10/site-packages/sacred/config/utils.py", line 42, in assert_is_valid_key
raise KeyError(
KeyError: 'Invalid key "DogmaticDict.setitem". Config-keys cannot contain "." or start with "$"'

I find the error happen in line 42, in assert_is_valid_key

def assert_is_valid_key(key):
        if SETTINGS.CONFIG.ENFORCE_KEYS_MONGO_COMPATIBLE and (
                isinstance(key, str) and ("." in key or key[0] == "$")
            ):
                raise KeyError(
                    'Invalid key "{}". Config-keys cannot '
                    'contain "." or start with "$"'.format(key)
                )

And the key is defined as:

def normalize_or_die(obj):
    if isinstance(obj, dict):
        res = dict()
        for key, value in obj.items():
            assert_is_valid_key(key)
            res[key] = normalize_or_die(value)
        return res
    elif isinstance(obj, (list, tuple)):
        return list([normalize_or_die(value) for value in obj])
    return normalize_numpy(obj)

But i cannot find the definition of obj. And I cannot find any solution online.

I would be grateful for some help!

Best,

[thequilo]

Hey! What does your config look like?

[Wznnnnn]

@thequilo
Hey！the same problem
My config is as follow:

def _loss_names(d):
    ret = {
        "itm": 0,
        "mlm": 0,
        "mpp": 0,
        "vqa": 0,
        "nlvr2": 0,
        "irtr": 0,
    }
    ret.update(d)
    return ret
    
@ex.config
def config():
    exp_name = "vilt"
    seed = 0
    datasets = ["coco", "vg", "sbu", "gcc"]
    loss_names = _loss_names({"itm": 1, "mlm": 1})
    batch_size = 4096  # this is a desired batch size; pl trainer will accumulate gradients when per step batch is smaller.

    # Image setting
    train_transform_keys = ["pixelbert"]
    val_transform_keys = ["pixelbert"]
    image_size = 384
    max_image_len = -1
    patch_size = 32
    draw_false_image = 1
    image_only = False

    # Text Setting
    vqav2_label_size = 3129
    max_text_len = 40
    tokenizer = "bert-base-uncased"
    vocab_size = 30522
    whole_word_masking = False
    mlm_prob = 0.15
    draw_false_text = 0

    # Transformer Setting
    vit = "vit_base_patch32_384"
    hidden_size = 768
    num_heads = 12
    num_layers = 12
    mlp_ratio = 4
    drop_rate = 0.1

    # Optimizer Setting
    optim_type = "adamw"
    learning_rate = 1e-4
    weight_decay = 0.01
    decay_power = 1
    max_epoch = 100
    max_steps = 25000
    warmup_steps = 2500
    end_lr = 0
    lr_mult = 1  # multiply lr for downstream heads

    # Downstream Setting
    get_recall_metric = False

    # PL Trainer Setting
    resume_from = None
    fast_dev_run = False
    val_check_interval = 1.0
    test_only = False

    # below params varies with the environment
    data_root = ""
    log_dir = "result"
    per_gpu_batchsize = 0  # you should define this manually with per_gpu_batch_size=#
    num_gpus = 1
    num_nodes = 1
    load_path = ""
    num_workers = 8
    precision = 16
    
    
@ex.named_config
def task_finetune_vqa_randaug():
    exp_name = "finetune_vqa_randaug"
    datasets = ["vqa"]
    train_transform_keys = ["pixelbert_randaug"]
    loss_names = _loss_names({"vqa": 1})
    batch_size = 256
    max_epoch = 10
    max_steps = None
    warmup_steps = 0.1
    draw_false_image = 0
    learning_rate = 1e-4
    val_check_interval = 0.1
    lr_mult = 10

[thequilo]

Hey @Wznnnnn! I just copied your config and added a simple main function. It works for me, I don't see the error. Can you provide a smaller example that reproduces the error?

It could also be related to the way you call the script. Another reason could be a debugger, which sometimes injects symbols into the local namespace. Weird things can happen if you add a breakpoint in a config scope.