ratt-ru/QuartiCal

Key 'directory' not in 'Outputs'

Closed this issue · 5 comments

viralp commented

Must be a naive query, missing something minor. I am getting the following error message. What does it mean key "directory"?

2023-05-23 15:36:57.361 | ERROR    | __main__:<module>:8 - An error has been caught in function '<module>', process 'MainProcess' (49435), thread 'MainThread' (140448923129664):
Traceback (most recent call last):

> File "/users/vparekh/venv3/bin/goquartical", line 8, in <module>
    sys.exit(execute())
    │   │    └ <function execute at 0x7fbc83091f70>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/quartical/executor.py", line 29, in execute
    _execute(stack)
    │        └ <contextlib.ExitStack object at 0x7fbcd01f71f0>
    └ <function _execute at 0x7fbc83093040>
  File "/users/vparekh/venv3/lib/py
[quartical_GK_selfcal.txt](https://github.com/ratt-ru/QuartiCal/files/11548636/quartical_GK_selfcal.txt)
thon3.8/site-packages/quartical/executor.py", line 38, in _execute
    opts, config_files = parser.parse_inputs()
                         │      └ <function parse_inputs at 0x7fbc8b7321f0>
                         └ <module 'quartical.config.parser' from '/users/vparekh/venv3/lib/python3.8/site-packages/quartical/config/parser.py'>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/quartical/config/parser.py", line 124, in parse_inputs
    config = oc.merge(config, *additional_config)
             │  │     │        └ [{'input_ms': {'path': '../msdir/A168_VLA_L_D_targets_allspw.ms', 'data_column': 'CORRECTED_DATA', 'sigma_column': None, 'wei...
             │  │     └ {'input_ms': {'path': '???', 'data_column': 'DATA', 'sigma_column': None, 'weight_column': None, 'time_chunk': '0', 'freq_chu...
             │  └ <staticmethod object at 0x7fbca9f1e880>
             └ <class 'omegaconf.omegaconf.OmegaConf'>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 273, in merge
    target.merge_with(*configs[1:])
    │      │           └ ({'input_ms': {'path': '???', 'data_column': 'DATA', 'sigma_column': None, 'weight_column': None, 'time_chunk': '0', 'freq_ch...
    │      └ <function BaseContainer.merge_with at 0x7fbca9f701f0>
    └ {'input_ms': {'path': '../msdir/A168_VLA_L_D_targets_allspw.ms', 'data_column': 'CORRECTED_DATA', 'sigma_column': None, 'weig...
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 492, in merge_with
    self._format_and_raise(key=None, value=None, cause=e)
    │    └ <function Node._format_and_raise at 0x7fbca9f5d4c0>
    └ {'input_ms': {'path': '../msdir/A168_VLA_L_D_targets_allspw.ms', 'data_column': 'CORRECTED_DATA', 'sigma_column': None, 'weig...
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
    └ <function format_and_raise at 0x7fbca9f501f0>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/_utils.py", line 819, in format_and_raise
    _raise(ex, cause)
    │      │   └ ConfigKeyError("Key 'directory' not in 'Outputs'\n    full_key: output.directory\n    reference_type=Outputs\n    object_type...
    │      └ ConfigKeyError("Key 'directory' not in 'Outputs'\n    full_key: output.directory\n    reference_type=Outputs\n    object_type...
    └ <function _raise at 0x7fbca9f50160>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
          │  │              │   └ <built-in function exc_info>
          │  │              └ <module 'sys' (built-in)>
          │  └ <method 'with_traceback' of 'BaseException' objects>
          └ ConfigKeyError("Key 'directory' not in 'Outputs'\n    full_key: output.directory\n    reference_type=Outputs\n    object_type...
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 490, in merge_with
    self._merge_with(*others)
    │    │            └ ({'input_ms': {'path': '../msdir/A168_VLA_L_D_targets_allspw.ms', 'data_column': 'CORRECTED_DATA', 'sigma_column': None, 'wei...
    │    └ <function BaseContainer._merge_with at 0x7fbca9f70280>
    └ {'input_ms': {'path': '../msdir/A168_VLA_L_D_targets_allspw.ms', 'data_column': 'CORRECTED_DATA', 'sigma_column': None, 'weig...
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 514, in _merge_with
    BaseContainer._map_merge(self, other)
    │             │          │     └ {'input_ms': {'path': '../msdir/A168_VLA_L_D_targets_allspw.ms', 'data_column': 'CORRECTED_DATA', 'sigma_column': None, 'weig...
    │             │          └ {'input_ms': {'path': '../msdir/A168_VLA_L_D_targets_allspw.ms', 'data_column': 'CORRECTED_DATA', 'sigma_column': None, 'weig...
    │             └ <staticmethod object at 0x7fbca9f60d00>
    └ <class 'omegaconf.basecontainer.BaseContainer'>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 399, in _map_merge
    dest_node._merge_with(src_node)
    │         │           └ {'directory': 'outputs_3.qc', 'overwrite': False, 'products': ['corrected_data', 'corrected_weight'], 'columns': ['CORRECTED_...
    │         └ <function BaseContainer._merge_with at 0x7fbca9f70280>
    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'log_to_terminal': True, 'overwrite': False, 'products': None, 'co...
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 514, in _merge_with
    BaseContainer._map_merge(self, other)
    │             │          │     └ {'directory': 'outputs_3.qc', 'overwrite': False, 'products': ['corrected_data', 'corrected_weight'], 'columns': ['CORRECTED_...
    │             │          └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'log_to_terminal': True, 'overwrite': False, 'products': None, 'co...
    │             └ <staticmethod object at 0x7fbca9f60d00>
    └ <class 'omegaconf.basecontainer.BaseContainer'>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 432, in _map_merge
    dest[key] = src._get_node(key)
    │    │      │   │         └ 'directory'
    │    │      │   └ <function DictConfig._get_node at 0x7fbca9f06670>
    │    │      └ {'directory': 'outputs_3.qc', 'overwrite': False, 'products': ['corrected_data', 'corrected_weight'], 'columns': ['CORRECTED_...
    │    └ 'directory'
    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'log_to_terminal': True, 'overwrite': False, 'products': None, 'co...
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 310, in __setitem__
    self._format_and_raise(
    │    └ <function Node._format_and_raise at 0x7fbca9f5d4c0>
    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'log_to_terminal': True, 'overwrite': False, 'products': None, 'co...
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
    └ <function format_and_raise at 0x7fbca9f501f0>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/_utils.py", line 819, in format_and_raise
    _raise(ex, cause)
    │      │   └ ConfigAttributeError("Key 'directory' not in 'Outputs'\n    full_key: output.directory\n    reference_type=Outputs\n    objec...
    │      └ ConfigKeyError("Key 'directory' not in 'Outputs'\n    full_key: output.directory\n    reference_type=Outputs\n    object_type...
    └ <function _raise at 0x7fbca9f50160>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
          │  │              │   └ <built-in function exc_info>
          │  │              └ <module 'sys' (built-in)>
          │  └ <method 'with_traceback' of 'BaseException' objects>
          └ ConfigKeyError("Key 'directory' not in 'Outputs'\n    full_key: output.directory\n    reference_type=Outputs\n    object_type...
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 308, in __setitem__
    self.__set_impl(key=key, value=value)
    │                   │          └ 'outputs_3.qc'
    │                   └ 'directory'
    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'log_to_terminal': True, 'overwrite': False, 'products': None, 'co...
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 318, in __set_impl
    self._set_item_impl(key, value)
    │    │              │    └ 'outputs_3.qc'
    │    │              └ 'directory'
    │    └ <function BaseContainer._set_item_impl at 0x7fbca9f70310>
    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'log_to_terminal': True, 'overwrite': False, 'products': None, 'co...
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 545, in _set_item_impl
    self._validate_set(key, value)
    │    │             │    └ 'outputs_3.qc'
    │    │             └ 'directory'
    │    └ <function DictConfig._validate_set at 0x7fbca9f70af0>
    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'log_to_terminal': True, 'overwrite': False, 'products': None, 'co...
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 180, in _validate_set
    target = self._get_node(key) if key is not None else self
             │    │         │       │                    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'log_to_terminal': True, 'overwrite': False, 'products': None, 'co...
             │    │         │       └ 'directory'
             │    │         └ 'directory'
             │    └ <function DictConfig._get_node at 0x7fbca9f06670>
             └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'log_to_terminal': True, 'overwrite': False, 'products': None, 'co...
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
    self._validate_get(key)
    │    │             └ 'directory'
    │    └ <function DictConfig._validate_get at 0x7fbca9f70ee0>
    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'log_to_terminal': True, 'overwrite': False, 'products': None, 'co...
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
    self._format_and_raise(
    │    └ <function Node._format_and_raise at 0x7fbca9f5d4c0>
    └ {'gain_directory': 'gains.qc', 'log_directory': 'logs.qc', 'log_to_terminal': True, 'overwrite': False, 'products': None, 'co...
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
    └ <function format_and_raise at 0x7fbca9f501f0>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
    │      │   └ ConfigAttributeError("Key 'directory' not in 'Outputs'")
    │      └ ConfigAttributeError("Key 'directory' not in 'Outputs'\n    full_key: output.directory\n    reference_type=Outputs\n    objec...
    └ <function _raise at 0x7fbca9f50160>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
          │  │              │   └ <built-in function exc_info>
          │  │              └ <module 'sys' (built-in)>
          │  └ <method 'with_traceback' of 'BaseException' objects>
          └ ConfigAttributeError("Key 'directory' not in 'Outputs'\n    full_key: output.directory\n    reference_type=Outputs\n    objec...

omegaconf.errors.ConfigKeyError: Key 'directory' not in 'Outputs'
    full_key: output.directory
    reference_type=Outputs
    object_type=Outputs
JSKenyon commented

Hi @viralp! This type of error usually occurs when an input is incorrectly specified. In this case, it looks like you are using an older command/config file, and output.directory has been deprecated. Instead, use output.gain_directory and output.log_directory to control outputs.

I will, however, look at capturing/formatting these errors to make them easier to understand.

JSKenyon commented

Fixed in #266.

viralp commented

Thanks Jon. It would be better if you provide the latest config file. Well, I made changes in the script but now getting the following error of reading table in ms file. I am testing quartical for JVLA data.

2023-05-24 11:32:57 | ERROR | goquartical:<module> | An error has been caught in function '<module>', process 'MainProcess' (194819), thread 'MainThread' (140115591432000):
Traceback (most recent call last):

> File "/users/vparekh/venv3/bin/goquartical", line 8, in <module>
    sys.exit(execute())
    │   │    └ <function execute at 0x7f6ee6eb0f70>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/quartical/executor.py", line 29, in execute
    _execute(stack)
    │        └ <contextlib.ExitStack object at 0x7f6f340161f0>
    └ <function _execute at 0x7f6ee6eb3040>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/quartical/executor.py", line 101, in _execute
    data_xds_list, ref_xds_list = read_xds_list(model_columns, ms_opts)
                                  │             │              └ MSInputs(path='A168_VLA_L_D_targets_allspw.ms', data_column='CORRECTED_DATA', sigma_column=None, weight_column=None, time_chu...
                                  │             └ {'MODEL_DATA'}
                                  └ <function read_xds_list at 0x7f6eef55ab80>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/quartical/data_handling/ms_handler.py", line 31, in read_xds_list
    antenna_xds = xds_from_storage_table(ms_opts.path + "::ANTENNA")[0]
                  │                      │       └ 'A168_VLA_L_D_targets_allspw.ms'
                  │                      └ MSInputs(path='A168_VLA_L_D_targets_allspw.ms', data_column='CORRECTED_DATA', sigma_column=None, weight_column=None, time_chu...
                  └ <function xds_from_storage_table at 0x7f6eef54fe50>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/daskms/dask_ms.py", line 339, in xds_from_storage_table
    typ = store.type()
          │     └ <function DaskMSStore.type at 0x7f6f0dc1de50>
          └ DaskMSStore(file:///lustre/aoc/users/vparekh/A168/stimela/quartical/A168_VLA_L_D_targets_allspw.ms::ANTENNA)
  File "/users/vparekh/venv3/lib/python3.8/site-packages/daskms/fsspec_store.py", line 82, in type
    raise UnknownStoreTypeError(f"Unable to infer table type at {self.full_path}")
          └ <class 'daskms.fsspec_store.UnknownStoreTypeError'>

daskms.fsspec_store.UnknownStoreTypeError: Unable to infer table type at /lustre/aoc/users/vparekh/A168/stimela/quartical/A168_VLA_L_D_targets_allspw.ms/ANTENNA
JSKenyon commented
JSKenyon commented

Additionally, you can always produce an up-to-date, unpopulated config by running goquartical-config.