Error in Running example_multicase

Question

Error in Running example_multicase

nishsilva opened this issue 6 months ago · 20 comments

I am trying to run the example_multicase POD introduced in the new MDTF manual.

After following the relevant steps, once I run the POD I am getting below error. Looks like the program could not direct into mdtf_test_data folder in my local path. I was unable to figure out why this happens. Can someone help me understand this issue?

Thank you in advance.

> netige@crhtc59:/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics> ./mdtf -f example_multicase.jsonc
> POD convention and data convention are both no_translation. No data translation will be performed for case CMIP_Synthetic_r1i1p1f1_gr1_19800101-19841231.
> POD convention and data convention are both no_translation. No data translation will be performed for case CMIP_Synthetic_r1i1p1f1_gr1_19850101-19891231.
> Preprocessing data for example_multicase
> CRITICAL: **********************************************************************
> Uncaught exception:
> Traceback (most recent call last):
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/intake_esm/source.py", line 244, in _open_dataset
>     datasets = dask.compute(*datasets)
>                ^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/dask/base.py", line 665, in compute
>     results = schedule(dsk, keys, **kwargs)
>               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/intake_esm/source.py", line 66, in _open_dataset
>     url = fsspec.open_local(urlpath, **storage_options)
>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/fsspec/core.py", line 502, in open_local
>     with of as files:
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/fsspec/core.py", line 178, in __enter__
>     return [s.__enter__() for s in self]
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/fsspec/core.py", line 178, in <listcomp>
>     return [s.__enter__() for s in self]
>             ^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/fsspec/core.py", line 103, in __enter__
>     f = self.fs.open(self.path, mode=mode)
>         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/fsspec/spec.py", line 1293, in open
>     f = self._open(
>         ^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/fsspec/implementations/local.py", line 197, in _open
>     return LocalFileOpener(path, mode, fs=self, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/fsspec/implementations/local.py", line 322, in __init__
>     self._open()
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/fsspec/implementations/local.py", line 327, in _open
>     self.f = open(self.path, mode=self.mode)
>              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> FileNotFoundError: [Errno 2] No such file or directory: '/Users/jess/mdtf/inputdata/mdtf_test_data/CMIP_Synthetic_r1i1p1f1_gr1_19800101-19841231/day/CMIP_Synthetic_r1i1p1f1_gr1_19800101-19841231.tas.day.nc'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/mdtf_framework.py", line 238, in <module>
>     exit_code = main(prog_name='MDTF-diagnostics')
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
>     return self.main(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 1078, in main
>     rv = self.invoke(ctx)
>          ^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
>     return ctx.invoke(self.callback, **ctx.params)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 783, in invoke
>     return __callback(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
>     return f(get_current_context(), *args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/mdtf_framework.py", line 196, in main
>     cat_subset = data_pp.process(cases, ctx.config, model_paths.MODEL_WORK_DIR)
>                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/src/preprocessor.py", line 1190, in process
>     cat_subset = self.query_catalog(case_list, config.DATA_CATALOG)
>                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/src/preprocessor.py", line 901, in query_catalog
>     cat_dict = cat_dict | cat_subset.to_dataset_dict(
>                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/pydantic/validate_call_decorator.py", line 59, in wrapper_function
>     return validate_call_wrapper(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/pydantic/_internal/_validate_call.py", line 81, in __call__
>     res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/intake_esm/core.py", line 686, in to_dataset_dict
>     raise exc
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/intake_esm/core.py", line 682, in to_dataset_dict
>     key, ds = task.result()
>               ^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/concurrent/futures/_base.py", line 449, in result
>     return self.__get_result()
>            ^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
>     raise self._exception
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/concurrent/futures/thread.py", line 58, in run
>     result = self.fn(*self.args, **self.kwargs)
>              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/intake_esm/core.py", line 833, in _load_source
>     return key, source.to_dask()
>                 ^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/intake_esm/source.py", line 272, in to_dask
>     self._load_metadata()
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/intake/source/base.py", line 283, in _load_metadata
>     self._schema = self._get_schema()
>                    ^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/intake_esm/source.py", line 208, in _get_schema
>     self._open_dataset()
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/intake_esm/source.py", line 264, in _open_dataset
>     raise ESMDataSourceError(
> intake_esm.source.ESMDataSourceError: Failed to load dataset with key='CMIP.synthetic.day.r1i1p1f1.day.gr.atmos.r1i1p1f1.1980-01-01-1984-12-31'
>                  You can use `cat['CMIP.synthetic.day.r1i1p1f1.day.gr.atmos.r1i1p1f1.1980-01-01-1984-12-31'].df` to inspect the assets/files for this key.

Answer 1 · 2024-04-24T21:55:41.000Z

@nishsilva You have to either generate a new catalog for the synthetic data on your local filesystem, or manually change the root paths in diagnostic/example_multicase/esm_catalog_CMIP_synthetic_r1i1p1f1_gr1.csv to your local synthetic data directory, and update the root directory in the catalog file entry in diagnostic/example_multicase/esm_catalog_CMIP_synthetic_r1i1p1f1_gr1.json .

Answer 2 · 2024-04-25T15:50:14.000Z

@wrongkindofdoctor Thank you. I was able to correct this issue following your instructions. Once this was corrected and run the package, I got an AttributeError indicating that the 'CondaEnvironmentManager' object does not have an attribute named 'log'. Do you think this has to do something with my Conda environment?

> netige@crhtc12:/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics> ./mdtf -f example_multicase.jsonc
> POD convention and data convention are both no_translation. No data translation will be performed for case CMIP_Synthetic_r1i1p1f1_gr1_19800101-19841231.
> POD convention and data convention are both no_translation. No data translation will be performed for case CMIP_Synthetic_r1i1p1f1_gr1_19850101-19891231.
> Preprocessing data for example_multicase
> WARNING: Dataset: discrepancy for attr 'intake_esm_vars': '['tas']' != '['tas']'.
> No standard_name for 'time_bnds' found in dataset; setting to 'time'.
> No units for 'time_bnds' found in dataset; setting to 'days since 1980-01-01'.
> No standard_name for 'lat_bnds' found in dataset; setting to 'latitude'.
> No units for 'lat_bnds' found in dataset; setting to 'degrees_north'.
> No standard_name for 'lon_bnds' found in dataset; setting to 'longitude'.
> No units for 'lon_bnds' found in dataset; setting to 'degrees_east'.
> WARNING: ('Conflict in scalar coordinates for tas: expected ', '[]; dataset has [('height', 'Z')].')
> Variable <#None:example_multicase.tas> data starts at hour 0
> Requested dates for <#None:example_multicase.tas> coincide with range of dataset '1980-01-01:00-00-00 -- 1984-12-31:00-00-00'; left unmodified.
> Converted units on <#None:example_multicase.tas>.
> WARNING: Dataset: discrepancy for attr 'intake_esm_vars': '['tas']' != '['tas']'.
> No standard_name for 'time_bnds' found in dataset; setting to 'time'.
> No units for 'time_bnds' found in dataset; setting to 'days since 1985-01-01'.
> No standard_name for 'lat_bnds' found in dataset; setting to 'latitude'.
> No units for 'lat_bnds' found in dataset; setting to 'degrees_north'.
> No standard_name for 'lon_bnds' found in dataset; setting to 'longitude'.
> No units for 'lon_bnds' found in dataset; setting to 'degrees_east'.
> WARNING: ('Conflict in scalar coordinates for tas: expected ', '[]; dataset has [('height', 'Z')].')
> Variable <#None:example_multicase.tas> data ends at hour 0
> Requested dates for <#None:example_multicase.tas> coincide with range of dataset '1985-01-01:00-00-00 -- 1989-12-31:00-00-00'; left unmodified.
> Converted units on <#None:example_multicase.tas>.
> Getting list of assets...
> 
> Successfully wrote ESM catalog json file to: file:///glade/work/netige/mdtf_Apr24_2/mdtf/wkdir/MDTF_output.v11/MDTF_postprocessed_data.json
> CRITICAL: **********************************************************************
> Uncaught exception:
> Traceback (most recent call last):
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/mdtf_framework.py", line 238, in <module>
>     exit_code = main(prog_name='MDTF-diagnostics')
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
>     return self.main(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 1078, in main
>     rv = self.invoke(ctx)
>          ^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
>     return ctx.invoke(self.callback, **ctx.params)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 783, in invoke
>     return __callback(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
>     return f(get_current_context(), *args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/mdtf_framework.py", line 205, in main
>     run_mgr.setup()
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/src/environment_manager.py", line 508, in setup
>     self.env_mgr.create_environment(env)
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/src/environment_manager.py", line 179, in create_environment
>     self.log.warning(("Conda env '%s' not found (grepped for '%s'); "
>     ^^^^^^^^
> AttributeError: 'CondaEnvironmentManager' object has no attribute 'log'

Answer 3 · 2024-04-25T18:22:45.000Z

@nishsilva I added a log attribute to the CondaEnvironmentManager class in the main branch. Please pull the update into your working branch and try again. Make sure that conda_env_root is set in your runtime configuration file.

Answer 4 · 2024-04-29T13:03:38.000Z

@wrongkindofdoctor I pulled the update and run the package again. Now I am getting below type error indicating that 'Logger' object is not callable.

> netige@crhtc62:/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics> ./mdtf -f example_multicase.jsonc
> POD convention and data convention are both no_translation. No data translation will be performed for case CMIP_Synthetic_r1i1p1f1_gr1_19800101-19841231.
> POD convention and data convention are both no_translation. No data translation will be performed for case CMIP_Synthetic_r1i1p1f1_gr1_19850101-19891231.
> Preprocessing data for example_multicase
> WARNING: Dataset: discrepancy for attr 'intake_esm_vars': '['tas']' != '['tas']'.
> No standard_name for 'time_bnds' found in dataset; setting to 'time'.
> No units for 'time_bnds' found in dataset; setting to 'days since 1980-01-01'.
> No standard_name for 'lat_bnds' found in dataset; setting to 'latitude'.
> No units for 'lat_bnds' found in dataset; setting to 'degrees_north'.
> No standard_name for 'lon_bnds' found in dataset; setting to 'longitude'.
> No units for 'lon_bnds' found in dataset; setting to 'degrees_east'.
> WARNING: ('Conflict in scalar coordinates for tas: expected ', '[]; dataset has [('height', 'Z')].')
> Variable <#None:example_multicase.tas> data starts at hour 0
> Requested dates for <#None:example_multicase.tas> coincide with range of dataset '1980-01-01:00-00-00 -- 1984-12-31:00-00-00'; left unmodified.
> Converted units on <#None:example_multicase.tas>.
> WARNING: Dataset: discrepancy for attr 'intake_esm_vars': '['tas']' != '['tas']'.
> No standard_name for 'time_bnds' found in dataset; setting to 'time'.
> No units for 'time_bnds' found in dataset; setting to 'days since 1985-01-01'.
> No standard_name for 'lat_bnds' found in dataset; setting to 'latitude'.
> No units for 'lat_bnds' found in dataset; setting to 'degrees_north'.
> No standard_name for 'lon_bnds' found in dataset; setting to 'longitude'.
> No units for 'lon_bnds' found in dataset; setting to 'degrees_east'.
> WARNING: ('Conflict in scalar coordinates for tas: expected ', '[]; dataset has [('height', 'Z')].')
> Variable <#None:example_multicase.tas> data ends at hour 0
> Requested dates for <#None:example_multicase.tas> coincide with range of dataset '1985-01-01:00-00-00 -- 1989-12-31:00-00-00'; left unmodified.
> Converted units on <#None:example_multicase.tas>.
> Getting list of assets...
> 
> Successfully wrote ESM catalog json file to: file:///glade/work/netige/mdtf_Apr24_2/mdtf/wkdir/MDTF_output.v14/MDTF_postprocessed_data.json
> CRITICAL: **********************************************************************
> Uncaught exception:
> Traceback (most recent call last):
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/mdtf_framework.py", line 238, in <module>
>     exit_code = main(prog_name='MDTF-diagnostics')
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
>     return self.main(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 1078, in main
>     rv = self.invoke(ctx)
>          ^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
>     return ctx.invoke(self.callback, **ctx.params)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 783, in invoke
>     return __callback(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
>     return f(get_current_context(), *args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/mdtf_framework.py", line 205, in main
>     run_mgr.setup()
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/src/environment_manager.py", line 511, in setup
>     self.env_mgr.create_environment(env)
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/src/environment_manager.py", line 182, in create_environment
>     self.log.warning(("Conda env '%s' not found (grepped for '%s'); "
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/src/util/logs.py", line 360, in warning
>     self.log(logging.WARNING, msg, *args, **kw)
> TypeError: 'Logger' object is not callable

Answer 5 · 2024-04-29T14:10:48.000Z

@nishsilva Sorry about that. I fixed the log object calls in the environment manager module. Please pull the main branch updates in and try again.

Answer 6 · 2024-04-29T18:33:34.000Z

@wrongkindofdoctor Thanks for this. This fix managed to run the package. But I ended up getting a different error. This maybe something from my end. But thought of getting your input as well. When running the example_multicase POD I end up getting below error in the log file:

> ### Start execution of <#c2eP:example_multicase>
> --------------------------------------------------------------------------------
> _CONDA_EXE=/glade/u/apps/opt/conda/bin/conda
> _CONDA_ROOT=/glade/u/apps/opt/conda
> 
> EnvironmentLocationNotFound: Not a conda environment: /glade/u/apps/opt/conda/envs/_MDTF_python3_base

My Conda root doesn't have write access, so I'm installing Conda to a different location following the instructions. Does this have to do something with that or am I doing something wrong?

I am attaching the log file for the POD which includes this error message.
example_multicase.log

I appreciate it taking your time to take a look into this!

Answer 7 · 2024-04-29T19:16:58.000Z

@nishsilva This does seem to be a write access issue. It appears that you were using an NCAR-managed version Anaconda installed with Spack, so installing your own Conda package to a writable location could resolve your problem. I have had nothing but trouble our institutionally-managed Conda at GFDL, and always install my own miniconda binaries.

Answer 8 · 2024-05-02T17:57:28.000Z

@wrongkindofdoctor Thank you. This makes sense. While working on this I realized that even though I provide the "conda_env_root" in the runtime configuration file, it treats as blank and tries to look for the environments in conda_root/envs.

I set my conda roots as below in the configuration file.


 // Location of the Anaconda/miniconda or micromamba installation to use for managing
  // dependencies (path returned by running `conda info --base` or `micromamba info`.)
  //"conda_root": "/glade/u/apps/opt/conda",
  "conda_root": "/glade/u/home/netige/miniconda3",

  // Directory containing the framework-specific conda environments. This should
  // be equal to the "--env_dir" flag passed to conda_env_setup.sh. If left
  // blank, the framework will look for its environments in conda_root/envs
  "conda_env_root": "/glade/u/home/netige/miniconda3/",

I get below error:

>  ### Start execution of <#jPCL:example_multicase>
> --------------------------------------------------------------------------------
> _CONDA_EXE=/glade/u/home/netige/miniconda3/bin/conda
> _CONDA_ROOT=/glade/u/home/netige/miniconda3
> 
> EnvironmentLocationNotFound: Not a conda environment: /glade/u/home/netige/miniconda3/envs/_MDTF_python3_base

Again, not sure if this is an issue from my end.

Thank you for your time.

Answer 9 · 2024-05-02T20:04:59.000Z

I've had an error like this in the past because the conda_env_root shows up in both the runtime config (jsonc or now yaml) file, as you've referenced, but also is in the executable mdtf in the top dir using the CONDA_ENV_DIR setting variables when it was installed (eg ./src/conda/conda_env_setup.sh --all --conda_root $CONDA_ROOT --env_dir $CONDA_ENV_DIR`) You can see it in the mdtf executable: % grep conda mdtf conda activate /glade/work/bundy/miniconda2/envs.MDTF.refactor_pp.20240328/_MDTF_base I've had success simply editing the path in the mdtf file; it is just a bash script.

…

On Thu, May 2, 2024 at 11:57 AM nishsilva ***@***.***> wrote: @wrongkindofdoctor <https://github.com/wrongkindofdoctor> Thank you. This makes sense. While working on this I realized that even though I provide the "conda_env_root" in the runtime configuration file, it treats as blank and tries to look for the environments in conda_root/envs. I set my conda roots as below in the configuration file. // Location of the Anaconda/miniconda or micromamba installation to use for managing // dependencies (path returned by running `conda info --base` or `micromamba info`.) //"conda_root": "/glade/u/apps/opt/conda", "conda_root": "/glade/u/home/netige/miniconda3", // Directory containing the framework-specific conda environments. This should // be equal to the "--env_dir" flag passed to conda_env_setup.sh. If left // blank, the framework will look for its environments in conda_root/envs "conda_env_root": "/glade/u/home/netige/miniconda3/", I get below error: > ### Start execution of <#jPCL:example_multicase> > -------------------------------------------------------------------------------- > _CONDA_EXE=/glade/u/home/netige/miniconda3/bin/conda > _CONDA_ROOT=/glade/u/home/netige/miniconda3 > > EnvironmentLocationNotFound: Not a conda environment: /glade/u/home/netige/miniconda3/envs/_MDTF_python3_base Again, not sure if this is an issue from my end. Thank you for your time. — Reply to this email directly, view it on GitHub <#549 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHLJVQBFB4SWJEWM3AJMYJ3ZAJ5B5AVCNFSM6AAAAABGXQM636VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJRGE3TMOBZGQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- My working day may not be the same as your working day. Please do not feel obliged to reply to this email outside of your normal hours.

Answer 10 · 2024-05-06T16:18:21.000Z

@bitterbark thank you for this. Could you be more specific whether I need to change /src/conda/conda_env_setup.sh before installing the framework or else? And where in /src/conda/conda_env_setup.sh should be modified. Thank you!

Answer 11 · 2024-05-07T17:07:24.000Z

You can make the change in the executable file called mdtf (top directory of your install) anywhere it says conda activate, just change the path to the CONDA_ENV_ROOT you want. eg: conda activate $CONDA_ENV_ROOT/_MDTF_base This is just a quick fix (and a test to see if this is the problem you are having). Normally you provide the CONDA_ENV_ROOT when you do the install with the command: /src/conda/conda_env_setup.sh --all --conda_root $CONDA_ROOT --env_dir $CONDA_ENV_DIR` This command writes the mdtf executable script. Let me know if that doesn't make sense and/or work! Cheers, Dani

…

On Mon, May 6, 2024 at 10:18 AM nishsilva ***@***.***> wrote: @bitterbark <https://github.com/bitterbark> thank you for this. Could you be more specific whether I need to change /src/conda/conda_env_setup.sh before installing the framework or else? And where in /src/conda/conda_env_setup.sh should be modified. Thank you! — Reply to this email directly, view it on GitHub <#549 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHLJVQGXOJNSQP44KLWND6TZA6UOFAVCNFSM6AAAAABGXQM636VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJWGQZDKNJRGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- My working day may not be the same as your working day. Please do not feel obliged to reply to this email outside of your normal hours.

Answer 12 · 2024-05-08T13:43:41.000Z

@bitterbark Thanks Dani for this. Following your instructions I modified the shell file to my CONDA_ENV_ROOT.

I first did below modification and ran the package.


if [ -z "$_CONDA_ENV_ROOT" ]; then
    echo "conda activate _MDTF_base" >> "$_CONDA_WRAPPER"
else
    #echo "conda activate ${_CONDA_ENV_ROOT}/_MDTF_base" >> "$_CONDA_WRAPPER"
    echo "conda activate /glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base" >> "$_CONDA_WRAPPER"

which did not work out. Then I did the following modification:

if [ -z "$_CONDA_ENV_ROOT" ]; then
    #echo "conda activate _MDTF_base" >> "$_CONDA_WRAPPER"
    echo "conda activate /glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base" >> "$_CONDA_WRAPPER"
else
    #echo "conda activate ${_CONDA_ENV_ROOT}/_MDTF_base" >> "$_CONDA_WRAPPER"
    echo "conda activate /glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base" >> "$_CONDA_WRAPPER"

In both instances the framework is searching for the Conda package in the CONDA_ROOT/env (see below) even though I provide the CONDA_ENV_ROOT in the runtime configuration file.

> ## Start execution of <#8kks:example_multicase>
> --------------------------------------------------------------------------------
> _CONDA_EXE=/glade/u/home/netige/miniconda3/bin/conda
> _CONDA_ROOT=/glade/u/home/netige/miniconda3
> 
> EnvironmentLocationNotFound: Not a conda environment: /glade/u/home/netige/miniconda3/envs/_MDTF_python3_base

Not sure if I am modifying the wrong location.

The only workaround at this point is installing the package to CONDA_ROOT/env (which_works) but this consumes a lot of space in my home directory.

Please let me know if you have any more ideas on this.

Thanks,
Nish

Answer 13 · 2024-05-08T17:05:47.000Z

The modifications you show are to 'echo' commands which give output while running but aren't the actual activate commands being used. My guess is what needs to be modified are the _CONDA_ENV_ROOT settings. However, this is not what the file looks like that I usually modify. I wonder if a quick zoom or google meets call would be a quicker way for us to figure this out?

…

On Wed, May 8, 2024 at 7:44 AM nishsilva ***@***.***> wrote: @bitterbark <https://github.com/bitterbark> Thanks Dani for this. Following your instructions I modified the shell file to my CONDA_ENV_ROOT. I first did below modification and ran the package. if [ -z "$_CONDA_ENV_ROOT" ]; then echo "conda activate _MDTF_base" >> "$_CONDA_WRAPPER" else #echo "conda activate ${_CONDA_ENV_ROOT}/_MDTF_base" >> "$_CONDA_WRAPPER" echo "conda activate /glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base" >> "$_CONDA_WRAPPER" which did not work out. Then I did the following modification: if [ -z "$_CONDA_ENV_ROOT" ]; then #echo "conda activate _MDTF_base" >> "$_CONDA_WRAPPER" echo "conda activate /glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base" >> "$_CONDA_WRAPPER" else #echo "conda activate ${_CONDA_ENV_ROOT}/_MDTF_base" >> "$_CONDA_WRAPPER" echo "conda activate /glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base" >> "$_CONDA_WRAPPER" In both instances the framework is searching for the Conda package in the CONDA_ROOT/env (see below) even though I provide the CONDA_ENV_ROOT in the runtime configuration file. > ## Start execution of <#8kks:example_multicase> > -------------------------------------------------------------------------------- > _CONDA_EXE=/glade/u/home/netige/miniconda3/bin/conda > _CONDA_ROOT=/glade/u/home/netige/miniconda3 > > EnvironmentLocationNotFound: Not a conda environment: /glade/u/home/netige/miniconda3/envs/_MDTF_python3_base Not sure if I am modifying the wrong location. The only workaround at this point is installing the package to CONDA_ROOT/env (which_works) but this consumes a lot of space in my home directory. Please let me know if you have any more ideas on this. Thanks, Nish — Reply to this email directly, view it on GitHub <#549 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHLJVQGSIQP7AZHO47D6HLLZBIT2HAVCNFSM6AAAAABGXQM636VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBQGYYTEMBZHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- My working day may not be the same as your working day. Please do not feel obliged to reply to this email outside of your normal hours.

Answer 14 · 2024-05-08T17:17:08.000Z

@bitterbark Thanks Dani. It will be helpful to do google meet/zoom call. Shall I email to your NCAR email?

Answer 15 · 2024-05-08T17:17:19.000Z

It looks to me like you are modifying the src/conda/conda_env_setup.sh file. This shouldn't need to be modified: you should be able to call it with your CONDA_ENV_ROOT setting, like this: ./src/conda/conda_env_setup.sh --all --conda_root $CONDA_ROOT --env_dir $CONDA_ENV_DIR You will need to either set the the CONDA_ROOT and the CONDA_ENV_DIR variables or replace them in the call: This might work: ./src/conda/conda_env_setup.sh --all --conda_root /glade/u/apps/opt/conda/condabin/ --env_dir /glade/work/netige/mdtf_Apr24_2/mdtf (This is the default conda_root, found with `which conda` on casper). But I have trouble with the NCAR-managed conda envs (as Jess mentioned above). You're welcome to point to mine for now setenv CONDA_ROOT /glade/u/home/bundy/miniconda2 Eventually you should have your own installation (in case I move or rename it or something) but for now this will tell you if everything else is working!

…

On Wed, May 8, 2024 at 11:05 AM Dani Coleman ***@***.***> wrote: The modifications you show are to 'echo' commands which give output while running but aren't the actual activate commands being used. My guess is what needs to be modified are the _CONDA_ENV_ROOT settings. However, this is not what the file looks like that I usually modify. I wonder if a quick zoom or google meets call would be a quicker way for us to figure this out? On Wed, May 8, 2024 at 7:44 AM nishsilva ***@***.***> wrote: > @bitterbark <https://github.com/bitterbark> Thanks Dani for this. > Following your instructions I modified the shell file to my CONDA_ENV_ROOT. > > I first did below modification and ran the package. > > > if [ -z "$_CONDA_ENV_ROOT" ]; then > echo "conda activate _MDTF_base" >> "$_CONDA_WRAPPER" > else > #echo "conda activate ${_CONDA_ENV_ROOT}/_MDTF_base" >> "$_CONDA_WRAPPER" > echo "conda activate /glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base" >> "$_CONDA_WRAPPER" > > which did not work out. Then I did the following modification: > > if [ -z "$_CONDA_ENV_ROOT" ]; then > #echo "conda activate _MDTF_base" >> "$_CONDA_WRAPPER" > echo "conda activate /glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base" >> "$_CONDA_WRAPPER" > else > #echo "conda activate ${_CONDA_ENV_ROOT}/_MDTF_base" >> "$_CONDA_WRAPPER" > echo "conda activate /glade/work/netige/mdtf_Apr24_2/mdtf/_MDTF_base" >> "$_CONDA_WRAPPER" > > In both instances the framework is searching for the Conda package in the > CONDA_ROOT/env (see below) even though I provide the CONDA_ENV_ROOT in the > runtime configuration file. > > > ## Start execution of <#8kks:example_multicase> > > -------------------------------------------------------------------------------- > > _CONDA_EXE=/glade/u/home/netige/miniconda3/bin/conda > > _CONDA_ROOT=/glade/u/home/netige/miniconda3 > > > > EnvironmentLocationNotFound: Not a conda environment: /glade/u/home/netige/miniconda3/envs/_MDTF_python3_base > > Not sure if I am modifying the wrong location. > > The only workaround at this point is installing the package to > CONDA_ROOT/env (which_works) but this consumes a lot of space in my home > directory. > > Please let me know if you have any more ideas on this. > > Thanks, > Nish > > — > Reply to this email directly, view it on GitHub > <#549 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AHLJVQGSIQP7AZHO47D6HLLZBIT2HAVCNFSM6AAAAABGXQM636VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBQGYYTEMBZHA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- My working day may not be the same as your working day. Please do not feel obliged to reply to this email outside of your normal hours.

-- My working day may not be the same as your working day. Please do not feel obliged to reply to this email outside of your normal hours.

Answer 16 · 2024-05-08T17:57:49.000Z

Just to update the issue: We met and I could see I have been on the wrong track: his paths are set correctly in mdtf* and the settings jsonc file. However, at run time, src/environment_manager.py:pre_run_setup() reports the conda env dir to be conda_root/envs. I feel like I've run into this in the past, where something is assuming that envs is under root, but I haven't been able to find a record in my notes or reported issues. There is a chance I just thought this was happening but it turned out to be a different error. So I will work on chasing it down in his version, but meanwhile if it sounds familiar to anyone else, please chime in!

…

On Wed, May 8, 2024 at 11:17 AM nishsilva ***@***.***> wrote: @bitterbark <https://github.com/bitterbark> Thanks Dani. It will be helpful to do google meet/zoom call. Shall I email to your NCAR email? — Reply to this email directly, view it on GitHub <#549 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHLJVQCNH3AVXGI3GV5HFWLZBJM2VAVCNFSM6AAAAABGXQM636VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBRGA2DOMZRGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- My working day may not be the same as your working day. Please do not feel obliged to reply to this email outside of your normal hours.

Answer 17 · 2024-05-20T16:03:14.000Z

@wrongkindofdoctor after installing the Conda environment to conda_root/envs I was successfully able to run the example_multicase (13 days ago). But when I ran it last Friday, I was facing the below error. Any idea why this issue has arisen?


> netige@crhtc38:/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics> ./mdtf -f example_multicase.jsonc
> POD convention and data convention are both no_translation. No data translation will be performed for case CMIP_Synthetic_r1i1p1f1_gr1_19800101-19841231.
> POD convention and data convention are both no_translation. No data translation will be performed for case CMIP_Synthetic_r1i1p1f1_gr1_19850101-19891231.
> Preprocessing data for example_multicase
> CRITICAL: **********************************************************************
> Uncaught exception:
> Traceback (most recent call last):
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/mdtf_framework.py", line 238, in <module>
>     exit_code = main(prog_name='MDTF-diagnostics')
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
>     return self.main(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 1078, in main
>     rv = self.invoke(ctx)
>          ^^^^^^^^^^^^^^^^
>   File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
>     return ctx.invoke(self.callback, **ctx.params)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.11/site-packages/click/core.py", line 783, in invoke
>     return __callback(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
>     return f(get_current_context(), *args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/mdtf_framework.py", line 196, in main
>     cat_subset = data_pp.process(cases, ctx.config, model_paths.MODEL_WORK_DIR)
>                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/src/preprocessor.py", line 1184, in process
>     cat_subset = self.query_catalog(case_list, config.DATA_CATALOG)
>                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/work/netige/mdtf_Apr24_2/mdtf/MDTF-diagnostics/src/preprocessor.py", line 884, in query_catalog
>     cat_subset = cat.search(**case_d.query)
>                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.11/site-packages/pydantic/validate_call_decorator.py", line 59, in wrapper_function
>     return validate_call_wrapper(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.11/site-packages/pydantic/_internal/_validate_call.py", line 81, in __call__
>     res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.11/site-packages/intake_esm/core.py", line 393, in search
>     esmcat_results = self.esmcat.search(require_all_on=require_all_on, query=query)
>                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.11/site-packages/intake_esm/cat.py", line 382, in search
>     else QueryModel(
>          ^^^^^^^^^^^
>   File "/glade/u/home/netige/miniconda3/envs/_MDTF_base/lib/python3.11/site-packages/pydantic/main.py", line 176, in __init__
>     self.__pydantic_validator__.validate_python(data, self_instance=self)
> pydantic_core._pydantic_core.ValidationError: 1 validation error for QueryModel
>   Value error, Column variable not in columns ['activity_id', 'branch_method', 'branch_time_in_child', 'branch_time_in_parent', 'experiment', 'experiment_id', 'frequency', 'grid', 'grid_label', 'institution_id', 'nominal_resolution', 'parent_activity_id', 'parent_experiment_id', 'parent_source_id', 'parent_time_units', 'parent_variant_label', 'product', 'realm', 'source_id', 'source_type', 'sub_experiment', 'sub_experiment_id', 'table_id', 'variable_id', 'variant_label', 'member_id', 'standard_name', 'long_name', 'units', 'vertical_levels', 'init_year', 'start_time', 'end_time', 'time_range', 'path', 'version'] [type=value_error, input_value={'query': {'frequency': '...ge', 'path', 'version']}, input_type=dict]
>     For further information visit https://errors.pydantic.dev/2.7/v/value_error

Answer 18 · 2024-05-20T16:45:56.000Z

@nishsilva The main branch has had several updates, and the catalog query requires refinement to work with the AM5 (and likely other) data besides the basic test case. If you updated your branch , you might want to try checking out the commit before any updates from last week, which seems like it should be this one: 56c3c8e. Otherwise, I'm not sure what the issues is at them moment.

Answer 19 · 2024-05-21T16:43:43.000Z

@wrongkindofdoctor thank you, Jess. Once I check out the 56c3c8e the example_multicase ran without any issue.

I am currently developing a new 4th-generation POD for a Sea Level Diagnostic. It seems testing that POD by keeping in this older commit should work for the time being.

Answer 20 · 2024-05-21T18:43:05.000Z

@nishsilva Great! Thanks for doing your best to work with new code. Trying to coordinate your development with a major framework transition adds another, thankfully temporary, layer of complexity to the submission process.