KeyError raised when any of the variable handler fails

[chengzhuzhang]

Initially reported by @forsyth2 in e3sm-unified rc testing, when any of the cmip variable handler fails, KeyError raised and the program exist. The expected behavior is to process every available target cmip variables, as previous versions.
My standalone e2c test, v1.9.0:

2023-08-01 20:45:20,343_343:INFO:main:input_path = /p/user_pub/e3sm/zhang40/e3sm_to_cmip_data/test_reference_v190/rgr
2023-08-01 20:45:20,343_343:INFO:main:output_path = /p/user_pub/e3sm/zhang40/e3sm_to_cmip_data/test_reference_v190
2023-08-01 20:45:20,343_343:INFO:main:precheck_path = None
[*] Loaded `tas` handler for variable `tas`.
[*] Loaded `pr` handler for variable `pr`.
[*] Loaded `od550aer` handler for variable `od550aer`.
[*] Loaded `abs550aer` handler for variable `abs550aer`.
[*] Writing log output to: /p/user_pub/e3sm/zhang40/e3sm_to_cmip_data/test_reference_v190/converter.log
[+] Running CMOR handlers in parallel
  0%|                                                     | 0/4 [00:00<?, ?it/s][-] abs550aer: Unable to find input files for AODABS
 25%|███████████▎                                 | 1/4 [00:01<00:03,  1.28s/it][-] Error running handler abs550aer
100%|█████████████████████████████████████████████| 4/4 [00:01<00:00,  3.10it/s]
[+] 3 of 4 handlers complete
[-] abs550aer failed to complete

v1.10.0rc:

2023-08-01 20:43:01,023_023:INFO:__init__:--------------------------------------
2023-08-01 20:43:01,024_024:INFO:__init__:| E3SM to CMIP Configuration
2023-08-01 20:43:01,024_024:INFO:__init__:--------------------------------------
2023-08-01 20:43:01,024_024:INFO:__init__:    * var_list='['tas', 'pr', 'od550aer', 'abs550aer']'
2023-08-01 20:43:01,024_024:INFO:__init__:    * input_path='/p/user_pub/e3sm/zhang40/e3sm_to_cmip_data/test_reference_v190/rgr'
2023-08-01 20:43:01,024_024:INFO:__init__:    * output_path='/p/user_pub/e3sm/zhang40/e3sm_to_cmip_data/test_reference_v190'
2023-08-01 20:43:01,024_024:INFO:__init__:    * precheck_path='None'
2023-08-01 20:43:01,024_024:INFO:__init__:    * freq='mon'
2023-08-01 20:43:01,025_025:INFO:__init__:    * realm='atm'
2023-08-01 20:43:01,025_025:INFO:__init__:    * Writing log output file to: logs/20230801_204301_015530
Traceback (most recent call last):
  File "/home/zhang40/miniconda3/envs/e2c_nco511/bin/e3sm_to_cmip", line 8, in <module>
    sys.exit(main())
  File "/home/zhang40/miniconda3/envs/e2c_nco511/lib/python3.9/site-packages/e3sm_to_cmip/__main__.py", line 792, in main
    app = E3SMtoCMIP(args)
  File "/home/zhang40/miniconda3/envs/e2c_nco511/lib/python3.9/site-packages/e3sm_to_cmip/__main__.py", line 152, in __init__
    self.handlers = self._get_handlers()
  File "/home/zhang40/miniconda3/envs/e2c_nco511/lib/python3.9/site-packages/e3sm_to_cmip/__main__.py", line 219, in _get_handlers
    handlers = derive_handlers(
  File "/home/zhang40/miniconda3/envs/e2c_nco511/lib/python3.9/site-packages/e3sm_to_cmip/cmor_handlers/utils.py", line 220, in derive_handlers
    raise KeyError(
KeyError: "No handlers could be derived for the variables: ['abs550aer']. Make sure the input E3SM datasets have the variables needed derivation."

Below is the script to reproduce on acme1:

#!/bin/bash

# Conda Environment
# -----------------
# Either comment out the lines below to use the E3SM Unified environment
# or use a local environment.
#source /usr/local/e3sm_unified/envs/load_latest_e3sm_unified_acme1.sh
#conda activate e2c_nco511_13

# -----------------
# SETUP
# -----------------
exp=historical
caseid=v2.LR.${exp}_0101
start=1850
end=1850
ypf=1

# -----------------
# PATHS
# -----------------
e2c_path=/p/user_pub/e3sm/zhang40/e3sm_to_cmip_data
model_data=$e2c_path/model-output
result_dir=${e2c_path}/test_reference_v1100

rgr_dir=${result_dir}/rgr
rgr_dir_vert=${result_dir}/rgr_vert
rgr_dir_vert_plev=${result_dir}/rgr_vert_plev
native_dir=${result_dir}/native

map_file=${e2c_path}/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
tables_path=${e2c_path}/cmor/cmip6-cmor-tables/Tables/
metadata_path=${e2c_path}/user_metadata.json

# NOTE: Space is not accepted in nco var list
## ------------------------------------------------------
## TEST CASE - atm monthly h0
## ------------------------------------------------------
input_path=${model_data}/v2.eam_input
flags='-7 --dfl_lvl=1 --no_cll_msr'
raw_var_list="TREFHT,PRECC,PRECL,AODVIS"

# 1. atm 2D variables
#--------------------------
cmip_var_list="tas, pr, od550aer, abs550aer"
ncclimo -P eam -j 1 --map=${map_file} --start=$start --end=$end --ypf=$ypf --split -c $caseid -o ${native_dir} -O ${rgr_dir} -v ${raw_var_list} -i ${input_path} ${flags}
# CMORIZE Atmosphere monthly variables: 2D and model level 3D variables (CLOUD,CLDICE,CLDLIQ)
e3sm_to_cmip -i ${rgr_dir} -o $result_dir  -v ${cmip_var_list} -t $tables_path -u ${metadata_path}