NOAA-GFDL/MDTF-diagnostics

Input data missing for ENSO_MSE POD

wrongkindofdoctor opened this issue · 5 comments

wrongkindofdoctor commented

@jhafner2

The scatter_0*.ncl scripts in the ENSO_MSE POD require input observational files that are missing from the input data file you sent when you submitted the POD. Please provide the following files by emailing me a tarball, or posting the files on an ftp server and sending the link:

  • central_pacific_MSE_terms_NEW_MODEL.txt
  • eastern_pacific_MSE_terms_NEW_MODEL.txt
jhafner2 commented
wrongkindofdoctor commented

@jhafner2 Okay, I'll comb through the code more deeply. I suspect it's because the location that these files are being written to is not parsed by the framework. I'm also having issues running ENSO_RWS because the input data files are not being copied to the working directory (likely because they are in several nested directories within inputdata/obs_data/ENSO_RWS). I'll see if moving all input obs data to inputdata/obs_data/ENSO_RWS, and redefining the hardcoded directories in the PODs solves the issue.

wrongkindofdoctor commented

@jhafner2 I've been testing the ENSO_MSE POD with the CESM data you provided, and I am getting errors that none of the model plots are getting output to their directories. There are several warnings originating from the ncl scripts that files are missing. All of the plots for the obs_data, however, are generated properly.

When you run the ENSO_MSE and ENSO_RWS PODs, do they write the plots to the correct locations (i.e., do the WKDIR/[CASENAME]/[POD_NAME]/[POD_NAME].log contents indicate that the PODs completed successfully, or are there errors for missing files)?

Output from ENSO_MSE.log (Skipped the SCATTER module, but the ENSO central and eastern Pac plots are also missing if I run it):

=========== COMPOSITE.py =======================================
   Start of Composite Module calculations  2021-12-22 11:24
===============================================================
  The following parameters are set in the Composite Module Calculations  
      the reference area for SST indices calculations is selected to:        
      lon =  190.0  -  240.0  E lat =  -5.0  -  5.0 N
      ENSO indices  based on SST reference anomalies +/-  1.0  of SST sigma
      Selected season  is :  DJF
      Selected year span for composites is :  1950 / 2005
      Selected model  :  CESM2
The following elements will be calculated 
 Seasonal Composites for El Nino/La Nina years 
 2 Year life cycle of ENSO:  Year(0) and Year(1) 
 Year (0) = developing phase and Year(1) = decaying phase 
       Reference area SST correlations will be calculated 
      Regressions to reference area SST will be calculated 
NCL routine /home//mdtf/MDTF-diagnostics/diagnostics/ENSO_MSE/COMPOSITE/NCL_DATA/get_composites.ncl:
	b'(0)\tstarted Wed Dec 22 11:24:30 AM EST 2021\n(0)\tThe number of  El Nino events is : 15\n(0)\tThe code will proceed  with this number of El Nino events\n(0)\tThe number of  El Nino events is : 17\n(0)\tThe code will proceed  with this number of La Nina events\nfatal:systemfunc: cannot create child process:[errno=12]\n\rfatal:["Execute.c":8637]:Execute: Error occurred at or near line 261 in file /home//mdtf/MDTF-diagnostics/diagnostics/ENSO_MSE/COMPOSITE/NCL_DATA/get_composites.ncl\n\n\r'
NCL routine /home//mdtf/MDTF-diagnostics/diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl:
	b'warning:_NclOpenFile: cannot open file </home//mdtf/wkdir/MDTF_CESM2_1950_2005/ENSO_MSE/COMPOSITE/model/netCDF/ELNINO/ts.nc>; No such file or directory\n\n\rwarning:_NclOpenFile: cannot open file </home//mdtf/wkdir/MDTF_CESM2_1950_2005/ENSO_MSE/COMPOSITE/model/netCDF/ts_clim.nc>; No such file or directory\n\n\rfatal:file (fin1) isn\'t defined\n\rfatal:["Execute.c":8637]:Execute: Error occurred at or near line 93 in file /home//mdtf/MDTF-diagnostics/diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl\n\n\r'
   Seasonal ENSO composites completed:  2021-12-22 11:24
   plots of ENSO seasonal composites finished  
   resulting plots are located in : /home//mdtf/wkdir/MDTF_CESM2_1950_2005/ENSO_MSE/COMPOSITE/model
   with prefix composite  + ELNINO/LANINA +  variable name 
NCL routine /home//mdtf/MDTF-diagnostics/diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl:
	b'warning:_NclOpenFile: cannot open file </home//mdtf/wkdir/MDTF_CESM2_1950_2005/ENSO_MSE/COMPOSITE/model/netCDF/CORR_pr.nc>; No such file or directory\n\n\rfatal:file (fin) isn\'t defined\n\rfatal:["Execute.c":8637]:Execute: Error occurred at or near line 57 in file /home//mdtf/MDTF-diagnostics/diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl\n\n\r'
 Seasonal  SST  correlations completed  2021-12-22 11:24
 plots of  seasonal correlations  finished  
 resulting plots are located in : /home//mdtf/wkdir/MDTF_CESM2_1950_2005/ENSO_MSE/COMPOSITE/model
 with prefix correlation + variable name 
COMPOSITE.py regression  1
NCL routine /home//mdtf/MDTF-diagnostics/diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl:
	b'warning:_NclOpenFile: cannot open file </home//mdtf/wkdir/MDTF_CESM2_1950_2005/ENSO_MSE/COMPOSITE/model/netCDF/REGRESS_pr.nc>; No such file or directory\n\n\rfatal:file (fin1) isn\'t defined\n\rfatal:["Execute.c":8637]:Execute: Error occurred at or near line 63 in file /home//mdtf/MDTF-diagnostics/diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl\n\n\r'
Seasonal SST  regressions completed  2021-12-22 11:24
 plots of seasonal regressions  finished  
 resulting plots are located in : /home//mdtf/wkdir/MDTF_CESM2_1950_2005/ENSO_MSE/COMPOSITE/model
 with prefix  regression  +  variable name 

Seasonal MODEL ENSO MSE Variance composites started  2021-12-22 11:25
NCL routine /home//mdtf/MDTF-diagnostics/diagnostics/ENSO_MSE/MSE_VAR/NCL_DATA/get_MSE_VAR.ncl:
	b'warning:_NclOpenFile: cannot open file </home//mdtf/wkdir/MDTF_CESM2_1950_2005/ENSO_MSE/MSE/model/netCDF/ELNINO/MSE_mse.nc>; No such file or directory\n\n\rwarning:_NclOpenFile: cannot open file </home//mdtf/wkdir/MDTF_CESM2_1950_2005/ENSO_MSE/MSE/model/netCDF/MSE_mse.nc>; No such file or directory\n\n\rfatal:file (fin1) isn\'t defined\n\rfatal:["Execute.c":8637]:Execute: Error occurred at or near line 54 in file /home//mdtf/MDTF-diagnostics/diagnostics/ENSO_MSE/MSE_VAR/NCL_DATA/get_MSE_VAR.ncl\n\n\r'
NCL routine /home//mdtf/MDTF-diagnostics/diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl:
	b'warning:_NclOpenFile: cannot open file </home//mdtf/wkdir/MDTF_CESM2_1950_2005/ENSO_MSE/MSE_VAR/model/netCDF/ELNINO/MSE_variance_C.nc>; No such file or directory\n\n\rfatal:file (fin1) isn\'t defined\n\rfatal:["Execute.c":8637]:Execute: Error occurred at or near line 53 in file /home//mdtf/MDTF-diagnostics/diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl\n\n\r'
NCL routine /home//mdtf/MDTF-diagnostics/diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_general.ncl:
	b'warning:_NclOpenFile: cannot open file </home//mdtf/wkdir/MDTF_CESM2_1950_2005/ENSO_MSE/MSE_VAR/model/netCDF/ELNINO/MSE_variance_general.nc>; No such file or directory\n\n\rfatal:file (fin1) isn\'t defined\n\rfatal:["Execute.c":8637]:Execute: Error occurred at or near line 58 in file /home//mdtf/MDTF-diagnostics/diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_general.ncl\n\n\r'
 ===================================================================
         Composite Module Finished  2021-12-22 11:25
 ===================================================================
<#kBZc:CESM2.ENSO_MSE> was terminated or exited abnormally.
Log for <#kBZc:CESM2.ENSO_MSE>:
    *** caught exception (#1):
    None: None
    
    INFO: 	Running ENSO_MSE.py for <#kBZc:CESM2.ENSO_MSE>.
    INFO: <#kBZc:CESM2.ENSO_MSE> was terminated or exited abnormally.
    INFO: Checking linked output files for <#kBZc:CESM2.ENSO_MSE>.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/composite_ELNINO_TS.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/composite_LANINA_TS.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/composite_ELNINO_PR.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/composite_LANINA_PR.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/composite_ELNINO_SHF.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/composite_LANINA_SHF.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/composite_ELNINO_LHF.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/composite_LANINA_LHF.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/composite_ELNINO_SW.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/composite_LANINA_SW.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/composite_ELNINO_LW.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/composite_LANINA_LW.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/composite_ELNINO_FRAD.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/composite_LANINA_FRAD.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/correlation_PR.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/correlation_SHF.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/correlation_LHF.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/correlation_SW.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/correlation_LW.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/regression_PR.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/regression_SHF.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/regression_LHF.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/regression_SW.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/regression_LW.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/ELNINO_mse.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/LANINA_mse.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/ELNINO_madv.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/LANINA_madv.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/ELNINO_mdiv.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/LANINA_mdiv.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/ELNINO_omse.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/LANINA_omse.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/ELNINO_tadv.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/LANINA_tadv.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/MSE_VAR_comp_ELNINO.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/MSE_VAR_comp_LANINA.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/MSE_VAR_comp_general_ELNINO.png'.
    ERROR: Missing '$WK_DIR/ENSO_MSE/model/MSE_VAR_comp_general_LANINA.png'.
    ERROR: Deactivated <#kBZc:CESM2.ENSO_MSE> due to MDTFFileNotFoundError("[Errno 2] No such file or directory: 'Missing 38 files.'").
jhafner2 commented
wrongkindofdoctor commented

@jhafner2 Thanks again for your help. It seems like my errors may have resulted from space/processor requirements, as I was able to run the ENSO_MSE POD to completion on my GFDL workstation and write the output to the shared drive.

I'm going to do a bit of cleanup in the POD code to, among other things, remove references to PDF files that are not stored in the repo, and fix the settings.jsonc files pod_env_vars section, which is in the wrong location. This wil fix the need to hard-code the values in the driver script. I'll tag you to review the PR when I submit it.