noaa-ocs-modeling/EnsemblePerturbation

Replicate NHC-collaboration, ensemble perturbation test case for hurricane florence

Closed this issue · 14 comments

FariborzDaneshvar-NOAA commented

Directory on PW clustre: /home/Fariborz.Daneshvar/nhc_colab_test/florence_2018

Notebooks:
setup_n_run.ipynb: to create ensembles of perturbed tracks (based on OFCL track), generate SCHISM inputs, and run them.
post_ensemble.ipynb: for post processing and plotting

Follow the instructions from this issue

saeed-moghimi-noaa commented

@FariborzDaneshvar-NOAA Please update the status

FariborzDaneshvar-NOAA commented

The following issues were identified while running the setup_n_run.ipyn notebook:

  1. Restarting the (NHC COLABORATION) cluster will revoke permission of the other users to access/create files (even in their home directories).

  2. Absolute to relative path conversion issue during the generation of SCHISM configurations:
    The notebook worked fine with the default paths (in /lustre/hurricanes/...) for generating SCHISM files, but it failed with a path to home directory (i.e. /home/Fariborz.Daneshvar/...) with a FileNotFoundError for accessing hgrid.grd file as executing the generate_schism_configuration() function. @SorooshMani-NOAA created a ticket for this issue. Needs more investigation (in progress...)

  3. Missing maxelev_###### files for some ensemble runs:
    Six out of 15 runs (for an ensemble of 15 perturbed tracks) are missing maxelev_###### files in their outputs/ directories. Based on the check_completion command, five out of these six runs (including original, 6, 7, 9, and 13) finished with 99.98% (instead of 100%) completion, while run#8 with 100% completion was also missing these outputs. So it might/not be related to the status of runs! As a result, the combine_results command did not generate maxele.63.nc file, so could not run the post_ensemble.ipynb notebook. Needs more investigation (in progress...)

FariborzDaneshvar-NOAA commented

The alternate approach was using the ondemand-storm-workflow singularity image (instead of notebooks).
I ran it for 50 ensembles on NHC_COLAB_2 cluster and unlike the previous approach (with the notebooks), all runs completed successfully.
Run_directory: /lustre/hurricanes/florence_2018_Fariborz_v1_50_ensembles

Review of outputs revealed an issue with the perturbed tracks (see issue #74 for more detail). I will re-run the model again once Soroosh resolves this issue.

I got an error for the run with BEST track (see issue #19)

saeed-moghimi-noaa commented

Great to hear that. thanks for the update! :-)

FariborzDaneshvar-NOAA commented

Update:
the perturb track issue (mentioned above) was resolved, and I ran the model with the updated singularity image:
directory: /lustre/hurricanes/florence_2018_Fariborz_v2_10_ensembles

However, review of analysis outputs revealed an inconsistency between model and surrogate validation plots (see an example below).

Image

To ensure that it's not due to limited number of ensembles (10), I re-ran it with 39 ensembles:
New_directory: /lustre/hurricanes/florence_2018_Fariborz_v3_39_ensembles

But still got the same patterns! Will discuss it with Soroosh and William during the Ensemble Analysis meeting.

FariborzDaneshvar-NOAA commented

Update:
William pointed to the _analyze function in analyze_ensemble.py, noting that node_status_mask should be None instead of always_wet !

Soroosh updated the singularity image, and I re-ran only the analyze part of the workflow.sh, but was still getting the same results. (here is an example of new analysis with k=1, p=1, and n=0.025).

Image

It should be noted that in all three test runs, there was only one eigenvalue!

Image

Review of maxele.63.nc (./setup/ensemble.dir/analyze/maxele.63.nc) does not show high elevations associated with the storm track, suggesting that storm track was not included in the SCHISM runs! (see the screenshot below).

Image

Will discuss it with the rest of the team during the Thursday tag-up meeting.

SorooshMani-NOAA commented

@FariborzDaneshvar-NOAA as we discussed, I believe this should have something to do with the track input. Let's talk with @pvelissariou1 in the next meeting. My guess is that the "faked" BEST track has some missing information.

This also could be something related to how SCHISM is compiled in the container (less likely).

Actually let's try running an actual best track as well before we ask Takis.

pvelissariou1 commented
FariborzDaneshvar-NOAA commented

Turning ON and OFF the --past-forecast flag!

Directory of a run without the flag: /lustre/hurricanes/florence_2018_Fariborz_v4_15_ensembles

@SorooshMani-NOAA, similar to the previous runs, no sign of storm track is visible in the scatter plot of maxele.63.nc:

image

However,
Unlike the previous runs (with the --past-forecast flag), more than one eigenvalue was found (i.e. for linear_k1_p1_n0.025):

KL_eigenvalues_025

Similar plot exist for linear_k1_p1_n0.025\, but the linear_k1_p1_n0.1\ directory is empty!

SorooshMani-NOAA commented

@FariborzDaneshvar-NOAA so in the zoomed in landfall region plot it's still missing the floodplain? This is strange. Maybe as Takis said the issue is with how I compiled SCHISM then. Let's talk about it and see what debugging approach to take.

FariborzDaneshvar-NOAA commented

@SorooshMani-NOAA , yes, here is an example:
path: /lustre/hurricanes/florence_2018_Fariborz_v4_15_ensembles/setup/ensemble.dir/analyze/linear_k1_p1_n0.025/validation_vortex_4_variable_korobov_3.png

validation_vortex_4_variable_korobov_3

SorooshMani-NOAA commented

@FariborzDaneshvar-NOAA using Thalassa on PW, I see a difference if I limit the max elevation colorbar so that I can see more contrast:

OFCL track (faked as BEST) BEST track
image image

So it seems that the issue might actually be the missing data on the OFCL track that we fake as BEST. I'd like to note here that we don't set the PaHM model in SCHISMS param.nml here and it's using the default model (i.e. GAHM)

I also see the hurricane signal in the BEST track run (original):
image

The fact that we have multiple eigenvalues is also a good sign, however I don't understand why we don't get the correct exceedance plots here

FariborzDaneshvar-NOAA commented

I was able to run the ondemand-storm-workflow and partially replicate Willam's paper results. However, there were some issues in the plotting routines (listed below) that will be investigated in the future.