Update GAEA.env for Compatibility with Current Global Workflow

Question

Update GAEA.env for Compatibility with Current Global Workflow

Opened this issue 3 months ago · 0 comments

What new functionality do you need?

The current global workflow is encountering issues on Gaea due to an outdated GAEA.env file. Some options that are required for the workflow are missing or incorrect in GAEA.env, causing errors when sourcing the file. For example:

When running gdasfcst or gfsfcst, the GAEA.env is sourced with the ecfs option, which does not exist in the current version of GAEA.env.
Similar issues are observed when running upp, eupd, and other components.

What are the requirements for the new functionality?

This update should address the missing or outdated options and ensure compatibility with all components of the workflow.

Acceptance Criteria

Successful completion of global workflow on Gaea-C5

Suggest a solution (optional)

Please consider updating the GAEA.env file to incorporate the following environment settings:

#! /usr/bin/env bash

if [[ $# -ne 1 ]]; then

    echo "Must specify an input argument to set runtime environment variables!"
    exit 1

fi

step=$1

export launcher="srun -l --export=ALL"
export mpmd_opt="--multi-prog --output=mpmd.%j.%t.out"

export OMP_STACKSIZE=2048000
export NTHSTACK=1024000000

ulimit -s unlimited
ulimit -a

# Calculate common variables
# Check first if the dependent variables are set
if [[ -n "${ntasks:-}" && -n "${max_tasks_per_node:-}" && -n "${tasks_per_node:-}" ]]; then
    max_threads_per_task=$((max_tasks_per_node / tasks_per_node))
    NTHREADSmax=${threads_per_task:-${max_threads_per_task}}
    NTHREADS1=${threads_per_task:-1}
    [[ ${NTHREADSmax} -gt ${max_threads_per_task} ]] && NTHREADSmax=${max_threads_per_task}
    [[ ${NTHREADS1} -gt ${max_threads_per_task} ]] && NTHREADS1=${max_threads_per_task}
    # This may be useful when Gaea is fully ported, so ignore SC warning
    # shellcheck disable=SC2034
    APRUN="${launcher} -n ${ntasks}"
else
    echo "ERROR config.resources must be sourced before sourcing GAEA.env"
    exit 2
fi

if [[ "${step}" = "prep" ]] || [[ "${step}" = "prepbufr" ]]; then

    export POE="NO"
    export BACK="NO"
    export sys_tp="GAEA"
    export launcher_PREP="srun"

elif [[ "${step}" = "prepsnowobs" ]]; then

    export APRUN_CALCFIMS="${APRUN}"

elif [[ "${step}" = "prep_emissions" ]]; then

    export APRUN="${APRUN}"

elif [[ "${step}" = "waveinit" ]] || [[ "${step}" = "waveprep" ]] || [[ "${step}" = "wavepostsbs" ]] || [[ "${step}" = "wavepostbndpnt" ]] || [[ "${step}" = "wavepostbndpntbll" ]] || [[ "${step}" = "wavepostpnt" ]]; then

    export CFP_MP="YES"
    if [[ "${step}" = "waveprep" ]]; then export MP_PULSE=0 ; fi
    export wavempexec=${launcher}
    export wave_mpmd=${mpmd_opt}

elif [[ "${step}" = "atmanlvar" ]]; then

    export NTHREADS_ATMANLVAR=${NTHREADSmax}
    export APRUN_ATMANLVAR="${APRUN} --cpus-per-task=${NTHREADS_ATMANLVAR}"

elif [[ "${step}" = "atmensanlletkf" ]]; then

    export NTHREADS_ATMENSANLLETKF=${NTHREADSmax}
    export APRUN_ATMENSANLLETKF="${APRUN} --cpus-per-task=${NTHREADS_ATMENSANLLETKF}"

elif [[ "${step}" = "atmensanlfv3inc" ]]; then

    export NTHREADS_ATMENSANLFV3INC=${NTHREADSmax}
    export APRUN_ATMENSANLFV3INC="${APRUN} --cpus-per-task=${NTHREADS_ATMENSANLFV3INC}"

elif [[ "${step}" = "aeroanlrun" ]]; then

    export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

    export NTHREADS_AEROANL=${NTHREADSmax}
    export APRUN_AEROANL="${APRUN} --cpus-per-task=${NTHREADS_AEROANL}"

elif [[ "${step}" = "atmanlfv3inc" ]]; then

    export NTHREADS_ATMANLFV3INC=${NTHREADSmax}
    export APRUN_ATMANLFV3INC="${APRUN} --cpus-per-task=${NTHREADS_ATMANLFV3INC}"

elif [[ "${step}" = "prepobsaero" ]]; then

    export NTHREADS_PREPOBSAERO=${NTHREADS1}
    export APRUN_PREPOBSAERO="${APRUN} --cpus-per-task=${NTHREADS_PREPOBSAERO}"

elif [[ "${step}" = "snowanl" ]]; then

    export NTHREADS_SNOWANL=${NTHREADSmax}
    export APRUN_SNOWANL="${APRUN} --cpus-per-task=${NTHREADS_SNOWANL}"

    export APRUN_APPLY_INCR="${launcher} -n 6"

elif [[ "${step}" = "marinebmat" ]]; then

    export APRUNCFP="${launcher} -n \$ncmd --multi-prog"
    export APRUN_MARINEBMAT="${APRUN}"

elif [[ "${step}" = "marinebmat" ]]; then

    export APRUNCFP="${launcher} -n \$ncmd --multi-prog"
    export APRUN_MARINEBMAT="${APRUN}"

elif [[ "${step}" = "ocnanalrun" ]]; then

    export APRUNCFP="${launcher} -n \$ncmd --multi-prog"

    export APRUN_OCNANAL="${APRUN}"

elif [[ "${step}" = "ocnanalchkpt" ]]; then

    export APRUNCFP="${launcher} -n \$ncmd --multi-prog"

    export APRUN_OCNANAL="${APRUN}"

elif [[ "${step}" = "ocnanalecen" ]]; then

    export NTHREADS_OCNANALECEN=${NTHREADSmax}
    export APRUN_OCNANALECEN="${APRUN} --cpus-per-task=${NTHREADS_OCNANALECEN}"

elif [[ "${step}" = "marineanalletkf" ]]; then

    export NTHREADS_MARINEANALLETKF=${NTHREADSmax}
    export APRUN_MARINEANALLETKF="${APRUN} --cpus-per-task=${NTHREADS_MARINEANALLETKF}"

elif [[ "${step}" = "anal" ]] || [[ "${step}" = "analcalc" ]]; then

    export MKL_NUM_THREADS=4
    export MKL_CBWR=AUTO

    export CFP_MP=${CFP_MP:-"YES"}
    export USE_CFP=${USE_CFP:-"YES"}
    export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

    export NTHREADS_GSI=${NTHREADSmax}
    export APRUN_GSI="${APRUN} --cpus-per-task=${NTHREADS_GSI}"

    export NTHREADS_CALCINC=${threads_per_task_calcinc:-1}
    [[ ${NTHREADS_CALCINC} -gt ${max_threads_per_task} ]] && export NTHREADS_CALCINC=${max_threads_per_task}
    export APRUN_CALCINC="${launcher} \$ncmd --cpus-per-task=${NTHREADS_CALCINC}"

    export NTHREADS_CYCLE=${threads_per_task_cycle:-12}
    [[ ${NTHREADS_CYCLE} -gt ${max_tasks_per_node} ]] && export NTHREADS_CYCLE=${max_tasks_per_node}
    ntasks_cycle=${ntiles:-6}
    export APRUN_CYCLE="${launcher} -n ${ntasks_cycle} --cpus-per-task=${NTHREADS_CYCLE}"

    export NTHREADS_GAUSFCANL=1
    ntasks_gausfcanl=${ntasks_gausfcanl:-1}
    export APRUN_GAUSFCANL="${launcher} -n ${ntasks_gausfcanl} --cpus-per-task=${NTHREADS_GAUSFCANL}"

elif [[ "${step}" = "sfcanl" ]]; then

    export NTHREADS_CYCLE=${threads_per_task:-14}
    export APRUN_CYCLE="${APRUN} --cpus-per-task=${NTHREADS_CYCLE}"

elif [[ "${step}" = "eobs" ]]; then

    export MKL_NUM_THREADS=4
    export MKL_CBWR=AUTO

    export NTHREADS_GSI=${NTHREADSmax}
    export APRUN_GSI="${APRUN} --cpus-per-task=${NTHREADS_GSI}"

    export CFP_MP=${CFP_MP:-"YES"}
    export USE_CFP=${USE_CFP:-"YES"}
    export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

elif [[ "${step}" = "eupd" ]]; then

    export NTHREADS_ENKF=${NTHREADSmax}
    export APRUN_ENKF="${launcher} -n ${ntasks_enkf:-${ntasks}} --cpus-per-task=${NTHREADS_ENKF}"

    export CFP_MP=${CFP_MP:-"YES"}
    export USE_CFP=${USE_CFP:-"YES"}
    export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

elif [[ "${step}" = "fcst" ]] || [[ "${step}" = "efcs" ]]; then

    (( nnodes = (ntasks+tasks_per_node-1)/tasks_per_node ))
    (( ufs_ntasks = nnodes*tasks_per_node ))
    # With ESMF threading, the model wants to use the full node
    export APRUN_UFS="${launcher} -n ${ufs_ntasks}"
    unset nnodes ufs_ntasks

elif [[ "${step}" = "upp" ]]; then

    export NTHREADS_UPP=${NTHREADS1}
    export APRUN_UPP="${APRUN} --cpus-per-task=${NTHREADS_UPP}"

elif [[ "${step}" = "atmos_products" ]]; then

    export USE_CFP="YES"  # Use MPMD for downstream product generation on Hera

elif [[ "${step}" = "oceanice_products" ]]; then

    export NTHREADS_OCNICEPOST=${NTHREADS1}
    export APRUN_OCNICEPOST="${launcher} -n 1 --cpus-per-task=${NTHREADS_OCNICEPOST}"

elif [[ "${step}" = "ecen" ]]; then

    export NTHREADS_ECEN=${NTHREADSmax}
    export APRUN_ECEN="${APRUN} --cpus-per-task=${NTHREADS_ECEN}"

    export NTHREADS_CHGRES=${threads_per_task_chgres:-12}
    [[ ${NTHREADS_CHGRES} -gt ${max_tasks_per_node} ]] && export NTHREADS_CHGRES=${max_tasks_per_node}
    export APRUN_CHGRES="time"

    export NTHREADS_CALCINC=${threads_per_task_calcinc:-1}
    [[ ${NTHREADS_CALCINC} -gt ${max_threads_per_task} ]] && export NTHREADS_CALCINC=${max_threads_per_task}
    export APRUN_CALCINC="${APRUN} --cpus-per-task=${NTHREADS_CALCINC}"

elif [[ "${step}" = "esfc" ]]; then

    export NTHREADS_ESFC=${threads_per_task_esfc:-${max_threads_per_task}}
    export APRUN_ESFC="${APRUN} --cpus-per-task=${NTHREADS_ESFC}"

    export NTHREADS_CYCLE=${threads_per_task_cycle:-14}
    [[ ${NTHREADS_CYCLE} -gt ${max_tasks_per_node} ]] && export NTHREADS_CYCLE=${max_tasks_per_node}
    export APRUN_CYCLE="${APRUN} --cpus-per-task=${NTHREADS_CYCLE}"

elif [[ "${step}" = "epos" ]]; then

    export NTHREADS_EPOS=${NTHREADSmax}
    export APRUN_EPOS="${APRUN} --cpus-per-task=${NTHREADS_EPOS}"

elif [[ "${step}" = "postsnd" ]]; then

    export CFP_MP="YES"

    export NTHREADS_POSTSND=${NTHREADS1}
    export APRUN_POSTSND="${APRUN} --cpus-per-task=${NTHREADS_POSTSND}"

    export NTHREADS_POSTSNDCFP=${threads_per_task_postsndcfp:-1}
    [[ ${NTHREADS_POSTSNDCFP} -gt ${max_threads_per_task} ]] && export NTHREADS_POSTSNDCFP=${max_threads_per_task}
    export APRUN_POSTSNDCFP="${launcher} -n ${ntasks_postsndcfp} ${mpmd_opt}"

elif [[ "${step}" = "awips" ]]; then

    export NTHREADS_AWIPS=${NTHREADS1}
    export APRUN_AWIPSCFP="${APRUN} ${mpmd_opt}"

elif [[ "${step}" = "gempak" ]]; then

    export CFP_MP="YES"

    export NTHREADS_GEMPAK=${NTHREADS1}
    [[ ${NTHREADS_GEMPAK} -gt ${max_threads_per_task} ]] && export NTHREADS_GEMPAK=${max_threads_per_task}

elif [[ "${step}" = "fit2obs" ]]; then

    export NTHREADS_FIT2OBS=${NTHREADS1}
    export MPIRUN="${APRUN} --cpus-per-task=${NTHREADS_FIT2OBS}"

fi