NWPC Data Client
A CLI client for operation system files in CEMC/CMA.
Features
- Find file path for operation system data in CMA HPC.
- 🚧 Find file path in external storage nodes from CMA HPC.
- 🚧 Download data file through a data service.
Installing
Download the latest release and build source code.
Use Makefile
to build project on Linux and
nwpc_data_client
command will be installed in bin
directory.
Getting Started
nwpc_data_client
has several sub-commands.
local
nwpc_data_client local
command finds local files on CMA HPC.
nwpc_data_client local --data-type some/data/type \
--start-time start_time \
--forecast-time forecast_time
data-type
is some relative path under config directory. Such as
grapes_gfs_gda/grib2/modelvar
gmf_graeps_gfs/bin/modelvar
start_time
is YYYYMMDDHH
and forecast_time
is FFFh
.
For example, use the command below to find GMF GRAPES GFS GRIB2 data of 24 forecast hour in start hour 00 on 2018/09/03.
$nwpc_data_client local --data-type=grapes_gfs_gmf/grib2/orig --start-time 2018090300 --forecast-time 24h
/g2/nwp_pd/NWP_PST_DATA/GMF_GRAPES_GFS_V2.2_POST/togrib2/output_togrib2/2018090300/gmf.gra.2018090300024.grb2
To list all data types available in some configure directory, run following command
nwpc_data_client local --show-types
Results may be like:
grapes_gfs_gda/bin/modelvar
grapes_gfs_gda/bin/postvar
grapes_gfs_gda/grib2/modelvar
grapes_gfs_gda/grib2/orig
grapes_gfs_gmf/bin/modelvar
grapes_gfs_gmf/bin/postvar
grapes_gfs_gmf/grib2/modelvar
grapes_gfs_gmf/grib2/ne
grapes_gfs_gmf/grib2/orig
Use --config-dir
to set a custom config file directory.
hpc
nwpc_data_client hpc
command find files on HPC-PI or external storage nodes from HPC-PI.
Data files could be on HPC's local storage nodes (eg. /g2) or external storage nodes which are mount to special HPC login nodes.
nwpc_data_client hpc --config-dir=config_dir --data-type=some/data/type \
--start-time start_time \
--forecast-time forecast_time
hpc
support all options of local
, and has more options to access external storage nodes using ssh protocol.
--storage-user
: user to login, default is environment variableUSER
--storage-host
: host to login, default is10.40.140.44:22
--private-key
: private key file path, default is$HOME/.ssh/id_rsa
--host-key
: host key file path, default is$HOME/.ssh/known_hosts
Currently only no password private key is supported, and user should test to access remote host manually before using this command.
paths
field in hpc
's config file has two types:
local
for local filesstorage
for files on external storage.
For example, use the command below to find GDA GRAPES GFS modelvar data of 000 forecast hour in start hour 00 on 2019/05/20.
$nwpc_data_client hpc --data-type=grapes_gfs_gda/bin/modelvar --start-time 2019050200 --forecast-time 0h
storage
/sstorage1/COMMONDATA/OPER/nwp/GRAPES_GFS/GDA_GRAPES_GFS/Fcst-9h/2019050121/modelvar2019050121_000
The command return two lines:
- first line is path type:
local
orstorage
. - second line is file path.
If no file is found, both lines will be value of default
field in config file.
paths
section of a config file may like this:
paths:
- type: local
level: archive
path: /g2/nwp/OPER_ARCH_TEST/nwp/GRAPES_GFS/GDA_GRAPES_GFS/Fcst-9h/{.Year4DV}{.Month4DV}{.Day4DV}{.Hour4DV}
- type: storage
level: storage
path: /sstorage1/COMMONDATA/OPER/nwp/GRAPES_GFS/GDA_GRAPES_GFS/Fcst-9h/{.Year4DV}{.Month4DV}{.Day4DV}{.Hour4DV}
NWPC Data Service
See README.md under data_service.
Test
Run make test
to run all tests. bats is required.
License
Copyright © 2019-2023 Perilla Roc at cemc-oper.
nwpc-data-client
is licensed under The MIT License.