wish list for the model tests …
Closed this issue · 5 comments
From @0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q:
- It would be really useful to have an overview table (say a .csv file) with the run name (e.g. SSP2EU-AMT-NDC), the date, and the RunStatus (maybe for good measure also the commit hash) to figure out since when certain runs fail in order to narrow down which changes might have caused this. Current example would be SSP2EU-AMT-NDC failing since some time. -> #59
- (Bit of a pet peeve) The test run output directories are partly truncated in the README.md (e.g. SSP2EU-AMT-calibrate_2022-11-26_00 instead of SSP2EU-AMT-calibrate_2022-11-26_00.07.54) which is a nuisance when looking up results. -> #22
While the rs2 features are certainly useful and have their applications, they do not address the heart of this issue.
The intention was specifically to have a table that can be read programmatically, not just manually, to filter for specific run types when looking into problems.
The output of rs2 is anything but machine-readable:
- it looks like somebody discovered ANSI codes for the first time and decided to use them all
- fields are not delimited by an explicit character, so field widths would have to be laboriously parsed
- the run time given in either minutes, hours, or days, is a pain to work with
- and for good measure it contains some useless ad blurb
$ rs2 -t SSP2EU-Base | cat -A | sed 's/\$$//'
Loading modelstats... Did you know? Show results from specific folders with: rs2 folder1,folder2
Results from /p/projects/remind/modeltests/remind/output/
# Color code: ^[[33mpending^[[39m/^[[33mstartup^[[39m, ^[[36mrunning^[[39m, ^[[4m^[[32mconverged^[[39m^[[24m, ^[[34mconverged (had INFES)^[[39m, ^[[32mfinished^[[39m, ^[[31merror^[[39m.
^[[4mFolder Runtime inSlurm RunType RunStatus Iter Conv modelstat Mif AppResults^[[24m
^[[4m^[[32mSSP2EU-Base-AMT_2023-08-04_22.08.55 3.2 hours no nash Normal completion 31/100 converged 2: Locally Optimal yes yes
^[[39m^[[24m^[[4m^[[32mSSP2EU-Base-AMT_2023-07-28_22.13.59 3 hours no nash Normal completion 31/100 converged 2: Locally Optimal yes yes
^[[39m^[[24m^[[4m^[[32mSSP2EU-Base-AMT_2023-07-21_22.07.42 3.2 hours no nash Normal completion 31/100 converged 2: Locally Optimal yes yes
^[[39m^[[24m^[[4m^[[32mSSP2EU-Base-AMT_2023-07-14_22.07.41 3.1 hours no nash Normal completion 31/100 converged 2: Locally Optimal yes yes
^[[39m^[[24m^[[4m^[[32mSSP2EU-Base-AMT_2023-07-10_13.07.37 3.4 hours no nash Normal completion 41/100 converged 2: Locally Optimal yes yes
^[[39m^[[24m^[[4m^[[32mSSP2EU-Base-AMT_2023-07-07_22.07.59 3.2 hours no nash Normal completion 35/100 converged 2: Locally Optimal yes yes
^[[39m^[[24m^[[31mSSP2EU-Base-AMT_2023-06-30_22.07.45 NA no nash full.log missing NA NA NA no no
^[[39m^[[4m^[[32mSSP2EU-Base-AMT_2023-06-23_22.16.19 3.3 hours no nash Normal completion 41/100 converged 2: Locally Optimal yes yes
^[[39m^[[24m^[[4m^[[32mSSP2EU-Base-AMT_2023-06-16_22.10.29 3.8 hours no nash Normal completion 43/100 converged 2: Locally Optimal yes yes
^[[39m^[[24m^[[4m^[[32mSSP2EU-Base-AMT_2023-06-09_22.08.24 3.6 hours no nash Normal completion 45/100 converged 2: Locally Optimal yes yes
^[[39m^[[24m^[[4m^[[32mSSP2EU-Base-AMT_2023-06-02_22.07.57 3.2 hours no nash Normal completion 43/100 converged 2: Locally Optimal yes yes
^[[39m^[[24m^[[4m^[[32mSSP2EU-Base-AMT_2023-05-23_16.06.55 3.7 hours no nash Normal completion 45/100 converged 2: Locally Optimal yes yes
^[[39m^[[24m^[[31mSSP2EU-Base-AMT_2023-05-23_14.07.00 NA no nash full.log missing NA NA NA no no
^[[39m^[[4m^[[32mSSP2EU-Base-AMT_2023-05-16_14.07.39 3.1 hours no nash Normal completion 38/100 converged 2: Locally Optimal yes yes
^[[39m^[[24m^[[4m^[[32mSSP2EU-Base-AMT_2023-05-06_00.08.25 3.1 hours no nash Normal completion 40/100 converged 2: Locally Optimal yes yes
^[[39m^[[24m^[[4m^[[32mSSP2EU-Base-AMT_2023-04-22_00.07.52 3.1 hours no nash Normal completion 38/100 converged 2: Locally Optimal yes yes
^[[39m^[[24m^[[34mSSP2EU-Base-AMT_2023-03-30_16.07.44 3.7 hours no nash Normal completion 50/100 converged (had INFES) 2: Locally Optimal yes yes
^[[39m^[[31mSSP2EU-Base-AMT_2023-03-21_00.07.38 6.8 hours no nash Normal completion 100/100 not_converged 2: Locally Optimal yes yes
^[[39m^[[31mAMT-SSP2EU-Base_2023-03-16_15.23.45 NA no nash Run interrupted 2/100 227222222222 2: Locally Optimal no no
^[[39m^[[31mAMT-SSP2EU-Base_2023-03-16_15.01.59 NA no nash Run interrupted 1/100 NA NA no no
The commit hash is missing, and rs2 is slow – getting an entire listing (which is what one would do when having no clear idea where to start looking and is what a static .csv file would provide) takes 20 minutes.
It is OK to decide to not implement something. But do not pretend that something was addressed when it was not.
P.S.: Just adding more space to the static columns [↑] works for now, but might as well break next week if somebody comes up with new scenario names. Why not use of-the-shelve solutions?
> format(knitr::kable(head(mtcars), 'simple'))
[1] " mpg cyl disp hp drat wt qsec vs am gear carb"
[2] "------------------ ----- ---- ----- ---- ----- ------ ------ --- --- ----- -----"
[3] "Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4"
[4] "Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4"
[5] "Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1"
[6] "Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1"
[7] "Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2"
[8] "Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1"
Not sure if things have changed in the meantime, and what exactly is looked for here - but just in case people are not aware of it: running modelstats::getRunStatus() in R is supposed to return a data frame with the run information, which can be further processed. As described also here: https://pik-piam.r-universe.dev/articles/modelstats/rs2.html
However it is done, ideally it should be done once when the runs are processed and then stored, not ad-hoc every time the information is of interest.
@0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q: Are you aware of:
data <- readRDS("/p/projects/remind/modeltests/remind/output/gRS.rds")
@0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q: Are you aware of:
data <- readRDS("/p/projects/remind/modeltests/remind/output/gRS.rds")
I was not.