Measure notebook runtimes
Opened this issue · 2 comments
Some of the Jupyter-Notebooks using the new Python Interface (see #22 ) take multiple minutes for a full run-through, which causes the test pipeline to run quite long. This table aims to summarize the current run times (using the notebooks in #22 ) for evaluation and adaption of simulation run times and pipeline tests. The full notebook runtime is measured in a single run of pytest --durations=0 -v
, the simulation run times are measured using the iPython %%timeit
cell magic with the time calculated as the average over seven runs. Simulation run times contain the full setup and run of the dynamic simulation without PowerFlow or later data evaluation / plotting. Please note that this is not meant as a complete, accurate benchmark but rather as an overview over which notebooks should be tweaked when tested in the CI-pipeline.
Overall
61 notebooks in Circuits, Components or Grids
7 notebooks skipped in pytest
(11,5%)
25 notebooks with asserts (41,0%)
Circuit-Examples
Full pytest
-run in 760.53s (0:12:40)
26 passed, 2 skipped (92.9% tested)
10 of 28 notebooks with asserts (35,7%)
Notebook | Pytest runtime | Simulation runtime(s) | Asserts | Remarks |
---|---|---|---|---|
Compare_EMT_DP_Slack_PiLine_PQLoad | 4.14s | 1.73s (EMT), 419 ms (DP) | DP vs EMT | |
Compare_EMT_SynGenDQ7odTrapez_ DP_SynGenTrStab_SMIB_Fault | Skipped | 48.6s (EMT), 4.25s (DP) | - | Notebook contains no error, so test could be enabled |
CS_R2CL print_attributes | 2.79s | 12.5 µs (EMT), 12.5 µs (DP) | - | Notebook exists to show off attribute print functions, so comparisons with SL got stripped |
CS_R2CL | 3.36s | 11.8 µs (EMT), 11.4 µs (DP) | EMT vs SL, DP vs SL | |
DP_Slack_PiLine_PQLoad_with_PF_Init | 2.49s | 71.6 µs | - | Plots currently plot the wrong time interval |
DP_Slack_PiLine_VSI_Ramp | 26.20s | - | - | Notebook does not use dpsimpy, plot labels are in german |
DP_Slack_PiLine_VSI_with_PF_Init | 47.27s | 45.8s | - | wrong time interval plotted, plot labels in german |
DP_SP_SynGenTrStab_3Bus_Fault | 32.01s | 8.94s (DP), 8.52s (SP) | - | |
DP_SP_SynGenTrStab_3Bus_SteadyState | 12.71s | 2.28s (DP), 2.11s (SP) | - | |
DP_SP_SynGenTrStab_SMIB_Fault | 17.57s | 4.06s (DP), 4.11s (SP) | - | |
DP_SP_SynGenTrStab_SMIB_SteadyState | 12.29s | 1.70s (DP), 1.45s (SP) | - | |
EMT_DP_Slack_PiLine_VSI_ Control_OnOff | 34.25s | 10.4s (EMT, control on), 11.2s (EMT, control off), 5.04s (DP, control on), 4.71s (DP, control off) | - | |
EMT_DP_SP_Slack_PiLine_VSI | 107.11s | 11.1s (EMT), 48s (DP), 47.6s (SP) | - | |
EMT_DP_VS_Init | 3.12s | 49.2ms (DP, set params), 88ms (EMT, set params), 45.1ms (DP, set attribute), 74.6ms (EMT, set attribute), 40.7ms (DP, set from node), 83.8ms (EMT, set from node) | EMT vs DP | DP vs EMT assert on set attribute commented out |
EMT_DP_VS_RLC | 19.35s | 1.62s (EMT, VS+R), 518ms (DP, VS+R), 480ms (SP, VS+R), 2.82s (EMT, VS+RL), 785ms (DP, VS+RL), 856ms (SP, VS+RL), 2.79s (EMT, VS+RC), 758ms (DP, VS+RC), 865ms (SP, VS+RC) | EMT vs DP vs SP | |
EMT_Slack_PiLine_PQLoad_with_PF_Init | 2.77s | 493ms | - | Plots the wrong time interval |
EMT_Slack_PiLine_VSI_with_PF_Init_Params | 69.97s | 27.9s (Kp variation), 43.7s (Ki variation), 4.33s (final choice) | - | |
EMT_Slack_PiLine_VSI_with_PF_Init | 12.65s | 11.3s | - | |
EMT_SynGenDQ7odTrapez_ OperationalParams_SMIB_Fault_JsonSyngenParams | 93.01s | 51.5s (no JSON-config), 50.4s (with JSON-config) | JSON-config vs No JSON-config | |
EMT_SynGenDQ7odTrapez_ OperationalParams_SMIB_Fault_ParamsModification | 94.22s | 50.8s (first parameter), 52.8s (second parameters) | - | |
EMT_SynGenDQ7odTrapez_ OperationalParams_SMIB_Fault | 93.72s | 52.9s (fundamental params), 54.5s (operational params) | Fundamental vs operational params | |
EMT_SynGenDQ7odTrapez_SMIB_Fault | 50.24s | 53.6s | - | |
RL_SourceStep_ParameterStudy | Skipped | - | - | Notebook does not use dpsimpy |
SP_Slack_PiLine_VSI_with_PF_Init | 3.69s | 2.17s | - | |
VS_CS_R4 | 3.39s | 29.8ms (EMT), 52.4ms (DP) | EMT vs SL, DP vs SL | |
VS_R2L3 | 3.47s | 33.7ms (EMT), 63.1ms (DP) | EMT vs SL, DP vs SL | |
VS_RC1 | 4.22s | 19.9ms (EMT), 37.5ms (DP) | EMT vs SL, DP vs SL | |
VS_RL1 | 3.77s | 21.2ms (EMT), 33.9ms (DP) | EMT vs SL, DP vs SL |
Component examples
Full pytest
-run in 133.57s (0:02:13)
10 passed, 1 skipped (92.9% tested)
7 of 11 notebooks with asserts (63,6%)
Notebook | Pytest runtime | Simulation runtime(s) | Asserts | Remarks |
---|---|---|---|---|
Diakoptics | 2.91s | 44ms (VS_CS_R4), 114ms (VS_CS_R4_Diakoptics), 56.3ms (VS_R2L3), 127ms (VS_R2L3_Diakoptics) | - | |
Inverter_Grid_Test | 49.46s | 46s | DP vs ML | DP vs SL has no asserts, comparison is only plotted |
Line | 19.37s | 904ms (SP Elements), 1.1s (SP Component), 951ms (DP Elements), 1.04s (DP Component), 2.13s (DP Diakoptics), 4.58s (EMT Elements), 6.14s (EMT Component), 79.9ms (Decoupling Line Elements), 109ms (Decoupling Line Component), 68.4ms (Decoupling Line EMT) | Elements vs Component for DP, SP, EMT; DP vs EMT component | |
Slack | 11.18s | 433ms (SP Elements), 774ms (SP Component), 481ms (DP Elements), 850ms (DP Component), 938ms (EMT Elements), 1.27s (EMT Component) | Elements vs Component for DP, SP, EMT; SP vs DP component, DP vs EMT component | |
SynGen_trStab_logger_test | 1.40s | - | - | This example exists to show errors in logger behaviour when multiple simulations interact with each other so timing it does not make sense here |
SynGen_trStab | 2.15s | 8.7ms (SteadyState), 17.8ms (LoadStep) | wr_gen(t=0) vs wr_gen(t=0.1) | |
SynGenDq7od_ParameterStudy | Skipped | 1.81s (DP ODE), 1.57s (EMT ODE), 1.34s (DP Trapez), 1.07s (EMT Trapez) | - | Tries to evaluate files which are not actually generated by the simulation |
SynGenDq7od_SteadyState _DP_EMT | 7.67s | 1.81s (DP ODE), 1.44s (EMT ODE), 1.29s (DP Trapez), 1.07s (EMT Trapez) | - | SL results are loaded and compared to in plots, but there are no asserts |
SynGenDq7od_ThreePhFault_DP_EMT | 8.15s | 1.84s (DP ODE), 1.55s (EMT ODE), 1.25s (DP Trapez), 1.09s (EMT Trapez) | EMT vs SL | |
SynGenDq7od | 6.77s | 22.6ms (DP ODE SteadyState), 1.82s (DP ODE Fault), 1.31s (DP Trapez Fault) | DP ODE Fault vs SL, DP Trapez Fault vs SL | |
Trafo | 12.79s | 736ms (SP Elements), 950ms (SP Component), 627ms (DP Elements), 851ms (DP Component), 2.03s (EMT Elements), 3.92s (EMT Components) | Elements vs Component for DP, SP, EMT; SP vs DP component, DP vs EMT component |
Grid examples
Full pytest
-run in 223.14s (0:03:43)
18 passed, 4 skipped (81,8% tested)
8 of 22 notebooks with asserts (36,4%)
Notebook | Pytest runtime | Simulation runtime(s) | Asserts | Remarks |
---|---|---|---|---|
case9 | 1.80s | 14.6ms | - | |
case14 | 1.58s | 26ms | - | |
case145 | 6.60s | 895ms | - | |
case300 | 8.87s | 2.0s | - | |
CIGRE_MV_pf-interactive-dpsimpy | Skipped | - | - | There is no full simulation, only one step |
CIGRE_MV_powerflow_profiles-dpsimpy | Skipped | 1.81s | - | Notebook contains no error, so test could be enabled |
CIGRE_MV_powerflow-dpsimpy | Skipped | 24.4ms | DPsim vs NEPLAN | Notebook contains no error, so test could be enabled |
CIGRE_MV_powerflow | 2.54s | - | DPsim vs NEPLAN | Non-dpsimpy version of CIGRE_MV_powerflow-dpsimpy, probably obsolete |
cigre-mv-pf-profiles-shmem | Skipped | - | - | Does not run at all |
DP_CIGRE_MV_withDG_withLoadStep | 19.76s | 14.3s | - | |
DP_CIGRE_MV_withDG | 17.84s | 14.1s | - | |
DP_CIGRE_MV_withoutDG | 6.06s | 2.13s | - | |
EMT_CIGRE_MV_withDG_ withLoadStep | 39.09s | 36.0s | - | |
EMT_CIGRE_MV_withDG | 38.16s | 35.0s | - | |
EMT_CIGRE_MV_withoutDG | 10.26s | 3.92s | - | |
IEEE_LV_powerflow | 2.91s | 539ms | DPsim vs NEPLAN | |
PF_CIGRE_MV_withDG | 2.52s | 105ms | - | |
SP_SMIB_SynGenTrStab_KundurExample1 | 19.56s | - | DPsim vs PSAT | |
SP_WSCC_9-bus_dyn_switch | 12.09s | 7.38s | DPsim vs PSAT | |
WSCC_9-bus_dyn_switch | 15.98s | 9.22s | DPsim vs SL | |
WSCC_9-bus_dyn | 13.18s | 9.95s | Last value vs first value for all logged phasors | |
WSCC_9-bus | 3.98s | 998ms | Last value vs first value for all logged phasors |
Grid examples
Notebook | Pytest runtime | Simulation runtime(s) | Asserts | Remarks |
---|---|---|---|---|
DP_Slack_PiLine_VSI_Ramp | - | - | DP vs EMT ref | Notebook does not use dpsimpy |
EMT_DP_SP_Slack_PiLine_PQLoad_FrequencyRamp_CosineFM | - | - | DP vs EMT, DP vs SP | Notebook does not use dpsimpy |
Next step is to remove unnecessary examples (especially C++) and reduce the runtime of the remaining ones.