Consider improvements to reducing testing time
berquist opened this issue · 2 comments
berquist commented
See openjournals/joss-reviews#6286 (comment):
The unit tests take a while to run (8 minutes in serial, 1.5 minutes with
-n 8 --dist workstealon a 16 core machine). The provided suggestion to use xdist is a good one. If you find that developers complain still, consider using pytest markers to group tests by time.
Moving the discussion here.
Here are the results for running python -m pytest --durations=0. It looks like the ~generally slowest tests are those having to do with featurization, but many are slow in setup. Based upon the common fixture names in test_write_description and test_text, I'm certain the slowness is in performing reanalysis each time the fixture is used. I think setting these fixtures to session scope will help without having to implement a cache.
50.16s setup tests/cohp/test_describe.py::TestDescribe::test_text
30.88s call tests/featurize/test_batch.py::TestExceptions::test_batch_summary_featurizer_exception
20.32s call tests/cli/test_cli.py::TestCLI::test_cli_interactive_plotter_coops
19.63s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_cdf_comp_range_coop
19.24s call tests/featurize/test_batch.py::TestBatchSummaryFeaturizer::test_summary_featurize_with_json_overall
15.49s call tests/featurize/test_batch.py::TestBatchSummaryFeaturizer::test_summary_featurize_orbitalwise
14.10s call tests/featurize/test_batch.py::TestBatchSummaryFeaturizer::test_summary_featurize_without_json
9.63s call tests/cohp/test_describe.py::TestCalcQualityDescribeWarnings::test_warnings
9.38s setup tests/cohp/test_describe.py::TestDescribe::test_write_description
7.76s call tests/featurize/test_batch.py::TestBatchSummaryFeaturizer::test_summary_featurize_with_json
7.22s call tests/featurize/test_batch.py::TestBatchSummaryFeaturizer::test_summary_featurize_with_json_antibonding
7.18s call tests/featurize/test_core.py::TestFeaturizeLobsterpy::test_featurize_mp463
7.16s call tests/featurize/test_batch.py::TestBatchSummaryFeaturizer::test_summary_featurize_with_json_bonding
7.07s call tests/featurize/test_batch.py::TestBatchStructureGraphs::test_batch_structure_graphs_cation_anion_bonds
6.99s call tests/featurize/test_batch.py::TestBatchStructureGraphs::test_batch_structure_graphs_all_bonds
6.94s call tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_plot_colors
6.77s call tests/cli/test_cli.py::TestCLI::test_nongz_file_cli
6.71s call tests/featurize/test_batch.py::TestBatchCoxxFingerprint::test_fp_cohp_overall
6.63s call tests/cli/test_cli.py::TestCLI::test_cli_interactive_plotter_cobi
6.55s call tests/featurize/test_batch.py::TestBatchCoxxFingerprint::test_fp_coop
6.54s call tests/featurize/test_batch.py::TestBatchCoxxFingerprint::test_fp_cohp_bonding
6.54s call tests/featurize/test_batch.py::TestBatchCoxxFingerprint::test_fp_cobi
5.98s call tests/featurize/test_batch.py::TestBatchSummaryFeaturizer::test_summary_featurize_with_no_bonds
5.17s setup tests/cohp/test_analyze.py::TestAnalyse::test_final_dicts
5.07s call tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_k3sb
5.01s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_nacl_comp_range_orbital
5.01s setup tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_nacl_cobi_orb
4.97s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_nacl_comp_range_cobi_orbital
4.89s setup tests/cohp/test_analyze.py::TestAnalyse::test_icohp_sum_nacl
4.83s call tests/structuregraph/test_graph.py::TestGraph::test_graph_nacl_close_fermi
4.81s call tests/structuregraph/test_graph.py::TestGraph::test_graph_nacl_cation_anion
4.78s call tests/structuregraph/test_graph.py::TestGraph::test_graph_nacl_all
4.75s call tests/structuregraph/test_graph.py::TestGraph::test_graph_nacl_without_add_data
4.75s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_analyse_nacl_comp_range_cobi
4.72s setup tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_nacl_cobi
4.68s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_k3sb_all_coop_orb
4.36s call tests/structuregraph/test_graph.py::TestGraph::test_graph_cdf_all
4.32s call tests/cli/test_cli.py::TestCLI::test_calc_quality_summary_nacl
4.13s setup tests/cohp/test_analyze.py::TestAnalyse::test_icohp_sum_cdf
4.11s setup tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_cdf
4.07s call tests/structuregraph/test_graph.py::TestGraph::test_graph_cdf_close_fermi
4.01s setup tests/cohp/test_describe.py::TestDescribe::test_plot
3.79s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_k3sb
3.76s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_k3sb_all
3.71s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_k3sb_all_cobi
3.69s setup tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_k3sb
3.60s setup tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_plot_colors
3.58s call tests/featurize/test_core.py::TestFeaturizeCOXX::test_featurize_k3sb_coxx
3.00s call tests/featurize/test_core.py::TestFeaturizeLobsterpy::test_featurize_csh_madelung [51/2876]
2.67s call tests/cli/test_cli.py::TestCLI::test_dos_plot
2.66s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_nasbf6
2.64s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_nasbf6_anbd
2.48s setup tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_cohps_from_plot_data
2.11s call tests/cli/test_cli.py::TestCLI::test_calc_quality_summary_k3sb
2.10s call tests/cohp/test_describe.py::TestCalcQualityDescribe::test_calc_quality_description_text
2.05s setup tests/plotting/test_plotting.py::TestPlotterExceptions::test_plotter_exception
1.67s setup tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_batio3_orb
1.65s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_cdf
1.32s call tests/cli/test_cli.py::TestCLI::test_hideplot_cli
0.99s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_batio3
0.93s call tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_nacl_cobi_orb
0.78s call tests/featurize/test_batch.py::TestBatchDosFeaturizer::test_batch_dos_featurizer_lso
0.75s call tests/featurize/test_batch.py::TestBatchDosFeaturizer::test_batch_dos_featurizer_non_lso
0.70s call tests/featurize/test_core.py::TestFeaturizeCOXX::test_featurize_cdf_coxx
0.67s call tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_batio3_orb
0.65s call tests/cohp/test_describe.py::TestDescribe::test_plot
0.59s call tests/featurize/test_core.py::TestFeaturizeCharges::test_featurize_k3sb_charge
0.55s call tests/cli/test_cli.py::TestCLI::test_plot_saved
0.51s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_nacl_mulliken
0.51s call tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_cohps_from_plot_data_json
0.50s call tests/plotting/test_plotting.py::TestPlainDosPlotter::test_nacl_dos
0.50s call tests/cli/test_cli.py::TestCLI::test_cli_results[args3]
0.50s call tests/cli/test_cli.py::TestCLI::test_icoxxlist_plots
0.46s call tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_cdf
0.44s setup tests/plotting/test_plotting.py::TestPlainDosPlotter::test_add_site_orbital_dos
0.44s call tests/featurize/test_core.py::TestFeaturizeDoscar::test_featurize_k3sb_dos
0.41s call tests/featurize/test_core.py::TestExceptions::test_featurize_coxx
0.33s call tests/cli/test_cli.py::TestCLI::test_cli_results[args0]
0.32s call tests/cli/test_cli.py::TestCLI::test_cli_exceptions
0.30s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_nacl_valences
0.30s call tests/cli/test_cli.py::TestCLI::test_cli_interactive_plotter
0.30s call tests/cli/test_cli.py::TestCLI::test_iaplot_saved
0.29s call tests/cli/test_cli.py::TestCLI::test_cli_results[args2]
0.29s call tests/cli/test_cli.py::TestCLI::test_cli_results[args1]
0.29s setup tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_cohps_by_lobster_label_nacl
0.28s setup tests/plotting/test_plotting.py::TestPlainDosPlotter::test_k3sb_dos
0.28s setup tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_nacl
0.27s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_nacl_all
0.27s call tests/cli/test_cli.py::TestCLI::test_cli_results[args4]
0.27s call tests/cli/test_cli.py::TestCLI::test_cli_results[args5]
0.27s call tests/cli/test_cli.py::TestCLI::test_json_saved
0.26s setup tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_cohp_nacl
0.25s call tests/featurize/test_core.py::TestFeaturizeLobsterpy::test_featurize_mp7000_json
0.25s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_nacl_nan
0.20s call tests/featurize/test_core.py::TestFeaturizeDoscar::test_featurize_nacl_dos
0.19s call tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_nacl
0.18s call tests/plotting/test_plotting.py::TestPlainDosPlotter::test_add_site_orbital_dos
<snip>berquist commented
Making all fixtures scoped to the session (since none seem to return anything mutable) appears to help, but time is actually only down from 7 and a half to 7 minutes. So there was not a substantial amount of duplicate work being done in setup after all. xdist over 8 workers is still one and a half minutes.
Also, if no one has complained about local testing time, the effort for adding markers may not be worth it. CI seems pretty fast thanks to using the splits.
Feel free to close this if you agree that it isn't worthwhile.
37.12s call tests/featurize/test_batch.py::TestExceptions::test_batch_summary_featurizer_exception
22.68s call tests/featurize/test_batch.py::TestBatchSummaryFeaturizer::test_summary_featurize_with_json_overall
20.21s call tests/cli/test_cli.py::TestCLI::test_cli_interactive_plotter_coops
19.96s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_cdf_comp_range_coop
17.37s call tests/featurize/test_batch.py::TestBatchSummaryFeaturizer::test_summary_featurize_orbitalwise
16.61s call tests/featurize/test_batch.py::TestBatchSummaryFeaturizer::test_summary_featurize_without_json
9.57s call tests/cohp/test_describe.py::TestCalcQualityDescribeWarnings::test_warnings
9.05s call tests/featurize/test_batch.py::TestBatchSummaryFeaturizer::test_summary_featurize_with_json_antibonding
9.00s call tests/featurize/test_batch.py::TestBatchSummaryFeaturizer::test_summary_featurize_with_json_bonding
8.82s call tests/featurize/test_batch.py::TestBatchSummaryFeaturizer::test_summary_featurize_with_json
8.35s call tests/featurize/test_batch.py::TestBatchStructureGraphs::test_batch_structure_graphs_cation_anion_bonds
8.19s call tests/featurize/test_core.py::TestFeaturizeLobsterpy::test_featurize_mp463
8.15s call tests/featurize/test_batch.py::TestBatchStructureGraphs::test_batch_structure_graphs_all_bonds
7.90s call tests/featurize/test_batch.py::TestBatchCoxxFingerprint::test_fp_cobi
7.72s call tests/featurize/test_batch.py::TestBatchCoxxFingerprint::test_fp_cohp_bonding
7.62s call tests/featurize/test_batch.py::TestBatchCoxxFingerprint::test_fp_cohp_overall
7.54s call tests/featurize/test_batch.py::TestBatchCoxxFingerprint::test_fp_coop
7.40s call tests/featurize/test_batch.py::TestBatchSummaryFeaturizer::test_summary_featurize_with_no_bonds
6.81s call tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_plot_colors
6.76s setup tests/cohp/test_describe.py::TestDescribe::test_text
6.44s call tests/cli/test_cli.py::TestCLI::test_nongz_file_cli
6.01s call tests/structuregraph/test_graph.py::TestGraph::test_graph_nacl_without_add_data
6.00s call tests/structuregraph/test_graph.py::TestGraph::test_graph_nacl_all
5.98s call tests/cli/test_cli.py::TestCLI::test_cli_interactive_plotter_cobi
5.85s call tests/structuregraph/test_graph.py::TestGraph::test_graph_nacl_close_fermi
5.74s call tests/structuregraph/test_graph.py::TestGraph::test_graph_nacl_cation_anion
5.59s call tests/structuregraph/test_graph.py::TestGraph::test_graph_cdf_all
5.57s setup tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_nacl_cobi_orb
5.50s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_analyse_nacl_comp_range_cobi
5.49s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_nacl_comp_range_orbital
5.44s setup tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_nacl_cobi
5.00s setup tests/cohp/test_analyze.py::TestAnalyse::test_icohp_sum_nacl
4.96s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_nacl_comp_range_cobi_orbital
4.93s call tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_k3sb
4.86s call tests/structuregraph/test_graph.py::TestGraph::test_graph_cdf_close_fermi
4.81s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_k3sb_all_coop_orb
4.70s setup tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_cdf
4.64s setup tests/cohp/test_analyze.py::TestAnalyse::test_icohp_sum_cdf
4.32s call tests/cli/test_cli.py::TestCLI::test_calc_quality_summary_nacl
4.32s setup tests/plotting/test_plotting.py::TestInteractiveCohpPlotter::test_add_all_relevant_cohps_k3sb
4.08s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_k3sb
3.87s call tests/featurize/test_core.py::TestFeaturizeCOXX::test_featurize_k3sb_coxx
3.73s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_k3sb_all
3.72s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_k3sb_all_cobi
3.50s setup tests/cohp/test_describe.py::TestDescribe::test_plot
3.20s setup tests/cohp/test_analyze.py::TestAnalyse::test_final_dicts
3.11s setup tests/cohp/test_analyze.py::TestAnalyse::test_all_attributes_nasbf6_anbd