Smart Tester provides the generation of unit tests for the given functions, through multi-step prompts to an LLM.
This approach is inspired by "Unit test writing using a multi-step prompt" by Ted Sanders (https://github.com/ted-at-openai), from the official OpenAI Cookbook on Github (https://github.com/openai/openai-cookbook/blob/main/examples/Unit_test_writing_using_a_multi-step_prompt.ipynb).
As suggested by the CookBook, 4 base steps are provided:
- Explain: Given a Python function, we ask GPT to explain what the function is doing and why..
- Plan: We ask GPT to plan a set of unit tests for the function.
- If the plan is too short, we ask GPT to elaborate with more ideas for unit tests.
- Execute: Finally, we instruct GPT to write unit tests that cover the planned cases.
In this project 2 additional steps are structured: 5. Post-processing: Fix common errors produced by the LLM-generated unit test code 6. Verify: Verify if the post-processed code is executable
- Develop an LLM-based service to generate test functions for Python, with pytest (and pytest.mark.parametrize)
- Saved intermediate steps in persistent files to review step outputs and restart computation from an intermediate step
- Clone/fork the project
- Open the project using your favorite SDK (PyCharm, Visual Studio Code, ...)
- Set the environment variable "OPENAI_API_KEY" to link your OpenAI API Key (https://platform.openai.com/account/api-keys)
- Execute the main function from smarttester\scripts\gtp35_python_smart_testing_example_runner.py, using the desired function_to_tests and parameters, as follows in this screenshot
- If parameter "print_text" is set as True, results are printed in the terminal:
- If parameter "save_text" is set as True, results and intermediate steps are saved as files in _smarttester\package\smarttester\resources\data\saved_files
-
Automated post-processing to fix usual LLM's specific errors in generating pytest tests
-
Exposition of API to call pytest test generation services from a client
-
Develop an LLM-based service to generate test functions for Java, with junit5 (and ParameterizedTest)
-
Automated post-processing to fix usual LLM's specific errors in generating junit tests
-
Exposition of API to call junit test generation services from a client
-
Exposition of a Frontend to provide an interface to the exposed API