commit0 test Output is Always the Same, Before and After Changes

[JingtianWu]

Description:

I am encountering an issue where the test results from commit0 test remain the same, regardless of changes made to the simpy files. The output always shows 57 failed, 83 passed, even after:

• Deleting all files in the simpy folder, leaving it empty.
• Reinstalling commit0 and all dependencies.
• Running commit0 test simpy tests/ before and after using Aider, which should affect the test outcomes.

Steps to Reproduce:

1. Run commit0 test simpy tests/ in my project directory.
2. Observe that the output remains the same (57 failed, 83 passed) regardless of any file changes or deletions in the simpy folder.
3. Attempt reinstallation and cleanup of the environment, which did not change the outcome.

Expected Behavior:
The test output should reflect changes in the simpy files, showing different results after modifications or deletions.

[wenting-zhao]

Your steps to reproduce are insufficient. Please also include your steps to set up the commit0 library (e.g., did you do pip installation? how did you set up the virtual environment)? Please also include your commit0 config file.

[JingtianWu]

Title: commit0 test Output is Always the Same, Even After Changes to SimPy Files

---

Description:

I am encountering an issue where the test results from commit0 test remain the same, regardless of changes made to the simpy files. The output always shows 57 failed, 83 passed, even after:

• Deleting all files in the simpy folder, leaving it empty.
• Reinstalling commit0 and all dependencies.
• Running commit0 test simpy tests/ before and after using Aider, which should affect the test outcomes.

Environment Setup

1. Virtual Environment Setup:

   • Created a virtual environment using Python 3.12:

     python3 -m venv .venv
     source .venv/bin/activate

2. Installing commit0 and Dependencies:

   • Installed commit0 and other dependencies using pip:

     pip install commit0
     pip install -r requirements-dev.txt

3. Commit0 Configuration File (.commit0.config):

   • Here is the content of my .commit0.config file:

     defaults:
       - _self_
     
     # shared in all steps
     dataset_name: wentingzhao/commit0_docstring
     dataset_split: test
     
     # clone related
     base_dir: repos/
     repo_split: all
     
     # build related
     num_workers: 8
     
     # test related
     backend: local
     timeout: 1_800
     num_cpus: 1
     
     # save related
     github_token: null

Steps to Reproduce

1. Set up the virtual environment as described above.
2. Run commit0 test:

   PYTHONPATH=$(pwd) commit0 test simpy tests/

[wenting-zhao]

Which commit0 version are you on? The dataset wentingzhao/commit0_docstring doesn't even exist any more

[wenting-zhao]

The issue was that changes were not properly committed.