Mutahunter

Open-Source Language Agnostic LLM-based Mutation Testing for Automated Software Testing

Overview
Features
Getting Started
CI/CD Integration
Roadmap
Cash Bounty Program

Overview

Mutation testing is used by big tech companies like Google to ensure the robustness of their test suites. With Mutahunter, we aim to empower other companies and developers to use this powerful tool to enhance their test suites and improve software quality.

Mutation testing verifies the effectiveness of your test cases by creating small changes, or “mutants,” in the code and checking if the test cases can catch these changes. Unlike line coverage, which only shows code execution, mutation testing reveals how well the code is tested.

Mutahunter uses LLM models to inject context-aware faults into your codebase. This AI-driven approach produces fewer equivalent mutants, mutants with higher fault detection potential, and those with higher coupling and semantic similarity to real faults, ensuring comprehensive and effective testing.

Features

Extreme Mutation Testing: Leverages language agnostic TreeSitter parser to apply extreme mutations to the codebase without using LLMs. Research shows that this approach is effective at detecting pseudo-tested methods with significantly lower computational cost. Currently supports Python, Java, JavaScript, and Go. Check the scheme files to see the supported operators. We welcome contributions to add more operators and languages.
LLM Context-aware Mutations: Utilizes LLM models to generate context-aware mutants. Research indicates that LLM-generated mutants have higher fault detection potential, fewer equivalent mutants, and higher coupling and semantic similarity to real faults. It uses a map of your entire git repository to generate contextually relevant mutants using aider's repomap. Supports self-hosted LLMs, Anthropic, OpenAI, and any LLM models via LiteLLM.
Change-Based Testing: Runs mutation tests on modified files and lines based on the latest commit or pull request changes, ensuring that only relevant parts of the code are tested.
Language Agnostic: Compatible with languages that provide coverage reports in Cobertura XML, Jacoco XML, and lcov formats. Extensible to additional languages and testing frameworks.
Detailed Mutant Reports: Provides comprehensive reports on mutation coverage, killed mutants, and survived mutants.

Recommended Mutation Testing Process

Achieve High Line Coverage: Ensure your test suite has high line coverage, preferably 100%.
Strict Mutation Testing: Use strict mutation testing to improve mutation coverage without additional cost. Utilize the --only-mutate-file-paths flag for targeted testing on critical files.
LLM-Based Mutation Testing on Changed Files: Inject context-aware mutants using LLMs on changed files during pull requests as the final line of defense. Use the --modified-files-only flag to focus on recent changes. In this way it will make the mutation testing significantly faster and cost effective.

Getting Started

# Install Mutahunter package via GitHub. Python 3.11+ is required.
$ pip install git+https://github.com/codeintegrity-ai/mutahunter.git

# Work with GPT-4o on your repo
$ export OPENAI_API_KEY=your-key-goes-here

# Or, work with Anthropic's models
$ export ANTHROPIC_API_KEY=your-key-goes-here

# Run Mutahunter on a specific file. 
# Coverage report should correspond to the test command.
$ mutahunter run --test-command "pytest tests/unit" --code-coverage-report-path "coverage.xml" --only-mutate-file-paths "app_1.py" "app_2.py"

# Run mutation testing on modified files based on the latest commit
$ mutahunter run --test-command "pytest tests/unit" --code-coverage-report-path "coverage.xml" --modified-files-only

.  . . . .-. .-. . . . . . . .-. .-. .-. 
|\/| | |  |  |-| |-| | | |\|  |  |-  |(  
'  ` `-'  '  ` ' ' ` `-' ' `  '  `-' ' ' 

2024-07-05 00:26:13,420 INFO: 📊 Line Coverage: 100% 📊
2024-07-05 00:26:13,420 INFO: 🎯 Mutation Coverage: 61.54% 🎯
2024-07-05 00:26:13,420 INFO: 🦠 Total Mutants: 13 🦠
2024-07-05 00:26:13,420 INFO: 🛡️ Survived Mutants: 5 🛡️
2024-07-05 00:26:13,420 INFO: 🗡️ Killed Mutants: 8 🗡️
2024-07-05 00:26:13,421 INFO: 🕒 Timeout Mutants: 0 🕒
2024-07-05 00:26:13,421 INFO: 🔥 Compile Error Mutants: 0 🔥
2024-07-05 00:26:13,421 INFO: 💰 Total Cost: $0.00583 USD 💰
2024-07-05 00:26:13,421 INFO: Report saved to logs/_latest/mutation_coverage.json
2024-07-05 00:26:13,421 INFO: Report saved to logs/_latest/mutation_coverage_detail.json
2024-07-05 00:26:13,421 INFO: Mutation Testing Ended. Took 43s

Examples

Go to the examples directory to see how to run Mutahunter on different programming languages:

Check Java Example to see some interesting LLM-based mutation testing examples.

Feel free to add more examples! ✨

Mutant Report

Check the logs directory to view the report:

mutants.json - Contains the list of mutants generated.
coverage.txt - Contains information about mutation coverage.

CI/CD Integration

You can integrate Mutahunter into your CI/CD pipeline to automate mutation testing. Here is an example GitHub Actions workflow file:

name: Mutahunter CI/CD 

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  mutahunter:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 2 # needed for git diff

      - name: Set up Python 
        uses: actions/setup-python@v5
        with:
          python-version: 3.11

      - name: Install Mutahunter
        run: pip install git+https://github.com/codeintegrity-ai/mutahunter.git

      - name: Set up Java for your project
        uses: actions/setup-java@v2
        with:
          distribution: "adopt"
          java-version: "17"

      - name: Install dependencies and run tests
        run: mvn test

      - name: Run Mutahunter
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          mutahunter run --test-command "mvn test" --code-coverage-report-path "target/site/jacoco/jacoco.xml" --coverage-type jacoco --model "gpt-4o" --modified-files-only 

      - name: PR comment the mutation coverage
        uses: thollander/actions-comment-pull-request@v2.5.0
        with:
          filePath: logs/_latest/coverage.txt

Roadmap

Fault Injection: Utilize advanced LLM models to inject context-aware faults into the codebase, ensuring comprehensive mutation testing.
Language Support: Expand support to include various programming languages.
Support for Other Coverage Report Formats: Add compatibility for various coverage report formats.
Change-Based Testing: Implement mutation testing on modified files based on the latest commit or pull request changes.
Extreme Mutation Testing: Apply mutations to the codebase without using LLMs to detect pseudo-tested methods with significantly lower computational cost.
CI/CD Integration: Display mutation coverage in pull requests and automate mutation testing using GitHub Actions.
Mutant Analysis: Automatically analyze survived mutants to identify potential weaknesses in the test suite.

Cash Bounty Program

Help us improve Mutahunter and get rewarded! We have a cash bounty program to incentivize contributions to the project. Check out the bounty board to see the available bounties and claim one today!

Acknowledgements

Mutahunter makes use of the following open-source libraries:

aider by Paul Gauthier, licensed under the Apache-2.0 license.
TreeSitter by TreeSitter, MIT License.
LiteLLM by BerriAI, MIT License.

For more details, please refer to the LICENSE file in the repository.

codeintegrity-ai/mutahunter