A simple Python3 tool (also available as a GitHub Action) to detect similarities between files within a repository.
A command line tool that receives a directory or a list of files and determines the degree of similarity between them.
The tool intends guide the refactoring efforts of a developer who wishes to reduce code duplication within a component and improve its software architecture.
Its development was initiated within the context of the DAT265 - Software Evolution Project.
The tool uses the gensim Python library to determine the similarity between source code files, supplied by the user. The default supported languages are C, C++, JAVA, Python and C#.
The following Python packages have to be installed:
- nltk
pip3 install --user nltk
- gensim
pip3 install --user gensim
- astor
pip3 install --user astor
- punkt
python3 -m nltk.downloader punkt
Suppress the warnings (generated by the used libraries)
as python3 -W ignore duplicate_code_detection.py and then supply the necessary
arguments. More details can be found by running the tool with the --help option.
Notice: Due to the way the models are created, the more source files you provide the tool the more accurate the similarity calculations are. In other words, the bigger the project, the more useful the tool is.
If duplicate-code-detection-tool is the name where the tool resides in and
smartcar_shield/src contains the repository you want to check for source code
similarities between the files, then you can run the following to get the
similarity report:
python3 -W ignore duplicate-code-detection-tool/duplicate_code_detection.py -d smartcar_shield/src/
The result should look something like this:
The tool is also available as a GitHub Action for easy integration with projects hosted on GitHub. An example output of the tool can be seen here.
The Action is meant to be triggered during pull requests to give the developers an impression over the degree of similarity between the files in the source code. Below you will find a sample workflow files that illustrate the usage.
Depending on the size of your project, you may want to have the tool running multiple times (i.e in diffferent steps) that test specific parts of your repository for duplicate code. This way you will not compare each file in your codebase with everything else and get back more meaningful reports.
In the following example the tool will examine source code (the languages supported by default)
in the src/ and test/ut directories relative to the root directory of your repository.
The results will be posted as a comment in the pull request that was opened.
name: Duplicate code
on: pull_request
jobs:
duplicate-code-check:
name: Check for duplicate code
runs-on: ubuntu-20.04
steps:
- name: Check for duplicate code
uses: platisd/duplicate-code-detection-tool@master
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
directories: "src/, test/ut"If you want to avoid the "spam" you should configure the tool to not always run. Specifically, if you wish to trigger the Action manually, you can do so by leaving a comment in the pull request.
The following action will trigger the tool to be run when a comment containig run_duplicate_code_detection_tool
is posted in a pull request. The tool will run using the code in the pull request.
name: Duplicate code
on: issue_comment
jobs:
duplicate-code-check:
name: Check for duplicate code
# Trigger the tool only when a comment containing the keyword is published in a pull request
if: github.event.issue.pull_request && contains(github.event.comment.body, 'run_duplicate_code_detection_tool')
runs-on: ubuntu-20.04
steps:
- name: Check for duplicate code
uses: platisd/duplicate-code-detection-tool@master
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
directories: "."Important: Please note that due to the way GitHub Actions work, you will first have to merge this into your main branch so it starts taking effect.
It may not make sense to compare all files or get a files with very low similarity reported. In the following workflow, the different optional arguments are demonstrated.
For the various default values, please consult action.yml.
name: Duplicate code
on: pull_request
jobs:
duplicate-code-check:
name: Check for duplicate code
runs-on: ubuntu-20.04
steps:
- name: Check for duplicate code
uses: platisd/duplicate-code-detection-tool@master
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
directories: "src"
# Ignore the specified directories
ignore_directories: "src/external_libraries"
# Only examine .h and .cpp files
file_extensions: "h, cpp"
# Only report similarities above 5%
ignore_below: 5
# If a file is more than 70% similar to another, then the job fails
fail_above: 70
# If a file is more than 15% similar to another, show a warning symbol in the report
warn_above: 15
# Remove `src/` from the file paths when reporting similarities
project_root_dir: "src"
# Remove docstrings from code before analysis
# For python source code only. This is checked on a per-file basis
only_code: true
# Leave only one comment with the report and update it for consecutive runs
one_comment: true
# The message to be displayed at the start of the report
header_message_start: "The following files have a similarity above the threshold:"To use Duplicate Code Detection Tool as a pre-commit hook with pre-commit add the following to your .pre-commit-config.yaml file:
- repo: https://github.com/platisd/duplicate-code-detection-tool.git
rev: '' # Use the sha / tag you want to point at
hooks:
- id: duplicate-code-detectionNOTE: that this repository sets args:
-f, if you are configuring duplicate-code-detection-tool using args you'll want to include either-f(--files) or-d(--directories).
only_codeoption only works with python files for now