source{d} datasets for source code analysis and machine learning on source code (ML on Code).
This repository contains all the needed tools and scripts to reproduce the datasets, as well as the academic papers they may relate to.
- Public Git Archive
- Size: 3TB
- Description: 180k+ top-bookmarked repositories from GitHub, consisting of 54M+ files and ~16 billion lines of code over 450+ programming languages.
- Programming Language Identifiers
- Size: 1GB
- Description: ~49M distinct identifiers extracted from 10+ programming languages.
- Manually labelled pairs of files and functions
- Size: 250MB
- Description: 2k Java file and 600 Java function pairs labeled as similar or different by several programmers.
- PR review comments
- Size: 1.5GB
- Description: 25.3 million GitHub PR review comments since January 2015 till December 2018.
- Commit messages
- Size: 46GB
- Description: 1.3 billion GitHub commit messages till March 2019.
- Structural commit features
- Size: 1.9GB
- Description: 1.6 million commits in 622 Java repositories on GitHub.
Contributions are very welcome, please see CONTRIBUTING.md and code of conduct.
The tools and scripts are licensed under Apache 2.0, see LICENSE.md.