/datasets

source{d} datasets ("big code") for source code analysis and machine learning on source code

Primary LanguageJupyter NotebookOtherNOASSERTION

source{d} Datasets Build Status Build status

source{d} datasets for source code analysis and machine learning on source code (ML on Code).

This repository contains all the needed tools and scripts to reproduce the datasets, as well as the academic papers they may relate to.

Available datasets

Public Git Archive

  • Public Git Archive
  • Size: 6TB
  • Description: 260k+ top-bookmarked repositories from GitHub, consisting of 136M+ files and ~28 billion lines of code.

Programming Language Identifiers

Code duplicates

Pull Request review comments

  • PR review comments
  • Size: 1.5GB
  • Description: 25.3 million GitHub PR review comments since January 2015 till December 2018.

Commit messages

  • Commit messages
  • Size: 46GB
  • Description: 1.3 billion GitHub commit messages till March 2019.

Structural commit features

DockerHub Metadata

  • DockerHub Metadata
  • Size: 1.4GB
  • Description: 1.46 million Docker image configuration and manifest files on DockerHub fetched in June 2019.

DockerHub Packages

  • DockerHub Packages
  • Size: 15GB
  • Description: 419092 analyzed Docker images: lists of native, Python and Node packages on DockerHub fetched in summer 2019.

Typos

  • Typos
  • Size: 1MB
  • Description: 7375 typos in source code identifier names found in GitHub repositories.

NuGet Namespaces

  • NugetNamespaces
  • Size: 13MB
  • Description: information about 681,858 .NET namespaces extracted from 227,839 NuGet packages.

Contributions

Contributions are very welcome, please see CONTRIBUTING.md and code of conduct.

License

The tools and scripts are licensed under Apache 2.0, see LICENSE.md.