Cobrijani/project-crawler

Abstraction over fetching data from repository management services

Project Crawler

Small project to iterate over and fetch files from different repository management tools like Github, Gitlab, BitBucket

Examples

Fetch the repositories for an org and download a file from it

ProjectCrawler crawler = new ProjectCrawler(OptionsBuilder.builder()
  // basing the root URL we can resolve the type of repo (e.g. https://gitlab.com)
  .rootUrl(urlRoot)
  // username to access the API
  .username(username)
  // password to access the API
  .password(password)
  // token to access the API
  .token(token)
  // repository type (GITHUB, BITBUCKET, GITLAB, OTHER)
  .repository(repoType)
  .build());
// get the repos from the org
List<Repository> repositories = crawler.repositories(org);
repositories.each { Repository repo ->
  // fetch a file from the repository
  String file = crawler.fileContent(org, repo.name, repo.requestedBranch, "path/to/file.txt")
}

For BitBucket:

• remember to pass the URL to the API (e.g. https://api.bitbucket.org).

• we support only the 2.0 API.

Adding your own implementation

If you’re using some other tool than Github, Gitlab or Bitbucket you can write an implementation that integrates with that tool.

We’re using the standard, Java ServiceLoader mechanism to load any extensions and the interface to implement is RepositoryManagementBuilder.

To do that just create a file called META-INF/io.cloudpipelines.projectcrawler.RepositoryManagementBuilder that is accessible on classpath. The file should contain a line with fully qualified name of your implementation class (e.g. com.example.TestRepositoryManagementBuilder)