Can the ORT Gitlab CI pipeline checkout private repositories ?

[bcopy]

I have setup the pipeline and required tokens, but when trying to checkout a private repository, the ORT git downloader fails with :

Downloading from Git VCS at https://gitlab.cern.ch/acc-co/myprivaterepo.git...
org.ossreviewtoolkit.downloader.DownloadException: Download failed for 'Downloader::myprivaterepo:'.
	at org.ossreviewtoolkit.downloader.Downloader.download(Downloader.kt:77)
	at org.ossreviewtoolkit.plugins.commands.downloader.DownloaderCommand.downloadFromProjectUrl(DownloaderCommand.kt:419)
	at org.ossreviewtoolkit.plugins.commands.downloader.DownloaderCommand.run(DownloaderCommand.kt:208)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:306)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:319)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:40)
(...)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:513)
	at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:66)
	Suppressed: org.ossreviewtoolkit.downloader.DownloadException: Git failed to get revisions from URL 'https://gitlab.cern.ch/myprivaterepo.git'.
		at org.ossreviewtoolkit.downloader.VersionControlSystem.download(VersionControlSystem.kt:249)
		at org.ossreviewtoolkit.downloader.Downloader.downloadFromVcs(Downloader.kt:271)
		at org.ossreviewtoolkit.downloader.Downloader.downloadFromVcs$default(Downloader.kt:184)
		at org.ossreviewtoolkit.downloader.Downloader.handleVcsDownload(Downloader.kt:[109](https://gitlab.cern.ch/acc-co/devops/dependency-management/ort-gitlab-ci/-/jobs/31863565#L109))
		at org.ossreviewtoolkit.downloader.Downloader.download(Downloader.kt:81)
		... 10 more
	Caused by: org.ossreviewtoolkit.downloader.DownloadException: Unable to determine a revision to checkout.
		at org.ossreviewtoolkit.downloader.VersionControlSystem.getRevisionCandidates-0E7RQCE(VersionControlSystem.kt:302)
		at org.ossreviewtoolkit.downloader.VersionControlSystem.download(VersionControlSystem.kt:248)
		... 14 more
		Suppressed: java.io.IOException: Cannot guess a revision name from a blank version.
			at org.ossreviewtoolkit.downloader.WorkingTree.guessRevisionName(WorkingTree.kt:89)
			at org.ossreviewtoolkit.downloader.VersionControlSystem.getRevisionCandidates_0E7RQCE$addGuessedRevision(VersionControlSystem.kt:306)
			at org.ossreviewtoolkit.downloader.VersionControlSystem.getRevisionCandidates-0E7RQCE(VersionControlSystem.kt:353)
			... 15 more
		Suppressed: java.io.IOException: Unable to list the remote branches.
			at org.ossreviewtoolkit.downloader.vcs.GitWorkingTree$listRemoteBranches$1.invoke(GitWorkingTree.kt:149)
			at org.ossreviewtoolkit.downloader.vcs.GitWorkingTree$listRemoteBranches$1.invoke(GitWorkingTree.kt:143)
			at org.ossreviewtoolkit.downloader.vcs.GitWorkingTree.useRepo(GitWorkingTree.kt:60)
			at org.ossreviewtoolkit.downloader.vcs.GitWorkingTree.listRemoteBranches(GitWorkingTree.kt:143)
			at org.ossreviewtoolkit.downloader.VersionControlSystem.isFixedRevision-gIAlu-s(VersionControlSystem.kt:404)
			at org.ossreviewtoolkit.downloader.VersionControlSystem.getRevisionCandidates_0E7RQCE$addMetadataRevision(VersionControlSystem.kt:328)
			at org.ossreviewtoolkit.downloader.VersionControlSystem.getRevisionCandidates-0E7RQCE(VersionControlSystem.kt:363)
			... 15 more
		Caused by: org.eclipse.jgit.api.errors.TransportException: https://gitlab.cern.ch/myprivaterepo.git: not authorized
			at org.eclipse.jgit.api.LsRemoteCommand.execute(LsRemoteCommand.java:192)
			at org.eclipse.jgit.api.LsRemoteCommand.call(LsRemoteCommand.java:131)
			at org.ossreviewtoolkit.downloader.vcs.GitWorkingTree$listRemoteBranches$1.invoke(GitWorkingTree.kt:145)
			... 21 more
		Caused by: org.eclipse.jgit.errors.TransportException: https://gitlab.cern.ch/myprivaterepo.git: not authorized
			at org.eclipse.jgit.transport.TransportHttp.connect(TransportHttp.java:696)
			at org.eclipse.jgit.transport.TransportHttp.openFetch(TransportHttp.java:465)
			at org.eclipse.jgit.api.LsRemoteCommand.execute(LsRemoteCommand.java:170)
			... 23 more
	Suppressed: org.ossreviewtoolkit.downloader.DownloadException: No source artifact URL provided.
		at org.ossreviewtoolkit.downloader.Downloader.downloadSourceArtifact(Downloader.kt:311)
		at org.ossreviewtoolkit.downloader.Downloader.handleSourceArtifactDownload(Downloader.kt:150)
		at org.ossreviewtoolkit.downloader.Downloader.download(Downloader.kt:82)
		... 10 more
The download took 796.761771ms.
07:26:00.198 [main] ERROR org.ossreviewtoolkit.plugins.commands.downloader.DownloaderCommand - The following failure(s) occurred:
DownloadException: Download failed for 'Downloader::myprivaterepo:'.
Suppressed: DownloadException: Git failed to get revisions from URL 'https://gitlab.cern.ch/myprivaterepo.git'.
    Caused by: DownloadException: Unable to determine a revision to checkout.
        Suppressed: IOException: Cannot guess a revision name from a blank version.
        Suppressed: IOException: Unable to list the remote branches.
            Caused by: TransportException: https://gitlab.cern.ch/myprivaterepo.git: not authorized
                Caused by: TransportException: https://gitlab.cern.ch/myprivaterepo.git: not authorizedSuppressed: DownloadException: No source artifact URL provided.

Does the Git downloader need to be provided with credentials ? How does one configure the authenticator ?
In particular, oss-review-toolkit/ort#2580

I would expect that Gitlab provides all required environment including authentication, but does JGit pick up on that ?

[bcopy]

One workaround is to use the CI token and place it in ~/.netrc - Shall I suggest a push request ?

[tsteenbe]

@bcopy Sorry for delay in commenting on this issue was dealing with family health issues and took a break from my FOSS work Yes, ORT can scan private repositories - we made this now hopefully clearer in the Run ORT on private repositories within the README.md.