Git HTTPS authentication error

ORT has trouble accessing password-protected GIT repository
This surprises me as I think it used to work fine.

I have configured .git-credentials to store the username/password, and it works well when running git clone manually.

But when run via Gradle, I run in this error Authentication is required but no CredentialsProvider has been registered.

I found this issue in Gradle, about non-working authentication, but for Git protocol

17:40:56.526 [ScanCode-1] INFO  kotlinx.coroutines.CoroutineScope - No stored results found, scanning package Unmanaged::POC:4213d9ba0ecb5ef96f080e233945751443cb6b05 in thread 'ScanCode-1' (1/1).
17:40:56.527 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.Downloader - Trying to download source code for 'Unmanaged::POC:4213d9ba0ecb5ef96f080e233945751443cb6b05'.
17:40:56.529 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.Downloader - Trying to download 'Unmanaged::POC:4213d9ba0ecb5ef96f080e233945751443cb6b05' sources to '/data/ort-home/download/_project_ScanCode_20200505_174051/Unmanaged/unknown/POC/4213d9ba0ecb5ef96f080e233945751443cb6b05' from VCS...
17:40:56.529 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.Downloader - Using processed VcsInfo(type=Git, url=https://<git-url>, revision=4213d9ba0ecb5ef96f080e233945751443cb6b05, resolvedRevision=null, path=). Original was VcsInfo(type=, url=, revision=, resolvedRevision=null, path=).
17:40:56.771 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.Downloader - Detected VCS type 'Git' from type name 'Git'.
17:40:56.976 [ScanCode-1] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running 'git --version' in '/opt/ORT/oss-review-toolkit'...
17:40:57.552 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - Meta-data has invalid Git revision '4213d9ba0ecb5ef96f080e233945751443cb6b05': IOException: Unable to list the remote branches.
Caused by: TransportException: https://<git-url>: Authentication is required but no CredentialsProvider has been registered
Caused by: TransportException: https://<git-url>: Authentication is required but no CredentialsProvider has been registered
17:40:57.577 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - No Git revision for version '4213d9ba0ecb5ef96f080e233945751443cb6b05' found: IOException: Unable to list the remote tags.
Caused by: TransportException: https://<git-url>: Authentication is required but no CredentialsProvider has been registered
Caused by: TransportException: https://<git-url>: Authentication is required but no CredentialsProvider has been registered
17:40:57.581 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.Downloader - Trying to download source artifact for 'Unmanaged::POC:4213d9ba0ecb5ef96f080e233945751443cb6b05' from ...
17:40:57.584 [ScanCode-1] ERROR java.lang.Object - Could not download 'Unmanaged::POC:4213d9ba0ecb5ef96f080e233945751443cb6b05': DownloadException: Download failed for 'Unmanaged::POC:4213d9ba0ecb5ef96f080e233945751443cb6b05'.
Suppressed: DownloadException: Unable to determine a revision to checkout.,
Suppressed: DownloadException: No source artifact URL provided for 'Unmanaged::POC:4213d9ba0ecb5ef96f080e233945751443cb6b05'.
17:40:57.584 [ScanCode-1] INFO  kotlinx.coroutines.CoroutineScope - Finished scanning Unmanaged::POC:4213d9ba0ecb5ef96f080e233945751443cb6b05 (1/1).
17:40:57.587 [main] INFO  org.ossreviewtoolkit.model.OrtResult - Computing excluded projects which may take a while...
17:40:57.589 [main] INFO  org.ossreviewtoolkit.model.OrtResult - Computing excluded projects done.

This surprises me as I think it used to work fine.

It probably used to work back when we were still calling git CLI, but broke when migrating to JGit. I'll try to take a look in the next days.

Here is some background info.

As a work-around, you should be able to provide credentials as part of a .netrc file as mentioned here.

As a work-around, you should be able to provide credentials as part of a .netrc file as mentioned here.

Thanks @sschuberth ; do you mean that writing a .netrc should work out of the box?
Couldn't make it work with fresh source code.

do you mean that writing a .netrc should work out of the box?

That's what I thought. But it looks like JGit only uses its NetRCCredentialsProvider when using the JGit CLI (aka jgit.pgm), not when using the JGit API 😞

@NicolasToussaint, as I'm planning to work on this soon, what would be your expectation / preferred way of providing HTTP(S) credentials to ORT? Indeed parsing .netrc, or via system properties, or via the ORT configuration file, or anything else?

Actually, it does not really matters, but since you ask (thanks!), I would have a slight preference for using the ORT configuration file (so it's all in the same place).

But any way is fine, really.

@NicolasToussaint thanks for you input. I somewhat "had" to still go for .netrc support as we're still calling Git CLI in addition to using JGit, and Git CLI already supports .netrc, so that was the "smallest common denomiator".

If you can spend some time, I'd appreciate if you could give the .netrc support in my http-auth branch a try.

@sschuberth
I did the following, but still observe the same error:

checked out and rebuilt the http-auth branch (commit 60845864a1705236e2f254af46bc6a49d0c71b54)
wrote .netrecis user home directory
ran an analysis and scan

15:43:04.094 [main] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running '/opt/ScanCode/scancode-toolkit/bin/scancode --version' in '/opt/ScanCode/scancode-toolkit/bin'...
15:43:05.138 [FileBasedStorage with XZCompressedLocalFileStorage backend-1] INFO  kotlinx.coroutines.CoroutineScope - Looking for stored scan results for Unmanaged::_project_:4213d9ba0ecb5ef96f080e233945751443cb6b05 and ScannerDetails(name=ScanCode, version=3.0.2, configuration=--copyright --license --ignore *.ort.yml --info --strip-root --timeout 300 --ignore HERE_NOTICE --ignore META-INF/DEPENDENCIES --json-pp) (1/1).
15:43:05.146 [ScanCode-1] INFO  kotlinx.coroutines.CoroutineScope - No stored result found for Unmanaged::_project_:4213d9ba0ecb5ef96f080e233945751443cb6b05 and ScannerDetails(name=ScanCode, version=3.0.2, configuration=--copyright --license --ignore *.ort.yml --info --strip-root --timeout 300 --ignore HERE_NOTICE --ignore META-INF/DEPENDENCIES --json-pp), scanning package in thread 'ScanCode-1' (1/1).
15:43:05.148 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.Downloader - Trying to download source code for 'Unmanaged::_project_:4213d9ba0ecb5ef96f080e233945751443cb6b05'.
15:43:05.150 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.Downloader - Trying to download 'Unmanaged::_project_:4213d9ba0ecb5ef96f080e233945751443cb6b05' sources to '/data/ort-home/download/.../Unmanaged/unknown/_project_/4213d9ba0ecb5ef96f080e233945751443cb6b05' from VCS...
15:43:05.150 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.Downloader - Using processed VcsInfo(type=Git, url=https://_git-server_/.../_git/_project_, revision=4213d9ba0ecb5ef96f080e233945751443cb6b05, resolvedRevision=null, path=). Original was VcsInfo(type=, url=, revision=, resolvedRevision=null, path=).
15:43:05.367 [ScanCode-1] INFO  org.ossreviewtoolkit.utils.OrtAuthenticator - Authenticator was successfully installed.
15:43:05.368 [ScanCode-1] INFO  org.ossreviewtoolkit.utils.OrtProxySelector - Proxy selector is already installed.
15:43:05.524 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.Downloader - Detected VCS type 'Git' from type name 'Git'.
15:43:05.703 [ScanCode-1] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running 'git --version' in '/opt/ORT/oss-review-toolkit'...
15:43:10.577 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - Meta-data has invalid Git revision '4213d9ba0ecb5ef96f080e233945751443cb6b05': IOException: Unable to list the remote branches.
Caused by: TransportException: https://_git-server_/tfs/MTM/_project_/_git/_project_: Authentication is required but no CredentialsProvider has been registered
Caused by: TransportException: https://_git-server_/tfs/MTM/_project_/_git/_project_: Authentication is required but no CredentialsProvider has been registered
15:43:15.070 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - No Git revision for version '4213d9ba0ecb5ef96f080e233945751443cb6b05' found: IOException: Unable to list the remote tags.
Caused by: TransportException: https://_git-server_/tfs/MTM/_project_/_git/_project_: Authentication is required but no CredentialsProvider has been registered
Caused by: TransportException: https://_git-server_/tfs/MTM/_project_/_git/_project_: Authentication is required but no CredentialsProvider has been registered
15:43:15.074 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.Downloader - Trying to download source artifact for 'Unmanaged::_project_:4213d9ba0ecb5ef96f080e233945751443cb6b05' from ...
15:43:15.078 [ScanCode-1] ERROR java.lang.Object - Could not download 'Unmanaged::_project_:4213d9ba0ecb5ef96f080e233945751443cb6b05': DownloadException: Download failed for 'Unmanaged::_project_:4213d9ba0ecb5ef96f080e233945751443cb6b05'.
Suppressed: DownloadException: Unable to determine a revision to checkout., 
Suppressed: DownloadException: No source artifact URL provided for 'Unmanaged::_project_:4213d9ba0ecb5ef96f080e233945751443cb6b05'.
15:43:15.079 [ScanCode-1] INFO  kotlinx.coroutines.CoroutineScope - Finished scanning Unmanaged::_project_:4213d9ba0ecb5ef96f080e233945751443cb6b05 in thread 'ScanCode-1' (1/1).

wrote .netrecis user home directory

That's a .netrc, right?

That's a .netrc, right?

It is, sorry, typo in the comment only.

Anything specific I can do to dig deeper ?

Tried to do a little bit of debug ... but it's not exaclty easy not knowing any of Kotlin.

It turns out that the .netrc file is detected: I enter this ifblock:

ort/utils/src/main/kotlin/OrtAuthenticator.kt

Line 89 in 89b8241

if (netrcFile.isFile) {

But can't find a way to use the log.info stanza in this function:
fun getNetrcAuthentication(contents: String, machine: String): PasswordAuthentication?

ort/utils/src/main/kotlin/OrtAuthenticator.kt

Line 108 in 89b8241

    
           fun getNetrcAuthentication(contents: String, machine: String): PasswordAuthentication? {

I get this build error: e: /opt/ORT/oss-review-toolkit/utils/src/main/kotlin/OrtAuthenticator.kt: (118, 4): Unresolved reference: log

Tried to do a little bit of debug

Thanks!

But can't find a way to use the log.info stanza in this function

The log magic is only available for class instances, not for top-level functions. You could either work-around this by logging in the caller instead, like

getNetrcAuthentication(netrcFile.readText(), requestingHost)?.let {
    log.debug { "Parsing '$netrcFile' for machine '$requestingHost'." }
    return it
}

or create an anonymous dummy object like object {}.log.debug { "Debug" } and use that to log within the getNetrcAuthentication() function.

So maybe the syntax of your .netrc is not what we're expecting. Does it look anything like the ones in these tests?

Hello @sschuberth
I was trying to set up the config .netrc file but sadly it doesn't work in my case
I placed the file .netrc inside the directory which I indicate in the scanning proces but
finally I still obtain: Authentication is required but no CredentialsProvider has been registered
Is there any way I could track the issue?

Authentication is required but no CredentialsProvider has been registered

Could you please copy & paste the full log line for this and maybe some surrounding lines, so I can see which class the log comes from?

I placed the file .netrc inside the directory which I indicate in the scanning proces

After re-reading that sentence, I guess that won't work: The .netrc file is always being looked for the in the user's home directory, no matter what the ORT config directory is. The reason being that .netrc is a "standard" file that's also used by other applications, not ORT alone.

Hi
Exactly the same came to my mind last night :)
I copied to $HOME/.netrc and that worked !!!
Thank you Sebastien

And by the way, sorry for not getting back on this, but it works for me too :-)