Support .ssh/config for specifying keys if ssh-agent fails

Question

Support .ssh/config for specifying keys if ssh-agent fails

alexcrichton opened this issue 9 years ago · 42 comments

Update (2018-10-30)

There's a workaround below for those interested:

[net]
git-fetch-with-cli = true

Original description

When cloning an SSH repository the only currently supported method of authenticating is picking up a key through ssh-agent. This can fail, however, for example if it's just not running! Cargo should support parsing .ssh/config and/or otherwise having a reasonable fallback in trying to find public/private keys on the filesystem. Currently libssh2 does not support this, so an external library will be required.

As to the rationale for this issue, apparently when using CircleCI with a deploy key it will add this to ~/.gitconfig:

[url "git@github.com:"]
  insteadOf = https://github.com/

Which means that clones of the index will be rewritten to git@github.com (SSH) instead of HTTPS. The ssh-agent apparently also isn't running, so it relies on ~/.ssh/config to point SSH at the right keys, which Cargo isn't itself looking at.

Answer 1 · 2016-06-19T20:53:56.000Z

Is there a workaround for this for CircleCI? Running into this:

$ cargo build --release --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`
error: failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  [23/-1] username does not match previous request

Answer 2 · 2016-06-19T21:26:50.000Z

@travisofthenorth that specific bug should be fixed on nightly I believe, although it may still not work on nightly when all put together

Answer 3 · 2016-06-19T22:13:19.000Z

Thanks @alexcrichton. The nightly build does fix that issue. cargo build --verbose still seems to hang. I suppose that's what you were alluding to?

Answer 4 · 2016-06-19T22:14:17.000Z

Hm, that's odd that it hangs! It should either always make progress in one way or another or give an error message. Maybe there's just a slow download or slow network performance?

Answer 5 · 2016-06-19T22:34:53.000Z

Network performance seems average on CircleCI. Downloading and installing rust takes ~15s. How long would you expect a build to take, and what output? This is all I saw after a few minutes:

$ cargo build --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`

Answer 6 · 2016-06-20T03:57:38.000Z

Weird, is it possible to get a stack trace of what Cargo is doing at that time?

Answer 7 · 2016-06-20T14:47:12.000Z

Hmmm, am I doing something wrong here? I'm so puzzled, I'm getting no additional output:

$ RUST_BACKTRACE=1 cargo build --release --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`

Answer 8 · 2016-06-20T16:14:45.000Z

Oh I meant moreso attaching a debugger while it's running (RUST_BACKTRACE is only used on panics).

Another option may be to use RUST_LOG=cargo and see if that prints something useful

Answer 9 · 2016-06-20T17:44:07.000Z

Are there docs on doing this? I'm pretty new to Rust (I'm actually working in a Ruby app and a Rust lib is getting compiled in a gem).

FWIW:

$ RUST_LOG=cargo cargo build --release --verbose
DEBUG:cargo::build: executing; cmd=cargo-build; args=["cargo", "build", "--release", "--verbose"]
DEBUG:cargo::ops::cargo_compile: compile; manifest-path=/home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7/Cargo.toml
TRACE:cargo::ops::cargo_read_manifest: read_package; path=/home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7/Cargo.toml; source-id=file:///home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7
DEBUG:cargo::ops::cargo_compile: loaded package; package=faster_path v0.0.1 (file:///home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7)
DEBUG:cargo::core::registry: load/missing  file:///home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7
TRACE:cargo::core::source: loading SourceId; file:///home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7
TRACE:cargo::ops::cargo_read_manifest: read_package; path=/home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7/Cargo.toml; source-id=file:///home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7
TRACE:cargo::core::resolver: resolve; summary=faster_path v0.0.1 (file:///home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7)
TRACE:cargo::core::resolver: activating faster_path v0.0.1 (file:///home/ubuntu/myapp/vendor/bundle/ruby/2.3.0/gems/faster_path-0.1.7)
DEBUG:cargo::core::registry: load/missing  registry https://github.com/rust-lang/crates.io-index
TRACE:cargo::core::source: loading SourceId; registry https://github.com/rust-lang/crates.io-index
    Updating registry `https://github.com/rust-lang/crates.io-index`

Answer 10 · 2016-06-20T18:22:49.000Z

Probably not, but that looks like it's just a slow clone. So far it doesn't look like it's necessarily a Cargo problem so much as ensuring that a filesystem/network connection is looking good. If you can ssh into the builder (I think circleci allows that?) and attach a debugger to the Cargo process you can find out for sure, but it looks like libgit2's just chugging along cloning the index.

Answer 11 · 2016-06-20T19:58:48.000Z

attach a debugger to the Cargo process

^ that's what I meant when I was asking "are there docs on doing this". These snippets are actually from my ssh session on a Circle box trying to install manually. I let it sit for 30m-1hr and it never finished, so I'm skeptical that it's just git.

Answer 12 · 2016-06-20T20:27:37.000Z

Ah yeah, to do that you should just need gdb -p <pid-of-cargo> which you can learn through something like ps aux | grep cargo. You could also just do gdb --args cargo build I believe. Once you're in gdb and it's paused when Cargo is hanging you should just need bt all I think

Answer 13 · 2016-06-20T21:15:17.000Z

@alexcrichton I just ran rm .gitignore and it worked fine. So, I suppose it's a similar issue still present in the nightly?

Edit: I don't know if this is at all helpful, but here's the backtrace before removing the .gitignore file:

#0  0x00007fcef083878d in recv () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fcef1344909 in _libssh2_recv ()
#2  0x00007fcef1341e5f in agent_transact_unix ()
#3  0x00007fcef13424a5 in libssh2_agent_list_identities ()
#4  0x00007fcef131a898 in _git_ssh_authenticate_session ()
#5  0x00007fcef131b2b5 in _git_ssh_setup_conn ()
#6  0x00007fcef131998c in git_smart__connect ()
#7  0x00007fcef12f3095 in git_remote_connect ()
#8  0x00007fcef12f488c in git_remote_fetch ()
#9  0x00007fcef12b4826 in git2::remote::Remote::fetch::h60731caef39218e3 ()
#10 0x00007fcef1125799 in cargo::sources::git::utils::fetch::_$u7b$$u7b$closure$u7d$$u7d$::h2be50155b946ea5e ()
#11 0x00007fcef111e373 in cargo::sources::git::utils::fetch::h9819a463fcc3e7bd ()
#12 0x00007fcef1140397 in cargo::sources::registry::RegistrySource::do_update::he4e9bb63bd164d9b ()
#13 0x00007fcef0fa3d25 in _$LT$sources..registry..RegistrySource$LT$$u27$cfg$GT$$u20$as$u20$core..registry..Registry$GT$::query::h6f58831bb6ae841b ()
#14 0x00007fcef0ff52a4 in _$LT$core..registry..PackageRegistry$LT$$u27$cfg$GT$$u20$as$u20$core..registry..Registry$GT$::query::hd4c3f0cd3e3a40c6 ()
#15 0x00007fcef0fdd5f1 in _$LT$$RF$$u27$a$u20$mut$u20$I$u20$as$u20$std..iter..Iterator$GT$::next::he344e9055ac2d583 ()
#16 0x00007fcef0fd6c86 in cargo::core::resolver::Context::build_deps::h1981aabeeff3bf33 ()
#17 0x00007fcef0fcf495 in cargo::core::resolver::activate::h63b38b2f3bd7dc52 ()
#18 0x00007fcef0fc8e73 in cargo::core::resolver::activate_deps_loop::h373099c8772ce62b ()
#19 0x00007fcef0fc7375 in cargo::core::resolver::resolve::h20a2a85e670c0bae ()
#20 0x00007fcef10095ee in cargo::ops::resolve::resolve_with_previous::hd06ae7cd3726d0b0 ()
#21 0x00007fcef0ffb8c2 in cargo::ops::resolve::resolve_pkg::he4cb4924304ee511 ()
#22 0x00007fcef1005302 in cargo::ops::cargo_compile::resolve_dependencies::hc76d5aefb9b3e742 ()
#23 0x00007fcef1001c36 in cargo::ops::cargo_compile::compile_pkg::h48656ffa9b1541f3 ()
#24 0x00007fcef1000aea in cargo::ops::cargo_compile::compile::h1b43b20047c53d10 ()
#25 0x00007fcef0f29b0b in cargo::call_main_without_stdin::hec0ecdd41bac3cd3 ()
#26 0x00007fcef0f1ad2f in cargo::execute::hab7accd6bf9b5c64 ()
#27 0x00007fcef0f17574 in cargo::call_main_without_stdin::hf099bd36acb849a3 ()
#28 0x00007fcef0f144a2 in cargo::main::h2b219a79f378c1ef ()
#29 0x00007fcef14d9a99 in std::panicking::try::call::hc5e1f5b484ec7f0e ()
#30 0x00007fcef14e4aac in __rust_try ()
#31 0x00007fcef14e4a4f in __rust_maybe_catch_panic ()
#32 0x00007fcef14d94c4 in std::rt::lang_start::h61f4934e780b4dfc ()
#33 0x00007fcef026eec5 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#34 0x00007fcef0f13e29 in _start ()

Answer 14 · 2016-06-21T03:12:13.000Z

Oh this may be libssh2 going into the weeds which is being redirected via ~/.gitconfig and replaceWith (or some key named like that...)

Answer 15 · 2016-06-21T15:11:07.000Z

It seems like insteadOf is the only key defined on CircleCI, at least on the box I was using.

Answer 16 · 2016-11-25T22:13:48.000Z

Seems like this is actually an issue with Circle. I worked around it with the following:

checkout:
  post:
    - git config --global --unset url.ssh://git@github.com:.insteadof

But the configuration is actually broken. The value assigned to this variable isn't correct which makes anything that gets transformed by it into a broken url, eg. ssh://git@github.com:/rust-lang/cargo

Answer 17 · 2017-08-23T11:05:28.000Z

Is there a reason this is stalled? The current behaviour (ssh-agent being required) is very frustrating.

Answer 18 · 2018-02-05T17:23:32.000Z

Agreed with @wrl. I'm having the same exact issue too. I don't use ssh-agent.

Answer 19 · 2018-02-07T23:27:01.000Z

It turns out to be pretty nontrivial to emulate ssh's behavior. We'd need to write a parser for ~/.ssh/config for example.

I think a more promising route is to support fetches by shelling out to the git binary if it's available. We'd use git2 for everything else and still support git2-based fetches if git isn't present, but it seems better to just use git rather than trying to emulate its exact authentication behavior.

Answer 20 · 2018-06-07T09:03:31.000Z

I think a more promising route is to support fetches by shelling out to the git binary if it's available.

Oh, yes, please do so! 👍

I am trying to get Cargo to work with a private git repository in a Single-Sign-On environment, but got lost in the limitations of libgit2. Here {ssh,https}://mygitserver.domain.tld/myproject works locally without authentication on the git layer, but happens on the transport layer - configured in OpenSSH configuration or HTTP libraries like libcurl understanding how to perform (negotiate) authentication. (In my case I use Kerberos authentication.) This means that users have a single canonical URL and giving that to git clone "just works" from all machines. I'd like to have Cargo use this system configuration as well and by shelling out to local git binary this should "just work".

Answer 21 · 2018-10-30T17:09:05.000Z

As an update for those following this issue, Cargo now supports global .cargo/config configuration that looks like:

[net]
git-fetch-with-cli = true

which will instruct Cargo to fetch git repositories with the git CLI rather than with libgit2. Using this will read .ssh/config because that's what git does.

Answer 22 · 2018-11-02T10:18:05.000Z

@alexcrichton I fetch failed by the following error when using the git client

fatal: Refusing to fetch into current branch refs/heads/master of non-bare repository
error: failed to load source for a dependency on `cargo`

Caused by:
  Unable to update https://github.com/rust-lang/cargo

Caused by:
  failed to clone into: /Users/wangsijie/.cargo/git/db/cargo-e7ff1db891893a9e

Caused by:
  process didn't exit successfully: `git fetch --tags --quiet 'https://github.com/rust-lang/cargo' 'refs/heads/*:refs/heads/*'` (exit code: 128)

I just create a new crate dependent on cargo

[dependencies]
cargo = { git = "https://github.com/rust-lang/cargo" }

My rust version is

rustc 1.31.0-nightly (1cf82fd9c 2018-10-30)

Is there something wrong with my git config?

Answer 23 · 2018-11-02T14:06:35.000Z

@xinghun92 hm there's probably not much wrong with your git config! The error message there:

fatal: Refusing to fetch into current branch refs/heads/master of non-bare repository

is one I'm not so sure of! I'm pretty unfamiliar with the advanced aspects of the git CLI, but it looks like Cargo isn't executing quite the right git command here. According to this question it sounds like we need to pass the --update-head-ok option, I'll send a PR for that!

Answer 24 · 2019-06-15T04:51:13.000Z

@alexcrichton thank you! That works.

Answer 25 · 2019-06-20T03:33:48.000Z

I tried the workaround by adding a ~/.cargo/config file with the code given above, but now I'm getting an authentication error, specifically:

error: failed to load source for a dependency on `csv`

Caused by:
  Unable to update registry `https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  process didn't exit successfully: `git fetch --tags --force --update-head-ok 'https://github.com/rust-lang/crates.io-index' 'refs/heads/master:refs/remotes/origin/master'` (exit code: 128)
--- stderr
fatal: unable to access 'https://github.com/rust-lang/crates.io-index/': SSL certificate problem: Invalid certificate chain

I'm on a MacBook running El Capitan and I just updated everything with rustup update. I can see that --update-head-ok is being used in the git call. How do I make sure I have a valid certificate chain?

Answer 26 · 2019-06-22T18:40:53.000Z

@alexcrichton just to add to your rationale, that .gitconfig setup

[url "git@github.com:"]
  insteadOf = https://github.com/

is frequent among Go programmers that have to work with private repositories. The Go dependency system tries to clone repos using HTTPS by default, but if you use SSH to authenticate with your private repos you must use that trick on .gitconfig. So it's not only for the CircleCI case. I guess there must be other cases too.

Answer 27 · 2019-06-25T15:42:54.000Z

is frequent among Go programmers that have to work with private repositories

It's unfortunate that such things require global configuration changes rather than being localized to the affected repositories.

Answer 28 · 2019-07-05T13:43:55.000Z

As an update for those following this issue, Cargo now supports global .cargo/config configuration that looks like:
[net]
git-fetch-with-cli = true

Is there any way to do this on the command line?

cargo install --git-fetch-with-cli --git ssh://...

Answer 29 · 2019-07-05T14:19:53.000Z

You should be able to set it via an environment variable:

$ env CARGO_NET_GIT_FETCH_WITH_CLI=true cargo install --git ssh://...

Answer 30 · 2019-08-16T09:30:59.000Z

@sfackler Works like a charm, thanks!

Answer 31 · 2019-11-05T21:53:07.000Z

To followup on this issue, when I updated my .cargo/config file, I started getting the following error:

Updating crates.io index
error: failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  could not execute process `git fetch --tags --force --update-head-ok 'https://github.com/rust-lang/crates.io-index' 'refs/heads/master:refs/remotes/origin/master'` (never executed)

Caused by:
  No such file or directory (os error 2)

Note that I am trying to run this inside of a docker container. Anyone have ideas on what the issue maybe here?

Answer 32 · 2019-11-05T21:54:24.000Z

You need to install git in the docker container.

Answer 33 · 2020-01-02T23:39:41.000Z

I'm on macOS 10.15.2 (beta), with cargo 0.26.0 (41480f5cc 2018-02-26) and the workaround doesn't work for me. I've tried both a ~/.cargo/config, and the env var (CARGO_NET_GIT_FETCH_WITH_CLI=true) approach. Am I missing a step?

Answer 34 · 2020-05-13T10:57:05.000Z

The explanation on this ticket doesn't seem quite right: I do use ssh-agent (and my .ssh/config file is completely empty) and yet I still hit this error with cargo.

The workaround using git CLI does fix the problem.

Answer 35 · 2020-11-13T21:47:24.000Z

@Diggsey Are you on Windows and using the built in OpenSSH ssh-agent? If so, libssh2 doesn't support that ssh-agent, though there is work being done to remedy that.

Answer 36 · 2020-11-13T22:34:33.000Z

@yodaldevoid yes

Answer 37 · 2021-11-19T22:35:09.000Z

I am experiencing this just trying to go through the getting started guide and run cargo build after adding the ferris-says dep: https://www.rust-lang.org/learn/get-started

Why does it even need an SSH key at all to download this?

Answer 38 · 2021-11-20T16:11:29.000Z

Why does it even need an SSH key at all to download this?

This should only be an issue if specifying dependencies via git with an ssh URL so something seems wrong in your case anyway.

Answer 39 · 2022-05-31T12:31:56.000Z

Seems like this is actually an issue with Circle. I worked around it with the following:
checkout:
  post:
    - git config --global --unset url.ssh://git@github.com:.insteadof
But the configuration is actually broken. The value assigned to this variable isn't correct which makes anything that gets transformed by it into a broken url, eg. ssh://git@github.com:/rust-lang/cargo

first, this is very helpful, thank you!
however, as of today this snippet doesn't work on CircleCI. i believe that's for these two reasons:

checkout doesn't support post (anymore)
there's a typo in the given command.

here's what i find working today:

some-job:
      (...)
      steps:
         - checkout
         - run: git config --global --unset url.ssh://git@github.com.insteadof

Answer 40 · 2022-10-12T05:29:33.000Z

Please also support custom SSH agents. I use my password manager to handle my SSH keys and as such my SSH config looks like this:

Host *
	IdentityAgent "~/Library/Group Containers/2BUA8C4S2C.com.1password/t/agent.sock"

Host github.com
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_github.pub
    IdentitiesOnly yes

What's more, I use https to ssh rewriting to support private repos in go projects, and my global git config is:

[url "git@github.com:"]
	insteadOf = https://github.com/

This setup absolutely doesn't work with cargo.

Answer 41 · 2023-02-27T03:14:59.000Z

export CARGO_NET_GIT_FETCH_WITH_CLI=true

Answer 42 · 2023-06-12T23:49:36.000Z

Using ssh access without the ssh-agent running seems like a fairly unusual setup to be using intentionally; I ran into this issue because I just forgot to start up the agent on OS X. Perhaps the hint message pointing users to net_git_fetch_with_cli could also suggest starting up ssh-agent as an alternative?