Using git credentials in sidecar plugin
patoarvizu opened this issue · 28 comments
Checklist:
- I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
- I've included steps to reproduce the bug.
- I've pasted the output of
argocd version.
Describe the bug
I have a sidecar plugin for Tanka, that uses jsonnet-bundler (jb) to pull dependencies from private repos. I have the ConfigManagementPlugin manifest configured to have init run jb install, which is how jsonnet-bundler pulls its dependencies, and it calls git under the covers. I assume that because these calls to git are unknown to/unmanaged by Argo, the pre-configured credentials aren't available, but up until 2.2.5, I was able to have a workaround this issue by mounting a git-ask-pass.sh script via a ConfigMap, and setting GIT_USERNAME and GIT_PASSWORD as environment variables on the sidecar container. Once I tried to upgrade to 2.3.1, this stopped working. (Note: Argo was still able to clone the private repo that my Application was pointing to, but what stopped working was the jb update command that runs in the sidecar as part of the initialization.)
I see here that argocd-git-ask-pass was introduced, but I don't quite understand if this is something that should be available in custom sidecars, or how to use it. It would be good if I can understand why the upgrade to 2.3.1 broke my workaround, but I'd be satisfied with a path forward if that's all we can find.
To Reproduce
I'm installing via the Helm chart, here are the relevant values:
repoServer:
volumes:
- configMap:
name: tanka-plugin
name: tanka-plugin
- configMap:
name: git-creds-config
items:
- key: git-ask-pass.sh
path: git-ask-pass.sh
mode: 0755
name: git-creds-config
extraContainers:
- name: tanka
command: [/var/run/argocd/argocd-cmp-server]
image: grafana/tanka:0.19.0
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 500m
memory: 512Mi
securityContext:
runAsNonRoot: true
runAsUser: 999
volumeMounts:
- mountPath: /var/run/argocd
name: var-files
- mountPath: /home/argocd/cmp-server/plugins
name: plugins
- mountPath: /home/argocd/cmp-server/config/plugin.yaml
subPath: tanka-plugin.yaml
name: tanka-plugin
- mountPath: /tmp
name: tmp-dir
- mountPath: /usr/local/bin/git-ask-pass.sh
subPath: git-ask-pass.sh
name: git-creds-config
env:
- name: GIT_USERNAME
valueFrom:
secretKeyRef:
name: argocd-repo-creds-ghe
key: username
- name: GIT_PASSWORD
valueFrom:
secretKeyRef:
name: argocd-repo-creds-ghe
key: password
The corresponding ConfigMaps are as follows:
apiVersion: v1
kind: ConfigMap
metadata:
name: tanka-plugin
namespace: argocd
data:
tanka-plugin.yaml: |-
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
name: tanka-plugin
spec:
version: v1.0
init:
command: [sh, -c, 'jb install']
generate:
command: [sh, -c, 'tk show ${TK_ENV} --dangerous-allow-redirect ${EXTRA_ARGS}']
discover:
fileName: jsonnetfile.json
allowConcurrency: true
lockRepo: true
---
apiVersion: v1
kind: ConfigMap
metadata:
name: git-creds-config
namespace: argocd
data:
git-ask-pass.sh: |-
#!/bin/sh
case "$1" in
Username*) echo "${GIT_USERNAME}" ;;
Password*) echo "${GIT_PASSWORD}" ;;
esac
Expected behavior
- I would expect that the workaround above still works in 2.3.1 as it did in 2.2.5, leveraging git's
GIT_ASKPASSfunctionality. - If this new
argocd-git-ask-passis available for custom sidecar plugins, I'd like to understand how to use it.
Screenshots
Version
I'm not running the argocd command line, but I'm using Argo 2.3.1
Logs
Some additional information: I had a hunch and tried using a custom image (instead of the upstream grafana/tanka) that copied /usr/local/bin/argocd from quay.io/argoproj/argocd:v2.3.1. When I did this, I got the following error message:
... level=fatal msg="rpc error: code = NotFound desc = unknown nonce" error: unable to read askpass response from 'argocd' fatal: could not read Username...
Which sounds like there's some misconfiguration somewhere, I just don't know where.
One thing I do want to point out though is that if this is the way that sidecars have to be configured to be able to get git credentials, then I feel like this cancels out the benefit of using sidecar plugins, which is not having to create custom images. For example, in this case I wouldn't be able to use the upstream grafana/tanka image on a sidecar cleanly, and would have to create my own image that includes argocd. Or am I missing something?
I'm also seeing this issue after updating to v2.3+, any apps using our tanka plugin fail with the same error message reported by @patoarvizu. Watching for updates on #8389. Ty.
@crenshaw-dev Is #8389 the issue to track now for the resolution to this?
@patoarvizu that's the PR to track (just merged), #7995 is the issue.
I'm going to put up a PR to cherry-pick this onto the next 2.3 release.
Awesome, thank you!
@patoarvizu have you tested this for your sidecar plugin? After reviewing the code with @leoluz I'm not convinced this will work for sidecars.
@crenshaw-dev No, I haven't tested it. I was waiting for 2.3.4 to be formally released to give it a try.
Okay cool. Will test. We might want to make the longer-lived credentials opt-in, because it's a bit less secure, and many folks won't need git credentials during manifest generation. Will keep this thread up to date.
Reopening because I think this needs some more thought/work on our side.
- I'd rather not implement the fix for argocd-cm (i.e. non-sidecar plugins). We're deprecating argocd-cm plugins in 2.4 and strongly encouraging folks to move to sidecars due to much better security.
- I want to look more carefully at when, why, and how we communicate git creds to CMPs. Specifically I'd like to avoid communicating creds at all by default. Many plugins won't use git, and sending the creds only risks them being stolen. Making cred passing opt-in (by the CMP author) will ensure the author thinks about how to secure them.
I wanted to clarify @crenshaw-dev comment above a bit more. There is recent security related work done in CMP sidecar that detaches its file system from the one used by repo-server. This change is currently in master and will be available in the next release. That being said, if there is a specific file required by a plugin, it needs to be available in the Application's source repo. Credentials will be passed as environment variables available in the process that executes the plugin's command. We have a e2e test to validate private repo access in CMP sidecar and it passes. Maybe jb update isn't able to get the necessary credentials currently available. This needs further investigation.
@patoarvizu can you please provide the full Application manifest that you are applying to reproduce the issue? Is this easily reproducible locally?
I can create a reproduction repo for you, including the full Application manifest, but it may need to be forked/edited to make it private to validate the credentials work as expected. I should be able to create it for you in the next day or two.
@leoluz @crenshaw-dev As promised, here's the reproduction repo: https://github.com/patoarvizu/argo-sidecar-plugin-creds-repro
It may need some tweaks to get it to work since it depends on access to private repos, but I think you should be able to take it from here. I added instructions for how to reproduce locally using a k3d cluster if it makes it easier, but I believe the Helm setup + the sample Application manifest there should be enough for you.
Please let me know if there's anything else I can help you with!
Wow this is brilliant. Thanks, @patoarvizu! Will give it a try Monday.
Also a tanka user and just got bit by this, since argo overwrites GIT_ASKPASS in here.
I worked around it by manually exporting GIT_ASKPASS to my mounted helper within the init command. E.g:
export GIT_ASKPASS=/usr/local/bin/git-ask-pass.sh &&
jb install
Confirming I ran into this issue too, thanks to @patoarvizu I knew what version to downgrade to. I'm just now trying out ArgoCD for the first time and wanted to use kustomize+envsubst so I added a simple custom tool via configmaps etc.
On the latest versions, it cannot authenticate with Git to pull from the remote Kustomize repo. On 2.2.5, it works fine.
For what it's worth, I was able to upgrade to 2.2.8 without issues.
I've used Git's AskPass to inject credentials in a similar case.
Git askpass script in a configmap:
https://github.com/HariSekhon/Kubernetes-configs/blob/master/git-askpass.configmap.yaml
patch the ArgoCD repo server with this script and environment variables to use the above script and whatever standard k8s secret credentials you want:
@HariSekhon can you please confirm the ArgoCD version you are running?
Thank you.
@leoluz I did this on an ArgoCD 2.0 that I use in production to solve my Kustomize external bases not working.
I've used Git's AskPass to inject credentials in a similar case.
Git askpass script in a configmap:
https://github.com/HariSekhon/Kubernetes-configs/blob/master/git-askpass.configmap.yaml
patch the ArgoCD repo server with this script and environment variables to use the above script and whatever standard k8s secret credentials you want:
I'm doing something similar for the sidecar plugins. While it works, this methods only allows using a single credentials rather than reusing the pattern matching of the Credential Templates from argoCD (though I'm not sure if the older version works like that, I haven't tested it yet).
I'm doing something similar for the sidecar plugins. While it works, this methods only allows using a single credentials rather than reusing the pattern matching of the Credential Templates from argoCD (though I'm not sure if the older version works like that, I haven't tested it yet).
In the above simple example this is true, but actually the git askpass mechanism can pass in variables, which means you can add logic to the script to return different credentials based on different protocol, host and repo path:
Reading through this issue, I think this bug affects only plugins that use kind of git integration themselves, right? E.g. those which need access to Git after Argo CD has provided the set of manifests it rendered to the plugin.
@jannfis that is my understanding as well. I do not believe that plugins which do not use git (either directly or indirectly) are impacted.
Confirming I ran into this issue too, thanks to @patoarvizu I knew what version to downgrade to. I'm just now trying out ArgoCD for the first time and wanted to use kustomize+envsubst so I added a simple custom tool via configmaps etc.
On the latest versions, it cannot authenticate with Git to pull from the remote Kustomize repo. On 2.2.5, it works fine.
same with me. Works on 2.2.5 fine. @crenshaw-dev any chance we can get a fix in for the latest version?
This is an issue still and is blocking me from upgrading past 2.2.x version.
In 2.2 git-ask-pass.sh were trivial as the GIT_USERNAME and GIT_PASSWORD were exposed via the environment.
In 2.5 that isn't the case.
This is blocking our organization from being on the latest version and inturn not able to use new features (Argocd notifications)
This is definitely still top of my mind. I'll do what I can to get something ready before 2.6. The solution has to prioritize security and Argo CD admin control over credentials. I want to take time to get that right.