Cannot apply CustomResourceDefinitions from namespaced install eventhough RBAC allows it
wilmardo opened this issue ยท 13 comments
Describe the bug
We are using ArgoCD in a managed cluster where we have rights within our namespaces. So we use the namespaced ArgoCD install with a cluster with namespaces defined:
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: cluster-kubernetes.default.svc
namespace: argocd
annotations:
managed-by: argocd.argoproj.io
labels:
argocd.argoproj.io/secret-type: cluster
stringData:
config: '{"tlsClientConfig":{"insecure":false}}'
name: iks-cluster
namespaces: argocd,playground,tools
server: https://kubernetes.default.svc
We also have the right to create CRDs in the clusterscope, but just CRDs nothing else. This gives us the ability to install applications within our namespaces that define CRDs, like ArgoCD or Gloo.
This gives us an issue with Argo, in the namespaced setup it errors when applying CRDs:
Cluster level ClusterIssuer "letsencrypt-production" can not be managed when in namespaced mode
When I set namespaces: *
Argo cannot connect to the cluster anymore since RBAC doesn't allow clusterresources other then the CRDs:
failed to sync cluster https://10.43.0.1:443: failed to load initial state of resource Deployment.apps: deployments.apps is forbidden: User "system:serviceaccount:argocd:argocd-application-controller" cannot list resource "deployments" in API group "apps" in the namespace "*
I traced the change back to this PR:
argoproj/gitops-engine#143
It shipped with ArgoCD 1.7.7, we are now pinned to 1.7.6. In that version it applies the CRDs fine, it just has no diff and is permanently OutOfSync
To Reproduce
- Setup ArgoCD as a namespaced install with a namespace set for the cluster.
- Create the RBAC underneath to give ArgoCD permission to apply CRDs at the clusterscope.
- Apply an CRD, I included a
ClusterIssuer
as an example.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
rbac.authorization.k8s.io/aggregate-to-admin: "true"
name: argocd:admin
rules:
- apiGroups:
- ""
resources:
- secrets
- configmaps
verbs:
- get
- list
- watch
- apiGroups:
- argoproj.io
resources:
- applications
- appprojects
verbs:
- create
- get
- list
- watch
- update
- patch
- delete
- apiGroups:
- ""
resources:
- events
verbs:
- create
- list
---
apiVersion: cert-manager.io/v1alpha3
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
email: info@example.org
privateKeySecretRef:
name: le-account-key
server: https://acme-staging-v02.api.letsencrypt.org/directory
solvers:
- http01:
ingress:
class: nginx
Expected behavior
ArgoCD to apply the CRD and have a Healthy
status.
Version
argocd: v1.4.2+48cced9
BuildDate: 2020-01-24T01:04:04Z
GitCommit: 48cced9d925b5bc94f6aa9fa4a8a19b2a59e128a
GitTreeState: clean
GoVersion: go1.12.6
Compiler: gc
Platform: linux/amd64
argocd-server: v1.7.4+f8cbd6b
BuildDate: 2020-09-05T02:45:44Z
GitCommit: f8cbd6bf432327cc3b0f70d23b66511bb906a178
GitTreeState: clean
GoVersion: go1.14.1
Compiler: gc
Platform: linux/amd64
Ksonnet Version: v0.13.1
Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
Kubectl Version: v1.17.8
As of v2.1 it is possible to enable cluster level resources in a namespaced mode using:
argocd cluster add <context> --cluster-resources=true --namespace my-namespace
I just gave it a try but that feature doesn't cover this issue. It tries to add a new cluster level service-account:
# argocd cluster add staging-1 --cluster-resources=true --namespace test
WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `imax-staging-1` with full cluster level admin privileges. Do you want to continue [y/N]? y
FATA[0002] Failed to create service account "argocd-manager" in namespace "kube-system": serviceaccounts is forbidden: User "u-gglgl5os6k" cannot create resource "serviceaccounts" in API group "" in the namespace "kube-system"
Which doesn't work and also isn't needed. The current serviceAccount for ArgoCD has the RBAC rights to adds some clusterlevel resources. It currently still complains with the same error:
Cluster level ClusterIssuer "letsencrypt-staging" can not be managed when in namespaced mode
Which isn't true, if Argo is enabled (with RBAC) to manage the CRDs.
I'm getting the same issue....
ComparisonError: Cluster level Namespace "elastic-test" can not be managed when in namespaced mode
ServiceAccount has cluster-wide policy. this whole thing worked fine until the auto-update on my argocd cluster/operator that occurred today.
Ugh, i just checked on my other "prod" clusters for a different customer... same issue. broke everything.
Ugh, i just checked on my other "prod" clusters for a different customer... same issue. broke everything.
are you done that?
Add a label to the namespace your application is deployed in so that the Argo CD instance in the openshift-gitops namespace can manage it:
$ oc label namespace spring-petclinic argocd.argoproj.io/managed-by=openshift-gitops
it was described here
ServiceAccount has cluster-wide policy. this whole thing worked fine until the auto-update on my argocd cluster/operator that occurred today.
We are experiencing the sames issues, after the auto-update of ArgoCd everything is messed up, apparently ArgoCD cannot manage the namespaces anymore.
Namespace "yyyy" for ConfigMap "jenkins-rrrrr" is not managed
Did you find a solution for this? We are running ArgoCD v2.1.6 in Openshift 4.6
Hello, Any solution for this issue: Namespace "yyyy" for "resources" "jenkins-rrrrr" is not managed
We are using the ArgoCD Operator (in Openshift). I'm not sure, but it could be that the operator reconciles the default secret.
We solved it by adding
config:
env:
- name: ARGOCD_CLUSTER_CONFIG_NAMESPACES
value: argocd
the following env variable to the Subscription of the Operator (where the value is the namespace of the argocd installation)
We are able to fix this issue by adding the below key/value in the
argocd-default-cluster-config
secret.clusterResources: true
We are running ArgoCD v2.1.6 in Openshift 4.7.30
Thanks a lot for the tip @gkmahendran09, really saved our day.
We have been struggling with this issue for some time now, and what it boiled down to in our case, was actually two parameters.
1 ) The one you are mentioning here, setting the "clusterResources: true" in the argocd-cluster secret. This, to my understanding, "tells" argocd that it's allowed to manage resources outside its own namespace.
However, doing so, we were faced with a new error, telling us,
"message\":\"Namespace \\\"foo\\\" for LimitRange \\\"bar\\\" is not managed\",\"type\":\"ComparisonError\"
which is becuase its still configured via the operator, to only "manage" resources within the namespace it recides in. So even if its allowed via the "clusterResources: true" parameter, it doesn't have any knowledge about the resources outside its own namespace.
2 ) To give argocd this knowledge, you have to provide the operator with the environment variable, as stated by user Cuball0 above. Ie., edit the subscription for the operator like
$ > oc get subscriptions.operators.coreos.com argocd-operator
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: argocd
namespace: argocd
spec:
channel: alpha
name: argocd-operator
source: operatorhubio-catalog
sourceNamespace: olm
config:
env:
- name: ARGOCD_CLUSTER_CONFIG_NAMESPACES
value: <argocd-namespaces(s)>
where <argocd-namespaces(s)> should be replaced by a comma separated list of namespaces in which you have argocd-instances that should manage cluster-wide-resources (ie. it should not be a list of namespaces you want to manage).
I've had issues finding this in the documentaion, but thanks to you we are all good again with our "namespaced argocd which handles cluster-wide resources".
Hope this helps somebody else and please correct me if I'm wrong about something.
๐ ๐ ๐ฏ ๐ ๐ฅ ๐ฐ
We are able to fix this issue by adding the below key/value in the
argocd-default-cluster-config
secret.clusterResources: true
Hm, I noticed today that this secret actually gets deleted when setting the environment variable in the subscription.
So it seems that you only would need to update the subsciption to get desired functionality.
Just thought I should share this knowledge.
Note that argocd-default-cluster-config
is a feature of the Argo CD Operator.
If you're not using the Argo CD Operator but are using declarative cluster configs, use the clusterResource
key. I've opened a PR to document that key.
Thanks for saving my day as well since I ran into the same issue after standing up ArgoCD using Openshift GitOps operator. Just curious is there a parameter you can set to enable the cluster resources for the in-build cluster while bringing up a brand new operator for the first time? Or this is something you need to change afterwards?