Unable to use aws-encryption-provider on bare metal vmware (Create /api/v1/namespaces/default/secrets fails)
Techn0logic opened this issue · 1 comments
Techn0logic commented
What happened:
When running aws-encryption-provider as a static pod generating secrets fails with
kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
Error from server (InternalError): Internal error occurred: rpc error: code = DeadlineExceeded desc = context deadline exceeded
What you expected to happen:
Expected the generated key to be encrypted with the help of aws-encryption-provider.
How to reproduce it (as minimally and precisely as possible):
encryption config /etc/kubernetes/enc-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- kms:
name: aws-encryption-provider
endpoint: unix:///var/run/kmsplugin/socket.sock
cachesize: 1000
timeout: 10s
- identity: {}
aws-encryption-provider pod /etc/kubernetes/manifests/aws-encryption-provider.yaml
apiVersion: v1
kind: Pod
metadata:
name: aws-encryption-provider
namespace: kube-system
spec:
containers:
- image: <repo>/encryption/kubernetes-sigs/aws-encryption-provider
imagePullPolicy: Never
name: aws-encryption-provider
command:
- /aws-encryption-provider
- -key=<arn-key>
- -region=<region>
- -listen=/var/run/kmsplugin/socket.sock
ports:
- containerPort: 8080
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 8080
volumeMounts:
- mountPath: /var/run/kmsplugin
name: var-run-kmsplugin
volumes:
- name: var-run-kmsplugin
hostPath:
path: /var/run/kmsplugin
type: DirectoryOrCreate
kube-apiserver.yaml in /etc/kubernets/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ""
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
- --authorization-mode=Node,RBAC
- --encryption-provider-config=/etc/kubernetes/enc-config.yaml
- --advertise-address=<host_ip>
- --allow-privileged=true
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
- --etcd-servers=https://127.0.0.1:2379
- --insecure-port=0
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
- --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
- --requestheader-allowed-names=front-proxy-client
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --requestheader-group-headers=X-Remote-Group
- --requestheader-username-headers=X-Remote-User
- --secure-port=6443
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-cluster-ip-range=10.96.0.0/12
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
image: k8s.gcr.io/kube-apiserver:v1.13.3
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: <host_ip>
path: /healthz
port: 6443
scheme: HTTPS
initialDelaySeconds: 15
timeoutSeconds: 15
name: kube-apiserver
resources:
requests:
cpu: 250m
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/ca-certificates
name: etc-ca-certificates
readOnly: true
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
- mountPath: /usr/local/share/ca-certificates
name: usr-local-share-ca-certificates
readOnly: true
- mountPath: /usr/share/ca-certificates
name: usr-share-ca-certificates
readOnly: true
- mountPath: /var/run/kmsplugin
name: kmsplugin
- mountPath: /etc/kubernetes
name: enc-config
hostNetwork: true
priorityClassName: system-cluster-critical
volumes:
- hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
name: ca-certs
- hostPath:
path: /etc/ca-certificates
type: DirectoryOrCreate
name: etc-ca-certificates
- hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
name: k8s-certs
- hostPath:
path: /usr/local/share/ca-certificates
type: DirectoryOrCreate
name: usr-local-share-ca-certificates
- hostPath:
path: /usr/share/ca-certificates
type: DirectoryOrCreate
name: usr-share-ca-certificates
- hostPath:
path: /var/run/kmsplugin
type: DirectoryOrCreate
name: kmsplugin
- hostPath:
path: /etc/kubernetes
name: enc-config
status: {}
Anything else we need to know?:
Everything seems to be running
kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
aws-encryption-provider-kube-master-1 1/1 Running 0 24m
aws-encryption-provider-kube-master-2 1/1 Running 0 24m
aws-encryption-provider-kube-master-3 1/1 Running 0 24m
coredns-86c58d9df4-54q46 1/1 Running 0 26m
coredns-86c58d9df4-g989w 1/1 Running 0 26m
etcd-kube-master-1 1/1 Running 0 25m
etcd-kube-master-2 1/1 Running 0 26m
etcd-kube-master-3 1/1 Running 0 26m
kube-apiserver-kube-master-1 1/1 Running 0 24m
kube-apiserver-kube-master-2 1/1 Running 0 24m
kube-apiserver-kube-master-3 1/1 Running 0 24m
kube-controller-manager-kube-master-1 1/1 Running 1 26m
kube-controller-manager-kube-master-2 1/1 Running 0 26m
kube-controller-manager-kube-master-3 1/1 Running 0 26m
kube-flannel-ds-amd64-2vcsq 1/1 Running 0 24m
kube-flannel-ds-amd64-df6zg 1/1 Running 0 24m
kube-flannel-ds-amd64-gjxbm 1/1 Running 0 25m
kube-flannel-ds-amd64-k6tg5 1/1 Running 0 24m
kube-flannel-ds-amd64-mnv2h 1/1 Running 0 24m
kube-flannel-ds-amd64-tttg5 1/1 Running 0 25m
kube-flannel-ds-amd64-xqttx 1/1 Running 0 25m
kube-proxy-48p5c 1/1 Running 0 24m
kube-proxy-8qgfm 1/1 Running 0 25m
kube-proxy-bsc7k 1/1 Running 0 24m
kube-proxy-gtth8 1/1 Running 0 24m
kube-proxy-mzbrt 1/1 Running 0 24m
kube-proxy-pm662 1/1 Running 0 26m
kube-proxy-wkj8s 1/1 Running 0 26m
kube-scheduler-kube-master-1 1/1 Running 1 26m
kube-scheduler-kube-master-2 1/1 Running 0 26m
kube-scheduler-kube-master-3 1/1 Running 0 26m
kube-controlplane logs
I1210 11:10:42.365060 1 leaderelection.go:205] attempting to acquire leader lease kube-system/kube-controller-manager...
E1210 11:12:31.605064 1 leaderelection.go:270] error retrieving resource lock kube-system/kube-controller-manager: Get https://kube.local:6443/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
kube-apiserver - when creating a secret
I1210 11:12:42.013655 1 storage_scheduling.go:100] all system priority classes are created successfully or already exist.
I1210 11:14:19.147213 1 trace.go:76] Trace[1824779100]: "Create /api/v1/namespaces/default/secrets" (started: 2019-12-10 11:14:16.143103286 +0000 UTC m=+103.169157206) (total time: 3.004069467s):
Trace[1824779100]: [3.004069467s] [3.003008862s] END
etcd logs
2019-12-10 11:10:41.933260 W | etcdserver: read-only range request "key:\"/registry/pods/kube-system/kube-apiserver-kube-master-3\" " with result "range_response_count:1 size:3247" took too long (151.983741ms) to execute
2019-12-10 11:10:47.831098 I | etcdserver/membership: added member 6c0b0a3075ed5024 [https://10.128.208.17:2380] to cluster f343a7a51e71fffe
2019-12-10 11:10:47.831266 I | rafthttp: starting peer 6c0b0a3075ed5024...
2019-12-10 11:10:47.831368 I | rafthttp: started HTTP pipelining with peer 6c0b0a3075ed5024
2019-12-10 11:10:47.832023 I | rafthttp: started peer 6c0b0a3075ed5024
2019-12-10 11:10:47.832117 I | rafthttp: added peer 6c0b0a3075ed5024
2019-12-10 11:10:47.832364 I | rafthttp: started streaming with peer 6c0b0a3075ed5024 (writer)
2019-12-10 11:10:47.832460 I | rafthttp: started streaming with peer 6c0b0a3075ed5024 (writer)
2019-12-10 11:10:47.832555 I | rafthttp: started streaming with peer 6c0b0a3075ed5024 (stream MsgApp v2 reader)
2019-12-10 11:10:47.832791 I | rafthttp: started streaming with peer 6c0b0a3075ed5024 (stream Message reader)
2019-12-10 11:10:50.490214 I | rafthttp: peer 6c0b0a3075ed5024 became active
2019-12-10 11:10:50.490504 I | rafthttp: established a TCP streaming connection with peer 6c0b0a3075ed5024 (stream Message reader)
2019-12-10 11:10:50.490992 I | rafthttp: established a TCP streaming connection with peer 6c0b0a3075ed5024 (stream MsgApp v2 reader)
2019-12-10 11:10:50.521595 I | rafthttp: established a TCP streaming connection with peer 6c0b0a3075ed5024 (stream MsgApp v2 writer)
2019-12-10 11:10:50.534045 I | rafthttp: established a TCP streaming connection with peer 6c0b0a3075ed5024 (stream Message writer)
2019-12-10 11:11:22.287619 W | etcdserver: read-only range request "key:\"/registry/minions\" range_end:\"/registry/miniont\" count_only:true " with result "range_response_count:0 size:7" took too long (138.207898ms) to execute
2019-12-10 11:22:22.485758 I | mvcc: store.index: compact 1716
2019-12-10 11:22:22.490529 I | mvcc: finished scheduled compaction at 1716 (took 3.99149ms)
2019-12-10 11:27:22.495454 I | mvcc: store.index: compact 2315
2019-12-10 11:27:22.497903 I | mvcc: finished scheduled compaction at 2315 (took 1.848932ms)
2019-12-10 11:28:35.411288 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 33.35803ms)
2019-12-10 11:28:35.411337 W | etcdserver: server is likely overloaded
2019-12-10 11:28:35.411345 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 33.422392ms)
2019-12-10 11:28:35.411349 W | etcdserver: server is likely overloaded
2019-12-10 11:31:35.939158 W | etcdserver: read-only range request "key:\"/registry/podsecuritypolicy\" range_end:\"/registry/podsecuritypolicz\" count_only:true " with result "range_response_count:0 size:5" took too long (150.832634ms) to execute
aws-encryption-provider logs
Passed healthceck: version:"v1beta1" runtime_name:"AWSKMS"
Environment:
kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
- Kubernetes version (use
kubectl version):
1.13.3 - Encryption provider plugin version:
0.0.1 - Cloud provider configuration:
bare metal vmware - OS (e.g:
cat /etc/os-release):
cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
Techn0logic commented
The issue was aws authentication not provisioned correctly.
Ensure aws-encryption-provider pod can auth with aws kms