Can't deploy Kubernetes Deployments with more than one container
Closed this issue · 6 comments
Hi.
I write because I've recently been testing the deployment of our apps with one of the latest Samson versions, v2385 (coming from v1556), and I've run into a problem when triyng to deploy our Kubernetes hosted apps.
Our apps are typically deployed along 2 other containers in the same k8s Deployment (app + nginx + heka) and the issue I'm having is that when I deploy an app with Samson, the container images that are not the main app (in this case nginx and heka) are replaced with the app's one. For this to be better understood, I show below the original YAML that resides in the repo, and the one that is modified and fed to kubernetes by Samson (actually it's the one retrieved from k8s via kubectl get deploy).
Original (in repo):
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: app-web
labels:
project: app
role: web
spec:
replicas: 1
strategy:
type: "RollingUpdate"
rollingUpdate:
maxUnavailable: 0
selector:
matchLabels:
project: app
role: web
template:
metadata:
labels:
project: app
role: web
spec:
terminationGracePeriodSeconds: 60
containers:
- name: app
image: <hidden-url>/app:latest
lifecycle:
postStart:
exec:
command: ["cp", "-a", "/app/public/", "/nginx-pd/"]
env:
- name: <HIDDEN>
valueFrom:
configMapKeyRef:
name: <hidden>
key: <hidden>
volumeMounts:
- mountPath: /nginx-pd
name: nginx-volume
readOnly: false
- name: nginx
image: <hidden-url>/nginx:v0.12
readinessProbe:
initialDelaySeconds: 5
httpGet:
path: /health
port: 80
livenessProbe:
initialDelaySeconds: 5
httpGet:
path: /health
port: 80
lifecycle:
preStop:
exec:
command: ["nginx", "-s", "quit"]
ports:
- containerPort: 80
volumeMounts:
- mountPath: /nginx-pd
name: nginx-volume
readOnly: true
resources:
requests:
cpu: 50m
memory: 32Mi
- name: heka
image: <hidden-url>/heka:v1.20
env:
- name: <HIDDEN>
valueFrom:
configMapKeyRef:
name: <hidden>
key: <hidden>
- name: APP_NAME
value: "app"
volumeMounts:
resources:
requests:
cpu: 50m
memory: 64Mi
volumes:
- name: nginx-volume
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: app-web
labels:
project: app
role: web
spec:
type: NodePort
ports:
- port: 80
name: http
protocol: TCP
selector:
project: app
role: webModified by samson (got via kubectl get deploy -o yaml)
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: '16'
creationTimestamp: '2017-12-19T22:29:30Z'
generation: 16
labels:
project: app
role: web
name: app-web
namespace: staging
resourceVersion: '50464924'
selfLink: "/apis/extensions/v1beta1/namespaces/staging/deployments/app-web"
uid: 14796b7b-e50c-11e7-a090-0ac24b9f0ab6
spec:
replicas: 1
revisionHistoryLimit: 1
selector:
matchLabels:
project: app
role: web
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
annotations:
deployer: <hidden>
owner: ''
samson/github-repo: <hidden>/app
creationTimestamp:
labels:
deploy_group: staging
deploy_group_id: '1'
deploy_id: '17'
project: app
project_id: '2'
release_id: '17'
revision: <hidden>
role: web
role_id: '2'
tag: master
spec:
containers:
- env:
- name: <HIDDEN>
valueFrom:
configMapKeyRef:
key: <hidden>
name: <hidden>
- name: REVISION
value: <hidden>
- name: TAG
value: master
- name: DEPLOY_ID
value: '17'
- name: DEPLOY_GROUP
value: staging
- name: PROJECT
value: app
- name: ROLE
value: web
- name: KUBERNETES_CLUSTER_NAME
value: <hidden>
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
image: <hidden-url>/app@sha256:<hidden-hash>
imagePullPolicy: IfNotPresent
lifecycle:
postStart:
exec:
command:
- cp
- "-a"
- "/app/public/"
- "/nginx-pd/"
preStop:
exec:
command:
- sleep
- '3'
name: app
resources:
limits:
cpu: 300m
memory: 512M
requests:
cpu: 150m
memory: 256M
terminationMessagePath: "/dev/termination-log"
terminationMessagePolicy: File
volumeMounts:
- mountPath: "/nginx-pd"
name: nginx-volume
- env:
- name: REVISION
value: <hidden>
- name: TAG
value: master
- name: DEPLOY_ID
value: '17'
- name: DEPLOY_GROUP
value: staging
- name: PROJECT
value: app
- name: ROLE
value: web
- name: KUBERNETES_CLUSTER_NAME
value: <hidden>
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
image: <hidden-url>/app@sha256:<hidden-hash>
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- nginx
- "-s"
- quit
livenessProbe:
failureThreshold: 3
httpGet:
path: "/health"
port: 80
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: nginx
ports:
- containerPort: 80
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: "/health"
port: 80
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: 50m
memory: 32Mi
terminationMessagePath: "/dev/termination-log"
terminationMessagePolicy: File
volumeMounts:
- mountPath: "/nginx-pd"
name: nginx-volume
readOnly: true
- env:
- name: <HIDDEN>
valueFrom:
configMapKeyRef:
key: <hidden>
name: <hidden>
- name: APP_NAME
value: app
- name: REVISION
value: <hidden>
- name: TAG
value: master
- name: DEPLOY_ID
value: '17'
- name: DEPLOY_GROUP
value: staging
- name: PROJECT
value: app
- name: ROLE
value: web
- name: KUBERNETES_CLUSTER_NAME
value: <hidden>
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
image: <hidden-url>/app@sha256:<hidden-hash>
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- sleep
- '3'
name: heka
resources:
requests:
cpu: 50m
memory: 64Mi
terminationMessagePath: "/dev/termination-log"
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 60
volumes:
- emptyDir: {}
name: nginx-volumeAs it can be seen, the images for heka and nginx are never used in the modified yaml (and the rest of the config, env vars, overriden commands, etc remain as in the original yaml) and the app one is used, causing errors for obvious reasons (and the app not to work).
I believe the problem resides somewhere along these lines, but I'm not sure if it's a bug or a desired behavior. This breaks all the deployments of our k8s apps that previously worked.
Thanks in advance.
PD: The questions I ask with this are: Are you aware this happens? This was a normal usage of k8s deployments with samson as per #966. Is there a workaround that you recommend? Or else, can you help me identify how to fix it? I'm happy to do the PR.
I think what you want is samson/dockerfile: none in the container definition to opt out of the replacement
https://github.com/zendesk/samson/blob/master/plugins/kubernetes/README.md#docker-images
@grosser I missed that in the readme! I will be testing that and I will get back with the results. Thanks!