jobTemplateReference option is not working
crolopez opened this issue · 9 comments
Hello,
I am trying to get my tests to create jobs based on custom templates but I must be doing something wrong following this article:
https://docs.testkube.io/articles/templates/
To do so, I modified a functional test to include the reference to the template (jobTemplateReference)
apiVersion: tests.testkube.io/v3
kind: Test
metadata:
creationTimestamp: "2024-04-15T05:44:53Z"
generation: 9
labels:
executor: maven-jdk11-executor
test-type: maven-test-jdk11
name: crlopez-test
namespace: testkube
resourceVersion: "192119998"
uid: 23423-4f9b-4b80-b7ea-d853253bbf84c5
spec:
content:
repository:
branch: master
tokenSecret:
key: git-token
name: crlopez-test-secrets
type: git
uri: xxxxxxxx
usernameSecret:
key: git-username
name: crlopez-test-secrets
type: git
executionRequest:
jobTemplateReference: crlopez-job-template
type: maven/test-jdk11
status:
latestExecution:
endTime: "2024-04-15T08:44:04Z"
id: 661ce86aa3eabe185f4d66f7
number: 11
startTime: "2024-04-15T08:42:18Z"
status: passedAnd this is the template:
apiVersion: tests.testkube.io/v1
kind: Template
metadata:
annotations:
creationTimestamp: "2024-04-30T06:52:03Z"
generation: 6
name: crlopez-job-template
namespace: testkube
resourceVersion: "204293257"
uid: f5cae436-0e65-46bf-8fab-523532
spec:
body: |-
apiVersion: batch/v1
kind: Job
metadata:
name: "{{ .Name }}"
nameeeeee: "TEEEEEST"
namespace: {{ .Namespace }}
spec:
{{- if gt .ActiveDeadlineSeconds 0 }}
activeDeadlineSeconds: {{ .ActiveDeadlineSeconds }}
{{- end }}
template:
spec:
initContainers:
- name: {{ .Name }}-init
{{- if .Registry }}
image: {{ .Registry }}/{{ .InitImage }}eeeeeeeeeee
{{- else }}
image: {{ .InitImage }}aaaaaa
{{- end }}
imagePullPolicy: IfNotPresent
command:
- "echo 'TEMPLATE MODIFICATION TEST' && /bin/runner"
- '{{ .Jsn }}'
volumeMounts:
- name: data-volume
mountPath: /data
{{- if .CertificateSecret }}
- name: {{ .CertificateSecret }}
mountPath: /etc/certs
{{- end }}
{{- if .ArtifactRequest }}
{{- if and .ArtifactRequest.VolumeMountPath (or .ArtifactRequest.StorageClassName .ArtifactRequest.UseDefaultStorageClassName) }}
- name: artifact-volume
mountPath: {{ .ArtifactRequest.VolumeMountPath }}
{{- end }}
{{- end }}
{{- range $configmap := .EnvConfigMaps }}
{{- if and $configmap.Mount $configmap.Reference }}
- name: {{ $configmap.Reference.Name }}
mountPath: {{ $configmap.MountPath }}
{{- end }}
{{- end }}
{{- range $secret := .EnvSecrets }}
{{- if and $secret.Mount $secret.Reference }}
- name: {{ $secret.Reference.Name }}
mountPath: {{ $secret.MountPath }}
{{- end }}
{{- end }}
containers:
{{ if .Features.LogsV2 -}}
- name: "{{ .Name }}-logs"
{{- if .Registry }}
image: {{ .Registry }}/{{ .LogSidecarImage }}
{{- else }}
image: {{ .LogSidecarImage }}
{{- end }}
imagePullPolicy: IfNotPresent
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: DEBUG
value: {{ if .Debug }}"true"{{ else }}"false"{{ end }}
- name: NAMESPACE
value: {{ .Namespace }}
- name: NATS_URI
value: {{ .NatsUri }}
- name: GROUP
value: test
- name: SOURCE
value: "job-pod:{{ .Name }}"
- name: ID
value: "{{ .Name }}"
{{- end }}
- name: "{{ .Name }}"
{{- if .Registry }}
image: {{ .Registry }}/{{ .Image }}
{{- else }}
image: {{ .Image }}
{{- end }}
imagePullPolicy: IfNotPresent
{{- if gt (len .Command) 0 }}
command:
{{- range $cmd := .Command }}
- {{ $cmd -}}
{{- end }}
{{- end -}}
{{- if gt (len .Args) 0 }}
args:
{{- range $arg := .Args }}
- {{ $arg -}}
{{- end }}
{{- end }}
{{- if .WorkingDir }}
workingDir: {{ .WorkingDir }}
{{- end }}
volumeMounts:
- name: data-volume
mountPath: /data
{{- if .CertificateSecret }}
- name: {{ .CertificateSecret }}
mountPath: /etc/certs
{{- end }}
{{- if .AgentAPITLSSecret }}
- mountPath: /tmp/agent-cert
readOnly: true
name: {{ .AgentAPITLSSecret }}
{{- end }}
{{- if .ArtifactRequest }}
{{- if and .ArtifactRequest.VolumeMountPath (or .ArtifactRequest.StorageClassName .ArtifactRequest.UseDefaultStorageClassName) }}
- name: artifact-volume
mountPath: {{ .ArtifactRequest.VolumeMountPath }}
{{- end }}
{{- end }}
{{- range $configmap := .EnvConfigMaps }}
{{- if and $configmap.Mount $configmap.Reference }}
- name: {{ $configmap.Reference.Name }}
mountPath: {{ $configmap.MountPath }}
{{- end }}
{{- end }}
{{- range $secret := .EnvSecrets }}
{{- if and $secret.Mount $secret.Reference }}
- name: {{ $secret.Reference.Name }}
mountPath: {{ $secret.MountPath }}
{{- end }}
{{- end }}
volumes:
- name: data-volume
emptyDir: {}
{{- if .CertificateSecret }}
- name: {{ .CertificateSecret }}
secret:
secretName: {{ .CertificateSecret }}
{{- end }}
{{- if .AgentAPITLSSecret }}
- name: { { .AgentAPITLSSecret } }
secret:
secretName: {{ .AgentAPITLSSecret }}
{{- end }}
{{- if .ArtifactRequest }}
{{- if and .ArtifactRequest.VolumeMountPath (or .ArtifactRequest.StorageClassName .ArtifactRequest.UseDefaultStorageClassName) }}
- name: artifact-volume
persistentVolumeClaim:
claimName: {{ .Name }}-pvc
{{- end }}
{{- end }}
{{- range $configmap := .EnvConfigMaps }}
{{- if and $configmap.Mount $configmap.Reference }}
- name: {{ $configmap.Reference.Name }}
configmap:
name: {{ $configmap.Reference.Name }}
{{- end }}
{{- end }}
{{- range $secret := .EnvSecrets }}
{{- if and $secret.Mount $secret.Reference }}
- name: {{ $secret.Reference.Name }}
secret:
secretName: {{ $secret.Reference.Name }}
{{- end }}
{{- end }}
restartPolicy: Never
{{- if .ServiceAccountName }}
serviceAccountName: {{.ServiceAccountName }}
{{- end }}
{{- if gt (len .ImagePullSecrets) 0 }}
imagePullSecrets:
{{- range $secret := .ImagePullSecrets }}
- name: {{ $secret -}}
{{- end }}
{{- end }}
backoffLimit: 0
ttlSecondsAfterFinished: {{ .DelaySeconds }}
type: job
As you can see, the template is a modified version of https://github.com/kubeshop/testkube/blob/e63540ff46f0db352274326b5c6981f82a1de9ea/config/job-template.yml.
I can see that the template loads correctly. If I specify an incorrect name in the test I get an error indicating that the template does not exist.
The problem comes when I run the test and the job is created. It does not follow the format I am asking for, ignoring the template.
Any idea?
hey @crolopez at the first glance it looks ok. Let me rerproduce your case based on submitted input
starting test execution testkube/crlopez-test (error: api/POST-testkube.Execution returned error: api server problem: failed to execute test: execution failed: test execution failed: executing job spec template: template: job:83:23: executing "job" at <.Command>: can't evaluate field Command in type client.JobOptions, context: null;
resp error:an error on the server ("unknown") has prevented the request from succeeding (post services testkube-api-server:8088))
we use .Command in job-container-template.yaml, maven is a prebuilt executor
But it was used in order to have this error
apiVersion: tests.testkube.io/v3
kind: Test
metadata:
name: crlopez-test
namespace: testkube
spec:
content:
repository:
branch: main
type: git
uri: https://github.com/testkube/test
type: git
executionRequest:
jobTemplateReference: crlopez-job-template
type: maven/test
apiVersion: tests.testkube.io/v1
kind: Template
metadata:
name: crlopez-job-template
namespace: testkube
spec:
body: |-
apiVersion: batch/v1
kind: Job
metadata:
name: "{{ .Name }}"
nameeeeee: "TEEEEEST"
namespace: {{ .Namespace }}
spec:
{{- if gt .ActiveDeadlineSeconds 0 }}
activeDeadlineSeconds: {{ .ActiveDeadlineSeconds }}
{{- end }}
template:
spec:
initContainers:
- name: {{ .Name }}-init
{{- if .Registry }}
image: {{ .Registry }}/{{ .InitImage }}eeeeeeeeeee
{{- else }}
image: {{ .InitImage }}aaaaaa
{{- end }}
imagePullPolicy: IfNotPresent
command:
- "echo 'TEMPLATE MODIFICATION TEST' && /bin/runner"
- '{{ .Jsn }}'
volumeMounts:
- name: data-volume
mountPath: /data
{{- if .CertificateSecret }}
- name: {{ .CertificateSecret }}
mountPath: /etc/certs
{{- end }}
{{- if .ArtifactRequest }}
{{- if and .ArtifactRequest.VolumeMountPath (or .ArtifactRequest.StorageClassName .ArtifactRequest.UseDefaultStorageClassName) }}
- name: artifact-volume
mountPath: {{ .ArtifactRequest.VolumeMountPath }}
{{- end }}
{{- end }}
{{- range $configmap := .EnvConfigMaps }}
{{- if and $configmap.Mount $configmap.Reference }}
- name: {{ $configmap.Reference.Name }}
mountPath: {{ $configmap.MountPath }}
{{- end }}
{{- end }}
{{- range $secret := .EnvSecrets }}
{{- if and $secret.Mount $secret.Reference }}
- name: {{ $secret.Reference.Name }}
mountPath: {{ $secret.MountPath }}
{{- end }}
{{- end }}
containers:
{{ if .Features.LogsV2 -}}
- name: "{{ .Name }}-logs"
{{- if .Registry }}
image: {{ .Registry }}/{{ .LogSidecarImage }}
{{- else }}
image: {{ .LogSidecarImage }}
{{- end }}
imagePullPolicy: IfNotPresent
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: DEBUG
value: {{ if .Debug }}"true"{{ else }}"false"{{ end }}
- name: NAMESPACE
value: {{ .Namespace }}
- name: NATS_URI
value: {{ .NatsUri }}
- name: GROUP
value: test
- name: SOURCE
value: "job-pod:{{ .Name }}"
- name: ID
value: "{{ .Name }}"
{{- end }}
- name: "{{ .Name }}"
{{- if .Registry }}
image: {{ .Registry }}/{{ .Image }}
{{- else }}
image: {{ .Image }}
{{- end }}
imagePullPolicy: IfNotPresent
{{- if gt (len .Command) 0 }}
command:
{{- range $cmd := .Command }}
- {{ $cmd -}}
{{- end }}
{{- end -}}
{{- if gt (len .Args) 0 }}
args:
{{- range $arg := .Args }}
- {{ $arg -}}
{{- end }}
{{- end }}
{{- if .WorkingDir }}
workingDir: {{ .WorkingDir }}
{{- end }}
volumeMounts:
- name: data-volume
mountPath: /data
{{- if .CertificateSecret }}
- name: {{ .CertificateSecret }}
mountPath: /etc/certs
{{- end }}
{{- if .AgentAPITLSSecret }}
- mountPath: /tmp/agent-cert
readOnly: true
name: {{ .AgentAPITLSSecret }}
{{- end }}
{{- if .ArtifactRequest }}
{{- if and .ArtifactRequest.VolumeMountPath (or .ArtifactRequest.StorageClassName .ArtifactRequest.UseDefaultStorageClassName) }}
- name: artifact-volume
mountPath: {{ .ArtifactRequest.VolumeMountPath }}
{{- end }}
{{- end }}
{{- range $configmap := .EnvConfigMaps }}
{{- if and $configmap.Mount $configmap.Reference }}
- name: {{ $configmap.Reference.Name }}
mountPath: {{ $configmap.MountPath }}
{{- end }}
{{- end }}
{{- range $secret := .EnvSecrets }}
{{- if and $secret.Mount $secret.Reference }}
- name: {{ $secret.Reference.Name }}
mountPath: {{ $secret.MountPath }}
{{- end }}
{{- end }}
volumes:
- name: data-volume
emptyDir: {}
{{- if .CertificateSecret }}
- name: {{ .CertificateSecret }}
secret:
secretName: {{ .CertificateSecret }}
{{- end }}
{{- if .AgentAPITLSSecret }}
- name: { { .AgentAPITLSSecret } }
secret:
secretName: {{ .AgentAPITLSSecret }}
{{- end }}
{{- if .ArtifactRequest }}
{{- if and .ArtifactRequest.VolumeMountPath (or .ArtifactRequest.StorageClassName .ArtifactRequest.UseDefaultStorageClassName) }}
- name: artifact-volume
persistentVolumeClaim:
claimName: {{ .Name }}-pvc
{{- end }}
{{- end }}
{{- range $configmap := .EnvConfigMaps }}
{{- if and $configmap.Mount $configmap.Reference }}
- name: {{ $configmap.Reference.Name }}
configmap:
name: {{ $configmap.Reference.Name }}
{{- end }}
{{- end }}
{{- range $secret := .EnvSecrets }}
{{- if and $secret.Mount $secret.Reference }}
- name: {{ $secret.Reference.Name }}
secret:
secretName: {{ $secret.Reference.Name }}
{{- end }}
{{- end }}
restartPolicy: Never
{{- if .ServiceAccountName }}
serviceAccountName: {{.ServiceAccountName }}
{{- end }}
{{- if gt (len .ImagePullSecrets) 0 }}
imagePullSecrets:
{{- range $secret := .ImagePullSecrets }}
- name: {{ $secret -}}
{{- end }}
{{- end }}
backoffLimit: 0
ttlSecondsAfterFinished: {{ .DelaySeconds }}
type: job
Hi @vsukhin , first of all thanks for the help.
I am having trouble understanding what is going on here, or where I can see if there is some kind of error log in the job generation.
When I launch a test from the Testkube interface that loads the template I included above, the result I expect is that the job that appears follows the format of the template. This means that it includes the extra metadata field and even fails to execute the job because it does not find the wrong images that I indicated. However, the job that is created seems to follow the original template.
Is it possible that there is some hidden error that is causing my template not to load but the default one? How can I check it?
Regards.
For my example above, job is not even created, because job template contains fields we don't use for prebuilt executor.
You can always check testkube api server logs
At the end, you might just need to adjust default template supplying only added parrts https://docs.testkube.io/articles/creating-tests#changing-the-default-job-template-used-for-test-execution
we use default job template, when template type doesn't match executor type, like using job template type for container executor instead of container type
As @vsukhin said, I just had to change type: job to type: container (last line of my template) and everything started working.
Additionally, I also needed to remove the . AgentAPITLSSecret block.
My problem is solved since the template is now taken into account for creating the new job.
Thank you!