kudobuilder/kuttl

TestAssert is assumed to be a CRD in the cluster

iainsproat opened this issue · 6 comments

iainsproat commented

What happened:

This is possibly a bug, or possibly my complete misunderstanding of how TestAssert is meant to work!

I have a TestAssert definition in 01-assert.yaml file:

apiVersion: kuttl.dev/v1beta1
kind: TestAssert
timeout: 5
collectors:
  - type: command
    command: echo hello

kuttl reports the following:

case.go:366: retrieving API resource for kuttl.dev/v1beta1, Kind=TestAssert failed: the server could not find the requested resource

What you expected to happen:

TestAssert is recognised by kuttl and has the behaviour described in the documentation.

How to reproduce it (as minimally and precisely as possible):

See above specification and below environment details.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
Client Version: v1.26.1
Kustomize Version: v4.5.7
Server Version: v1.25.4
  • KUTTL version (use kubectl kuttl version):
KUTTL Version: version.Info{GitVersion:"0.15.0", GitCommit:"f6d64c9", BuildDate:"2023-01-03T21:32:35Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"darwin/arm64"}
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:
iblancasa commented

This is not a bug. Maybe it should be explained in a better way in the documentation.

When you use a filename xx-assert.yaml, KUTTL expects to match exactly the object from your file. So, with this content file:

apiVersion: kuttl.dev/v1beta1
kind: TestAssert
timeout: 5
collectors:
  - type: command
    command: echo hello

KUTTL will try to look for a TestAssert object in your Kubernetes cluster.

Since what you want is KUTTL to "execute" that file, you need to rename your file from 01-assert.yaml to something like 01-check-command.yaml for instance.

iainsproat commented

Thanks for responding @iblancasa

The documentation, quoted below (emphasis mine), would imply it should be in the xx-assert.yaml file:

The TestAssert object can be used to specify settings for a test step's assert and must be specified in the test step's assert YAML.

From searching GitHub for examples of usage, I see that TestAssert is commonly being used within xx-assert.yaml files which aligns with the documentation.

I unfortunately continue to receive the error message for every combination I've tried. I'll continue trying different combinations and report back if I find success.

iblancasa commented

Thanks for responding @iblancasa

The documentation, quoted below (emphasis mine), would imply it should be in the xx-assert.yaml file:

The TestAssert object can be used to specify settings for a test step's assert and must be specified in the test step's assert YAML.

Sorry. I misunderstood something from the first message.

I tried the file you provided and it works for me:

$ kuttl test check
2023/03/03 10:23:44 kutt-test config testdirs is overridden with args: [ check ]
=== RUN   kuttl
    harness.go:462: starting setup
    harness.go:252: running tests using configured kubeconfig.
    harness.go:275: Successful connection to cluster at: https://127.0.0.1:37213
    harness.go:360: running tests
    harness.go:73: going to run test suite with timeout of 30 seconds for each step
    harness.go:372: testsuite: check has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/mytest
=== PAUSE kuttl/harness/mytest
=== CONT  kuttl/harness/mytest
    logger.go:42: 10:23:44 | mytest | Creating namespace: kuttl-test-resolved-boa
    logger.go:42: 10:23:44 | mytest/1- | starting test step 1-
    logger.go:42: 10:23:44 | mytest/1- | test step completed 1-
    logger.go:42: 10:23:44 | mytest | mytest events from ns kuttl-test-resolved-boa:
    logger.go:42: 10:23:44 | mytest | Deleting namespace: kuttl-test-resolved-boa
=== CONT  kuttl
    harness.go:405: run tests finished
    harness.go:513: cleaning up
    harness.go:570: removing temp folder: ""
--- PASS: kuttl (5.15s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/mytest (5.12s)
PASS

$ kuttl version
KUTTL Version: version.Info{GitVersion:"0.15.0", GitCommit:"f6d64c9", BuildDate:"2023-01-03T21:32:35Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.26.1
Kustomize Version: v4.5.7
Server Version: v1.25.3
  1. Just to verify: did you try to rename the file from 01-assert.yaml to 01-check.yaml (e.g.)?
  2. How your kuttl-test.yaml file looks like?
jaypipes commented

Thanks for responding @iblancasa
The documentation, quoted below (emphasis mine), would imply it should be in the xx-assert.yaml file:

The TestAssert object can be used to specify settings for a test step's assert and must be specified in the test step's assert YAML.

Sorry. I misunderstood something from the first message.

I tried the file you provided and it works for me:

$ kuttl test check
2023/03/03 10:23:44 kutt-test config testdirs is overridden with args: [ check ]
=== RUN   kuttl
    harness.go:462: starting setup
    harness.go:252: running tests using configured kubeconfig.
    harness.go:275: Successful connection to cluster at: https://127.0.0.1:37213
    harness.go:360: running tests
    harness.go:73: going to run test suite with timeout of 30 seconds for each step
    harness.go:372: testsuite: check has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/mytest
=== PAUSE kuttl/harness/mytest
=== CONT  kuttl/harness/mytest
    logger.go:42: 10:23:44 | mytest | Creating namespace: kuttl-test-resolved-boa
    logger.go:42: 10:23:44 | mytest/1- | starting test step 1-
    logger.go:42: 10:23:44 | mytest/1- | test step completed 1-
    logger.go:42: 10:23:44 | mytest | mytest events from ns kuttl-test-resolved-boa:
    logger.go:42: 10:23:44 | mytest | Deleting namespace: kuttl-test-resolved-boa
=== CONT  kuttl
    harness.go:405: run tests finished
    harness.go:513: cleaning up
    harness.go:570: removing temp folder: ""
--- PASS: kuttl (5.15s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/mytest (5.12s)
PASS

$ kuttl version
KUTTL Version: version.Info{GitVersion:"0.15.0", GitCommit:"f6d64c9", BuildDate:"2023-01-03T21:32:35Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.26.1
Kustomize Version: v4.5.7
Server Version: v1.25.3
1. Just to verify: did you try to rename the file from `01-assert.yaml` to `01-check.yaml` (e.g.)?

2. How your `kuttl-test.yaml` file looks like?

@iblancasa I'm running into the exact same problem. You are executing kubectl kuttl test against a Kube cluster that already has the kuttl.dev/v1beta1 CRDs installed into it. But when those CRDs are not installed in the target Kube cluster, you get the problem that @iainsproat wrote about.

And by default, kuttl.dev's CRDs are not installed into a Kubernetes cluster under test. I think that there should be a Helm chart that installs these kuttl.dev CRDs along with the kuttl krew plugin otherwise users are not able to use the TestStep/TestAssert/TestSuite CRs.

  • 👍1
porridge commented

I tried to reproduce this, unsuccessfully:

[mowsiany@mowsiany x]$ find
.
./tt
./tt/01-assert.yaml
[mowsiany@mowsiany x]$ cat tt/01-assert.yaml 
apiVersion: kuttl.dev/v1beta1
kind: TestAssert
timeout: 5
collectors:
  - type: command
    command: echo hello
[mowsiany@mowsiany x]$ KUBECONFIG=$HOME/.../kubeconfig.txt .../kubectl-kuttl test .
2023/08/09 09:00:41 running without a 'kuttl-test.yaml' configuration
2023/08/09 09:00:41 kutt-test config testdirs is overridden with args: [ . ]
=== RUN   kuttl
    harness.go:462: starting setup
    harness.go:252: running tests using configured kubeconfig.
    harness.go:275: Successful connection to cluster at: https://<censored>
    harness.go:360: running tests
    harness.go:73: going to run test suite with timeout of 30 seconds for each step
    harness.go:372: testsuite: . has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/tt
=== PAUSE kuttl/harness/tt
=== CONT  kuttl/harness/tt
    logger.go:42: 09:00:44 | tt | Creating namespace: kuttl-test-prime-jawfish
    logger.go:42: 09:00:45 | tt/1- | starting test step 1-
    logger.go:42: 09:00:46 | tt/1- | test step completed 1-
    logger.go:42: 09:00:46 | tt | tt events from ns kuttl-test-prime-jawfish:
    logger.go:42: 09:00:46 | tt | Deleting namespace: kuttl-test-prime-jawfish
=== NAME  kuttl
    harness.go:405: run tests finished
    harness.go:513: cleaning up
    harness.go:570: removing temp folder: ""
--- PASS: kuttl (11.14s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/tt (7.18s)
PASS
[mowsiany@mowsiany x]$

This cluster definitely does not have the kuttl CRDs installed. This should not be necessary.

FTR, kuttl is supposed to filter out the TestAssert objects from the list of objects it will try to match against. Apparently for some reason this condition is not kicking in, and as a result the TestAssert is treated like any other k8s resource and kuttl tries to find it on the server, which fails.

I suspect something is broken in the way kuttl loads objects from the yaml file.
Whoever can reproduce this ( @jaypipes or @iainsproat ) , please provide the full output from kuttl. I wonder if you get any "object detected with no GVK Kind for path" messages there.

iblancasa commented

You are executing kubectl kuttl test against a Kube cluster that already has the kuttl.dev/v1beta1 CRDs installed into it. But when those CRDs are not installed in the target Kube cluster, you get the problem that @iainsproat wrote about.

No. I'm using a newly created kind cluster. Without installing anything.

And by default, kuttl.dev's CRDs are not installed into a Kubernetes cluster under test. I think that there should be a Helm chart that installs these kuttl.dev CRDs along with the kuttl krew plugin otherwise users are not able to use the TestStep/TestAssert/TestSuite CRs.

As I pointed out, it is not needed.

Can you provide a reproducer for your problem?