Getting Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws) after upgrade to 1.15.0

Question

Getting Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws) after upgrade to 1.15.0

cbugneac-nex opened this issue 10 months ago · 12 comments

What happened:
After upgrade to 1.15.0 I started to get a lot of error messages:

$ kubectl -n kube-system logs aws-node-7k5lb
Defaulted container "aws-node" out of: aws-node, aws-eks-nodeagent, aws-vpc-cni-init (init)
Installed /host/opt/cni/bin/aws-cni
Installed /host/opt/cni/bin/egress-cni
time="2023-09-28T14:52:22Z" level=info msg="Starting IPAM daemon... "
time="2023-09-28T14:52:22Z" level=info msg="Checking for IPAM connectivity... "
E0928 14:52:24.675031      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
time="2023-09-28T14:52:24Z" level=info msg="Copying config file... "
time="2023-09-28T14:52:24Z" level=info msg="Successfully copied CNI plugin binary and config file."
E0928 14:52:25.481191      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 14:52:27.686591      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 14:52:32.446596      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 14:52:41.691178      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 14:52:56.185271      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 14:53:34.141211      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 14:54:07.818534      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 14:54:45.390576      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 14:55:30.741028      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 14:56:17.951797      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 14:56:57.626522      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 14:57:30.824984      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 14:58:08.047201      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 14:58:47.684812      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 14:59:25.239971      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:00:19.086564      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:01:00.100615      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:01:51.692482      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:02:48.166933      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:03:44.738870      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:04:16.841938      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:05:03.209249      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:05:37.478366      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:06:16.821894      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:07:07.152870      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:07:57.974485      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:08:45.497795      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:09:38.117638      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:10:36.382931      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E0928 15:11:23.286600      12 reflector.go:148] pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)

I'm not using Network Policies in EKS cluster.

Attach logs

What you expected to happen:
No error messages.

How to reproduce it (as minimally and precisely as possible):

Deploy EKS cluster v1.25
Deploy VPC CNI Add-on version v1.15.0-eksbuild.2 with custom networking.
Run kubectl -n kube-system logs aws-node-*****

Anything else we need to know?:
Nope.

Environment:

Kubernetes version (use kubectl version):
Client Version: v1.25.12
Server Version: v1.25.12-eks-2d98532
CNI Version: v1.15.0-eksbuild.2
OS (e.g: cat /etc/os-release): Bottlerocket OS 1.14.2 (aws-k8s-1.25)
Kernel (e.g. uname -a): Linux ip-10-151-8-251.eu-west-1.compute.internal 5.15.117 #1 SMP Thu Jul 6 03:52:07 UTC 2023 x86_64 GNU/Linux

Answer 1 · 2023-09-28T16:33:09.000Z

Yep, this is harmless and fixed by #2570. Will ship in v1.15.1 in mid-October

Answer 2 · 2023-10-08T09:23:43.000Z

what exactly time v1.15.1 release? I'm waiting for it to test.

Answer 3 · 2023-10-08T23:39:06.000Z

@jdn5126 Any idea when the v1.15.1 will be ready? Waiting for it too.

Answer 4 · 2023-10-09T02:36:29.000Z

The target is mid-October, so hoping to finish up release prep this week and make it available early next week

Answer 5 · 2023-10-09T03:23:23.000Z

The target is mid-October, so hoping to finish up release prep this week and make it available early next week

Thanks for the update @jdn5126.

Answer 6 · 2023-10-13T15:57:59.000Z

Closing now that v1.15.1 is released on GitHub

Answer 7 · 2023-10-13T15:58:17.000Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Answer 8 · 2023-10-24T18:27:43.000Z

@jdn5126 I'm still seeing this occur in v1.15.1-eksbuild.1 in one of my clusters:

kubectl logs -n kube-system aws-node-bwzv2
Defaulted container "aws-node" out of: aws-node, aws-eks-nodeagent, aws-vpc-cni-init (init)
Installed /host/opt/cni/bin/aws-cni
Installed /host/opt/cni/bin/egress-cni
time="2023-10-24T18:18:11Z" level=info msg="Starting IPAM daemon... "
time="2023-10-24T18:18:11Z" level=info msg="Checking for IPAM connectivity... "
E1024 18:18:12.922316      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
time="2023-10-24T18:18:13Z" level=info msg="Copying config file... "
time="2023-10-24T18:18:13Z" level=info msg="Successfully copied CNI plugin binary and config file."
E1024 18:18:14.240615      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1024 18:18:17.131308      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1024 18:18:22.002189      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1024 18:18:31.339359      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1024 18:18:54.020008      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1024 18:19:40.235436      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1024 18:20:15.197762      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1024 18:21:13.720003      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1024 18:22:10.592531      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1024 18:23:00.954364      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1024 18:23:36.662313      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1024 18:24:32.650883      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1024 18:25:17.224438      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1024 18:25:52.749505      11 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)

kubectl logs -n kube-system aws-node-6d99v -c aws-eks-nodeagent
{"level":"info","ts":"2023-10-24T20:30:27.245Z","caller":"runtime/asm_amd64.s:1598","msg":"version","GitVersion":"","GitCommit":"","BuildDate":""}
2023-10-24 20:30:27.248535848 +0000 UTC Logger.check error: failed to get caller
2023-10-24 20:30:27.248557739 +0000 UTC Logger.check error: failed to get caller
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x55f9d221f6a7]

goroutine 48 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:115 +0x1fa
panic({0x55f9d2e4f980, 0x55f9d423afc0})
        /root/sdk/go1.20.4/src/runtime/panic.go:884 +0x213
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).configureeBPFProbes(0xc00053a000, {0x55f9d31b4100, 0xc0007196b0}, {0xc000da13e0, 0x2c}, {0xc000df0640?, 0x1, 0xc00088d560?}, {0xc000dd5b80, 0x1, ...}, ...)
        /workspace/controllers/policyendpoints_controller.go:258 +0x3e7
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).reconcilePolicyEndpoint(0xc00053a000, {0x55f9d31b4100, 0xc0007196b0}, 0xc000deed00)
        /workspace/controllers/policyendpoints_controller.go:232 +0x7b1
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).reconcile(0xc00053a000, {0x55f9d31b4100, 0xc0007196b0}, {{{0xc0009a7a70, 0x6}, {0xc00088d560, 0x1d}}})
        /workspace/controllers/policyendpoints_controller.go:149 +0x24c
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).Reconcile(0xc00053a000, {0x55f9d31b4100, 0xc0007196b0}, {{{0xc0009a7a70, 0x6}, {0xc00088d560, 0x1d}}})
        /workspace/controllers/policyendpoints_controller.go:130 +0x11f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x55f9d31b4100?, {0x55f9d31b4100?, 0xc0007196b0?}, {{{0xc0009a7a70?, 0x55f9d2ce7000?}, {0xc00088d560?, 0x55f9d31a0ee8?}}})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:118 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0005a4000, {0x55f9d31b4058, 0xc00057f540}, {0x55f9d2f3b200?, 0xc0001542e0?})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:314 +0x377
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0005a4000, {0x55f9d31b4058, 0xc00057f540})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:265 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:226 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:222 +0x587

Bug: #2562

Answer 9 · 2023-10-25T07:23:37.000Z

@oxr463 I don't see that issue anymore on my end with v1.15.1. Try setting the network policy to false and give it a try. You may want to check on the config map amazon-vpc-cni's config enable-network-policy-controller and container aws-eks-nodeagent's args --enable-network-policy both set to false.

Answer 10 · 2023-10-25T10:53:46.000Z

We also keep seeing the error in the logs:

git:(master) k logs aws-node-2bcjr -n kube-system                                                                                                                                                                                                  

Defaulted container "aws-node" out of: aws-node, aws-eks-nodeagent, aws-vpc-cni-init (init)
Installed /host/opt/cni/bin/aws-cni
Installed /host/opt/cni/bin/egress-cni
time="2023-10-25T10:18:20Z" level=info msg="Starting IPAM daemon... "
time="2023-10-25T10:18:20Z" level=info msg="Checking for IPAM connectivity... "
E1025 10:18:22.469300      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
time="2023-10-25T10:18:23Z" level=info msg="Copying config file... "
time="2023-10-25T10:18:23Z" level=info msg="Successfully copied CNI plugin binary and config file."
E1025 10:18:23.974926      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:18:26.864920      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:18:30.829626      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:18:39.449934      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:18:59.236748      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:19:36.231684      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:20:08.902293      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:20:52.805911      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:21:26.119054      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:22:03.120822      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:22:47.070255      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:23:40.077920      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:24:21.297450      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:25:08.758548      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:26:05.408468      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:26:53.571851      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)
E1025 10:27:45.762904      12 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.CNINode: unknown (get cninodes.vpcresources.k8s.aws)

the version of vpc-cni plugin is amazon-k8s-cni:v1.15.1-eksbuild.1

You may want to check on the config map amazon-vpc-cni's config enable-network-policy-controller and container aws-eks-nodeagent's args --enable-network-policy both set to false

both set to false (default)

Answer 11 · 2023-10-25T14:02:29.000Z

@oxr463 I don't see that issue anymore on my end with v1.15.1. Try setting the network policy to false and give it a try. You may want to check on the config map amazon-vpc-cni's config enable-network-policy-controller and container aws-eks-nodeagent's args --enable-network-policy both set to false.

Yes, those were already set to false like in this comment, #2562 (comment)

Answer 12 · 2023-10-25T14:49:18.000Z

It sounds like the fix to prevent the Failed to watch *v1alpha1.CNINode log was not complete. Thank you for updating, I will look at why this is still logging when it is not being cached. Note that this log is harmless