Local deployment on KIND needs update
BrandonYifanLiu opened this issue · 8 comments
Describe the bug
When deploying MLX by following instructions here, hitting some errors in below format.
unable to recognize "STDIN": no matches for kind "XXXXX" in version "xxxx/xxxx"
To Reproduce
With following CLI versions.
brandonliu@Brandons-MacBook-Pro ~ % kind --version
kind version 0.12.0
brandonliu@Brandons-MacBook-Pro ~ % kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:30:48Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-03-06T21:39:59Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/arm64"}
brandonliu@Brandons-MacBook-Pro ~ % kustomize version
{Version:kustomize/v4.5.2 GitCommit:9091919699baf1c5a5bf71b32ca73a993e98088b BuildDate:2022-02-09T23:19:28Z GoOs:darwin GoArch:arm64}
Steps to reproduce the behavior:
- Go to this link and follow instructions.
- Running command
while ! kustomize build mlx-single-kind | \ kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done. - See error.
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "RoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "RoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "VirtualService" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "VirtualService" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "VirtualService" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "VirtualService" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
unable to recognize "STDIN": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
unable to recognize "STDIN": no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ValidatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ValidatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
Expected behavior
The command should return without error
Screenshots
If applicable, add screenshots to help explain your problem.
Environment
- OS: MacOS Monterey V12.3
- MLX Version [e.g. 22]
Additional context
Add any other context about the problem here.
Checked k8s doc, it should relate to API deprecation. Link here https://kubernetes.io/docs/reference/using-api/deprecation-guide/
Hi @BrandonYifanLiu thanks for reporting this. Did the deployment succeed eventually?
@Tomcli I think I have seen (some of) these error/warning messages before but the deployment seemed to succeed eventually
You had mentioned to
# run the below command two times if the CRDs take too long to provision
Is this connected?
As a workaround discussed today, I will downgrade the version to make deployment succeed locally and document the process.
Hi @BrandonYifanLiu thanks for reporting this. Did the deployment succeed eventually?
@Tomcli I think I have seen (some of) these error/warning messages before but the deployment seemed to succeed eventually
You had mentioned to
# run the below command two times if the CRDs take too long to provisionIs this connected?
Yes, the CRDs take some time to deploy, so we might need to run it twice if we deploy MLX on a new cluster.
As a workaround discussed today, I will downgrade the version to make deployment succeed locally and document the process.
Hi @BrandonYifanLiu -- could you create a PR to update the docs with the Kubernetes versions we need to use and how to do it with KIND?
@rafvasq -- I think this is the issue you encountered today? Could you verify the Kubernetes version you were using? i.e. Brandon had 1.23.4.
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", > GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:30:48Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/arm64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-03-06T21:39:59Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/arm64"}
Supported (required) Kubernetes should probably be no newer than 1.21.
Action items:
- Update KIND deployment doc:
- mention Kubernetes high version boundary in Create KIND Cluster paragraph
- describe and link to KIND docs how to create clusters for older Kubernetes versions
- K8s versions should align with
kindest/nodetags
- K8s versions should align with
- update the
kind create cluster --name mlxcode snippet to specify the K8s versionv1.21(kindest/node:v1.21.12)kind create cluster ... --image kindest/node:v1.21.12
- add Troubleshooting section with error message and pointer to use K8s version stated in prereqs
- Update Kubernetes deployment doc:
- mention Kubernetes high version boundary in prereqs
- add Troubleshooting section with error message and pointer to use K8s version stated in prereqs
- Update MLX deployment specs for Kubernetes
1.22and1.23
@ckadner, I've tested and continue to run into this issue with kubectl versions 1.20 and 1.21. Continuing to test but my latest attempt used the following:
-
kind version
0.14.0
-
kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.15", GitCommit:"8f1e5bf0b9729a899b8df86249b56e2c74aebc55", GitTreeState:"clean", BuildDate:"2022-01-19T17:27:39Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"darwin/amd64"}
-
kustomize version
Version: {KustomizeVersion:3.2.0 GitCommit:a3103f1e62ddb5b696daa3fd359bb6f2e8333b49 BuildDate:2019-09-18T16:26:36Z GoOs:darwin GoArch:amd64}
@rafvasq -- I just merged the PR from @kiranp2396 so the kind create cluster command should work as expected now.