After replacing the kubelet binary, the kubelet will automatically exit
lolo-pop opened this issue · 3 comments
I followed the readme and configured the environment until step 7. When I initialize the k8s-cluster, I got [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Checking the kubelet status reveals that the kubelet repeatedly starts and exits.
I used the original k8s-1.19 binaries instead of the binaries provided in the readme. Then, I found that it could initialize the k8s-cluster.
I need your help, thanks.
Thank you for your reply. According to init-cluster-containerd-CRIU.md, I reconfigured it and now the k8s cluster has been created.
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-586dd6b6d7-ptlnr 1/1 Running 0 4h41m
kube-system coredns-586dd6b6d7-qmzlf 1/1 Running 0 4h41m
kube-system etcd-dragonlab01 1/1 Running 0 4h41m
kube-system kube-apiserver-dragonlab01 1/1 Running 0 4h41m
kube-system kube-controller-manager-dragonlab01 1/1 Running 0 4h41m
kube-system kube-flannel-ds-gjk5x 1/1 Running 0 177m
kube-system kube-flannel-ds-rrhj8 1/1 Running 0 177m
kube-system kube-flannel-ds-s7hm5 1/1 Running 0 177m
kube-system kube-proxy-cr62w 1/1 Running 0 177m
kube-system kube-proxy-k4q7x 1/1 Running 0 4h12m
kube-system kube-proxy-kqmgv 1/1 Running 0 4h41m
kube-system kube-scheduler-dragonlab01 1/1 Running 0 4h41m
I'm trying to restore the example of migrating the video-stream application in the readme.md and I'm facing another error. When I am executing the command kubebuilder create api --group podmig --version v1 --kind Podmigration. I got the following error.
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing scaffold for you to edit...
api/v1/podmigration_types.go
controllers/podmigration_controller.go
Running make:
$ make
go: creating new go.mod: module tmp
go get: added sigs.k8s.io/controller-tools v0.2.5
/root/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Error: go [list -e -json -compiled=true -test=false -export=false -deps=true -find=false -tags ignore_autogenerated -- ./...]: exit status 1: go: example.com/m/controllers: package github.com/go-logr/logr imported from implicitly required module; to add missing requirements, run:
go get github.com/go-logr/logr@v0.1.0
Usage:
controller-gen [flags]
Examples:
# Generate RBAC manifests and crds for all types under apis/,
# outputting crds to /tmp/crds and everything else to stdout
controller-gen rbac:roleName=<role name> crd paths=./apis/... output:crd:dir=/tmp/crds output:stdout
# Generate deepcopy/runtime.Object implementations for a particular file
controller-gen object paths=./apis/v1beta1/some_types.go
# Generate OpenAPI v3 schemas for API packages and merge them into existing CRD manifests
controller-gen schemapatch:manifests=./manifests output:dir=./manifests paths=./pkg/apis/...
# Run all the generators for a given project
controller-gen paths=./apis/...
# Explain the markers for generating CRDs, and their arguments
controller-gen crd -ww
Flags:
-h, --detailed-help count print out more detailed help
(up to -hhh for the most detailed output, or -hhhh for json output)
--help print out usage and a summary of options
--version show version
-w, --which-markers count print out all markers available with the requested generators
(up to -www for the most detailed output, or -wwww for json output)
Options
generators
+webhook package generates (partial) {Mutating,Validating}WebhookConfiguration objects.
+schemapatch:manifests=<string>[,maxDescLen=<int>] package patches existing CRDs with new schemata.
+rbac:roleName=<string> package generates ClusterRole objects.
+object[:headerFile=<string>][,year=<string>] package generates code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations.
+crd[:crdVersions=<[]string>][,maxDescLen=<int>][,preserveUnknownFields=<bool>][,trivialVersions=<bool>] package generates CustomResourceDefinition objects.
generic
+paths=<[]string> package represents paths and go-style path patterns to use as package roots.
output rules (optionally as output:<generator>:...)
+output:artifacts[:code=<string>],config=<string> package outputs artifacts to different locations, depending on whether they're package-associated or not.
+output:dir=<string> package outputs each artifact to the given directory, regardless of if it's package-associated or not.
+output:none package skips outputting anything.
+output:stdout package outputs everything to standard-out, with no separation.
run `controller-gen object:headerFile=hack/boilerplate.go.txt paths=./... -w` to see all available markers, or `controller-gen object:headerFile=hack/boilerplate.go.txt paths=./... -h` for usage
make: *** [Makefile:55: generate] Error 1
2021/03/26 11:53:19 failed to create API: exit status 2
I followed the reminder and executed go get github.com/go-logr/logr@v0.1.0 and go get sigs.k8s.io/controller-tools@v0.2.5. But it didn't solve the error.
I found out that it seems to be caused by the lack of controller-gen CLI, but after installing kubebuiler, I couldn't find any installation guidelines for controller-gen CLI. I executed go get sigs.k8s.io/controller-tools/cmd/controller-gen@v0.4.1 to get the controller-gen, but it didn't work. Is the controller-gen CLI not working because I misconfigured kubebuilder?
My environment:
KubeBuilderVersion:"2.3.1"
K8s : 1.19 (using the custom binaries)
Go version : go1.16.2 linux/amd64
System : Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-58-generic x86_64)
I look forward to your guidance, and I continue to try to fix the error.
PS: since I am new to kubebuilder, please forgive me if I may be asking too simple questions.
Thank you for your interest. I am really happy to help you since your simple question is exactly what I've met.
To run the controller, you don't need to run kubebuilder create api --group podmig --version v1 --kind Podmigration. This command is used by kubebuilder, the first time I build kuberbuilder-based operator, to generate some source code template ( something like this). I will fix the document so that there is nothing to confuse here.
- Firstly, the document is missing the part, in which I setup the NFS server sharing between all hosts in K8S cluster. Since I create a folder
/var/lib/kubelet/migrationat all hosts. I installed NFS server in the master node, NFS client in worker nodes and export/mount the folder from master node to worker node. So that they can share the same folder, which I used to exchange the checkpoint information. You also need to set the folder permission for your user$ sudo chmod 777 /var/lib/kubelet/migration. If you don't know how to config NFS, please refer to my note: https://github.com/vutuong/personal-notes/blob/master/configNFS.md
I'm sorry but I will update the document as soon as possible. - You just need to start from How to run (in the document):
$ sudo apt-get install gcc
$ sudo snap install kustomize
$ make manifests
$ make install
$ make run
- In another terminal, run the Api-server:
$ go run ./api-server/cmd/main.go - Then, you can test with curl command as you can see in the doc.
- P/S: Please check the kubelet, contained before all. You should see the logs as below at worker nodes:
root@cluster2-worker1:/home/dcn# kubelet --version
Kubernetes v1.19.0-beta.0.1010+a94a66e8033cf4-dirty
root@cluster2-worker1:/home/dcn# journalctl -fu kubelet
-- Logs begin at Wed 2021-03-03 06:22:44 UTC. --
Mar 26 12:53:51 cluster2-worker1 kubelet[2670]: I0326 12:53:51.050450 2670 kuberuntime_manager.go:841] Should we migrate?Runningfalse
...
root@cluster2-worker1:/home/dcn# containerd --version
WARN[2021-03-26T12:56:39.392192110Z] This customized containerd is only for CI test, DO NOT use it for distribution.
containerd github.com/containerd/containerd e5ffc7a4-TEST