chaosblade-io/chaosblade-operator

(chaosblade-operator 1.6.1) cgroups load failed, cgroups: cgroup deleted

Closed this issue · 4 comments

crushCoin commented

Issue Description

Type: bug report or feature request

Describe what happened (or what feature you want)

chaosblade-box trigger

Describe what you expected to happen

cpu fullload

How to reproduce it (as minimally and precisely as possible)

  1. helm install chaosblade-operator chaosblade-operator-1.6.1.tgz --namespace chaosblade --set blade.repository=chaosbladeio/chaosblade-tool,operator.repository=chaosbladeio/chaosblade-operator

Tell us your environment

chaosblade-operator 1.6.1
k8s 1.20.0
docker 19.03.9

Anything else we need to know?

no

chaosblade-box log

level=info msg="execute identifier: {ContainerObjectMeta:{Id: ContainerRuntime:docker ContainerId:54bee0d61848 ContainerName:busybox PodName:busybox-7577d445dc-mdr8q NodeName:master Namespace:default} Command:/opt/chaosblade/blade create cri cpu fullload --cpu-percent=90 --debug=true --timeout=65 --container-id 54bee0d61848 --container-runtime docker Error: Code:0 ChaosBladePodName:chaosblade-tool-svcsn ChaosBladeNamespace:chaosblade ChaosBladeContainerName:chaosblade-tool}" experiment=288b3fdd4d271824
time="2022-07-11T07:53:50Z" level=info msg="Exec command in pod" command="[/opt/chaosblade/blade create cri cpu fullload --cpu-percent=90 --debug=true --timeout=65 --container-id 54bee0d61848 --container-runtime docker]" container=chaosblade-tool podName=chaosblade-tool-svcsn podNamespace=chaosblade
time="2022-07-11T07:53:51Z" level=info msg="get err message" command="[/opt/chaosblade/blade create cri cpu fullload --cpu-percent=90 --debug=true --timeout=65 --container-id 54bee0d61848 --container-runtime docker]" container=chaosblade-tool err="{"code":63063,"success":false,"error":"cgroups load failed, cgroups: cgroup deleted"}" out= podName=chaosblade-tool-svcsn podNamespace=chaosblade

jiangqiwei commented

same problem
image

time="2022-07-09T23:31:45Z" level=info msg="start to execute: &{Target:cpu Scope:container ActionName:fullload ActionFlags:map[container-ids:e144e9a932d3 cpu-percent:100 names:nginx-54b64fc459-pspxt namespace:dev] ActionPrograms:[] ActionCategories:[] ActionProcessHang:false}" experiment=increase-container-cpu-load-by-id
time="2022-07-09T23:31:45Z" level=info msg="experiment identifiers: [{{ docker e144e9a932d3 nginx nginx-54b64fc459-pspxt node2 dev} /opt/chaosblade/blade create cri cpu fullload --cpu-percent=100 --container-id e144e9a932d3 --container-runtime docker 0 chaosblade-tool-j8nqm chaosblade chaosblade-tool}]" experiment=increase-container-cpu-load-by-id
time="2022-07-09T23:31:45Z" level=info msg="execute identifier: {ContainerObjectMeta:{Id: ContainerRuntime:docker ContainerId:e144e9a932d3 ContainerName:nginx PodName:nginx-54b64fc459-pspxt NodeName:node2 Namespace:dev} Command:/opt/chaosblade/blade create cri cpu fullload --cpu-percent=100 --container-id e144e9a932d3 --container-runtime docker Error: Code:0 ChaosBladePodName:chaosblade-tool-j8nqm ChaosBladeNamespace:chaosblade ChaosBladeContainerName:chaosblade-tool}" experiment=increase-container-cpu-load-by-id
time="2022-07-09T23:31:45Z" level=info msg="Exec command in pod" command="[/opt/chaosblade/blade create cri cpu fullload --cpu-percent=100 --container-id e144e9a932d3 --container-runtime docker]" container=chaosblade-tool podName=chaosblade-tool-j8nqm podNamespace=chaosblade
time="2022-07-09T23:31:46Z" level=info msg="get err message" command="[/opt/chaosblade/blade create cri cpu fullload --cpu-percent=100 --container-id e144e9a932d3 --container-runtime docker]" container=chaosblade-tool err="{"code":63063,"success":false,"error":"cgroups load failed, cgroups: cgroup deleted"}" out= podName=chaosblade-tool-j8nqm podNamespace=chaosblade
time="2022-07-09T23:31:46Z" level=error msg="pods/exec: k8s exec failed, err: {"code":63063,"success":false,"error":"cgroups load failed, cgroups: cgroup deleted"}\n" location=github.com/chaosblade-io/chaosblade-operator/exec/model.execCommands.func1 uid=
time="2022-07-09T23:31:46Z" level=info msg="success: false, statuses: [{Id: State:Error Code:63063 Error:cgroups load failed, cgroups: cgroup deleted Success:false Kind:container Identifier:dev/node2/nginx-54b64fc459-pspxt/nginx/e144e9a932d3/docker}]" experiment=increase-container-cpu-load-by-id

tiny-x commented

same problem image

time="2022-07-09T23:31:45Z" level=info msg="start to execute: &{Target:cpu Scope:container ActionName:fullload ActionFlags:map[container-ids:e144e9a932d3 cpu-percent:100 names:nginx-54b64fc459-pspxt namespace:dev] ActionPrograms:[] ActionCategories:[] ActionProcessHang:false}" experiment=increase-container-cpu-load-by-id time="2022-07-09T23:31:45Z" level=info msg="experiment identifiers: [{{ docker e144e9a932d3 nginx nginx-54b64fc459-pspxt node2 dev} /opt/chaosblade/blade create cri cpu fullload --cpu-percent=100 --container-id e144e9a932d3 --container-runtime docker 0 chaosblade-tool-j8nqm chaosblade chaosblade-tool}]" experiment=increase-container-cpu-load-by-id time="2022-07-09T23:31:45Z" level=info msg="execute identifier: {ContainerObjectMeta:{Id: ContainerRuntime:docker ContainerId:e144e9a932d3 ContainerName:nginx PodName:nginx-54b64fc459-pspxt NodeName:node2 Namespace:dev} Command:/opt/chaosblade/blade create cri cpu fullload --cpu-percent=100 --container-id e144e9a932d3 --container-runtime docker Error: Code:0 ChaosBladePodName:chaosblade-tool-j8nqm ChaosBladeNamespace:chaosblade ChaosBladeContainerName:chaosblade-tool}" experiment=increase-container-cpu-load-by-id time="2022-07-09T23:31:45Z" level=info msg="Exec command in pod" command="[/opt/chaosblade/blade create cri cpu fullload --cpu-percent=100 --container-id e144e9a932d3 --container-runtime docker]" container=chaosblade-tool podName=chaosblade-tool-j8nqm podNamespace=chaosblade time="2022-07-09T23:31:46Z" level=info msg="get err message" command="[/opt/chaosblade/blade create cri cpu fullload --cpu-percent=100 --container-id e144e9a932d3 --container-runtime docker]" container=chaosblade-tool err="{"code":63063,"success":false,"error":"cgroups load failed, cgroups: cgroup deleted"}" out= podName=chaosblade-tool-j8nqm podNamespace=chaosblade time="2022-07-09T23:31:46Z" level=error msg="pods/exec: k8s exec failed, err: {"code":63063,"success":false,"error":"cgroups load failed, cgroups: cgroup deleted"}\n" location=github.com/chaosblade-io/chaosblade-operator/exec/model.execCommands.func1 uid= time="2022-07-09T23:31:46Z" level=info msg="success: false, statuses: [{Id: State:Error Code:63063 Error:cgroups load failed, cgroups: cgroup deleted Success:false Kind:container Identifier:dev/node2/nginx-54b64fc459-pspxt/nginx/e144e9a932d3/docker}]" experiment=increase-container-cpu-load-by-id

可以钉钉加群 23177705

crushCoin commented

23177705

same problem image
time="2022-07-09T23:31:45Z" level=info msg="start to execute: &{Target:cpu Scope:container ActionName:fullload ActionFlags:map[container-ids:e144e9a932d3 cpu-percent:100 names:nginx-54b64fc459-pspxt namespace:dev] ActionPrograms:[] ActionCategories:[] ActionProcessHang:false}" experiment=increase-container-cpu-load-by-id time="2022-07-09T23:31:45Z" level=info msg="experiment identifiers: [{{ docker e144e9a932d3 nginx nginx-54b64fc459-pspxt node2 dev} /opt/chaosblade/blade create cri cpu fullload --cpu-percent=100 --container-id e144e9a932d3 --container-runtime docker 0 chaosblade-tool-j8nqm chaosblade chaosblade-tool}]" experiment=increase-container-cpu-load-by-id time="2022-07-09T23:31:45Z" level=info msg="execute identifier: {ContainerObjectMeta:{Id: ContainerRuntime:docker ContainerId:e144e9a932d3 ContainerName:nginx PodName:nginx-54b64fc459-pspxt NodeName:node2 Namespace:dev} Command:/opt/chaosblade/blade create cri cpu fullload --cpu-percent=100 --container-id e144e9a932d3 --container-runtime docker Error: Code:0 ChaosBladePodName:chaosblade-tool-j8nqm ChaosBladeNamespace:chaosblade ChaosBladeContainerName:chaosblade-tool}" experiment=increase-container-cpu-load-by-id time="2022-07-09T23:31:45Z" level=info msg="Exec command in pod" command="[/opt/chaosblade/blade create cri cpu fullload --cpu-percent=100 --container-id e144e9a932d3 --container-runtime docker]" container=chaosblade-tool podName=chaosblade-tool-j8nqm podNamespace=chaosblade time="2022-07-09T23:31:46Z" level=info msg="get err message" command="[/opt/chaosblade/blade create cri cpu fullload --cpu-percent=100 --container-id e144e9a932d3 --container-runtime docker]" container=chaosblade-tool err="{"code":63063,"success":false,"error":"cgroups load failed, cgroups: cgroup deleted"}" out= podName=chaosblade-tool-j8nqm podNamespace=chaosblade time="2022-07-09T23:31:46Z" level=error msg="pods/exec: k8s exec failed, err: {"code":63063,"success":false,"error":"cgroups load failed, cgroups: cgroup deleted"}\n" location=github.com/chaosblade-io/chaosblade-operator/exec/model.execCommands.func1 uid= time="2022-07-09T23:31:46Z" level=info msg="success: false, statuses: [{Id: State:Error Code:63063 Error:cgroups load failed, cgroups: cgroup deleted Success:false Kind:container Identifier:dev/node2/nginx-54b64fc459-pspxt/nginx/e144e9a932d3/docker}]" experiment=increase-container-cpu-load-by-id

可以钉钉加群 23177705

已经解决了，修改了 chaosblade-tool（DaemonSet）的 yaml 文件 中 数据卷 路径 就可以了

  • hostPath: - hostPath:
    path: /host-sys -----> path: /sys
    name: sys name: sys

请问 chaosblade-operator 打印如下日志，有问题吗？
time="2022-07-14T08:37:23Z" level=error msg="pods/exec: k8s exec failed, err: {"code":200,"success":true,"result":{"Uid":"dd46beac82be3d99","Command":"cri","SubCommand":"cpu fullload","Flag":" --container-runtime=docker --container-id=f3e7bdb1d3e7 --cpu-percent=50 --timeout=65","Status":"Destroyed","Error":"","CreateTime":"2022-07-14T08:35:48.035019842Z","UpdateTime":"2022-07-14T08:35:53.565399056Z"}}\n" location=github.com/chaosblade-io/chaosblade-operator/exec/model.checkExperimentStatus.func1.2 uid=

k8s 环境 chaosblade-box-agent 上报心跳到 chaosblade-box 存在 ip参数丢失 问题，导致 t_chaos_device 表 public_ip为空，chaosblade-box 界面 k8s 实验无法进行

chaosblade-box-1.0.1.tgz chaosblade-box-agent-1.0.0.tgz

chaosblade-box 日志信息：
start handle client heartbeat request,{}{"ak":"9b2432e91e164f3499494f2df37a574f","cbv":"1.5.0","cid":"9c720c27a190d758124ea2b12fb43ec2","ip":"","pid":"1","rid":"d6441fca-6c5c-45d7-a4f7-8488fbf7dedd","uid":"1090011579","v":"1.0.0"}

tiny-x commented

1.6.0 需要手动改成 /sys， 1.6.1 直接使用即可