Pod Security preventing kube-burner pod to start running
paigerube14 opened this issue · 5 comments
paigerube14 commented
Hitting issue with pod security running scale-ci tests on the newest nightly build.
% oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.11.0-0.nightly-2022-06-01-200905 True False 4h41m Cluster version is 4.11.0-0.nightly-2022-06-01-200905
% oc get pods -n benchmark-operator
NAME READY STATUS RESTARTS AGE
benchmark-controller-manager-7fc66ff769-stwlh 2/2 Running 0 111m
kube-burner-00834398-nkwm4 0/1 Init:Error 0 46m
Traceback (most recent call last):
File "stockpile-wrapper.py", line 257, in <module>
sys.exit(main())
File "stockpile-wrapper.py", line 224, in main
run = _mark_node(r, my_node, my_uuid, es, check_val)
File "stockpile-wrapper.py", line 147, in _mark_node
current_val = r.get(check_val)
File "/usr/local/lib/python3.6/site-packages/redis/commands/core.py", line 1600, in get
return self.execute_command("GET", name)
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 1222, in execute_command
lambda error: self._disconnect_raise(conn, error),
File "/usr/local/lib/python3.6/site-packages/redis/retry.py", line 45, in call_with_retry
return do()
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 1220, in <lambda>
conn, command_name, *args, **options
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 1195, in _send_command_parse_response
return self.parse_response(conn, command_name, **options)
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 1234, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 836, in read_response
raise response
redis.exceptions.ResponseError: DENIED Redis is running in protected mode because protected mode is enabled and no password is set for the default user. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a an authentication password for the default user. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.
Saw PR, not sure if that will help with this or not
elenagerman commented
I have just the same problem with uperf, almost sure it is not related to the ocp version. I'm using 4.7 stable
Unable to run any netperf traffic from today's morning, no client or server pods created, just stuck on this error backpack pods:
Traceback (most recent call last):
File "stockpile-wrapper.py", line 257, in <module>
sys.exit(main())
File "stockpile-wrapper.py", line 224, in main
run = _mark_node(r, my_node, my_uuid, es, check_val)
File "stockpile-wrapper.py", line 147, in _mark_node
current_val = r.get(check_val)
File "/usr/local/lib/python3.6/site-packages/redis/commands/core.py", line 1600, in get
return self.execute_command("GET", name)
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 1222, in execute_command
lambda error: self._disconnect_raise(conn, error),
File "/usr/local/lib/python3.6/site-packages/redis/retry.py", line 45, in call_with_retry
return do()
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 1220, in <lambda>
conn, command_name, *args, **options
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 1195, in _send_command_parse_response
return self.parse_response(conn, command_name, **options)
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 1234, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 836, in read_response
raise response
redis.exceptions.ResponseError: DENIED Redis is running in protected mode because protected mode is enabled and no password is set for the default user. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a an authentication password for the default user. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.
elenagerman commented
This one should help: cloud-bulldozer/benchmark-operator#766
Thank you @jtaleric for catch it so quick!!
rsevilla87 commented
I think it's been fixed with cloud-bulldozer/benchmark-operator#752, can you confirm?
paigerube14 commented
@rsevilla87 seeing separate issue after that with prom_token being set. Should I open a new issue?
TASK [Launching kube-burner job] ********************************
�[0;31mfatal: [localhost]: FAILED! => {�[0m
�[0;31m "msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'prom_token'\n\nThe error appears to be in '/opt/ansible/roles/kube-burner/tasks/main.yml': line 217, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: Launching kube-burner job\n ^ here\n"�[0m
�[0;31m}�[0m
-------------------------------------------------------------------------------
{"level":"error","ts":1654265081.9354627,"logger":"logging_event_handler","msg":"","name":"kube-burner-pod-density-046cf042-9b3b-44ad-86c1-1ff69eab5658","namespace":"benchmark-operator","gvk":"ripsaw.cloudbulldozer.io/v1alpha1, Kind=Benchmark","event_type":"runner_on_failed","job":"4037200794235010051","EventData.Task":"Launching kube-burner job","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/kube-burner/tasks/main.yml:217","error":"[playbook task failed]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132\ngithub.com/operator-framework/operator-sdk/internal/ansible/events.loggingEventHandler.Handle\n\t/workspace/internal/ansible/events/log_events.go:110"}
Edit: see PR open already for this
paigerube14 commented
Now able to run with no issues! Thanks for all the updates @rsevilla87