Cannot access to external Redis in TLS
vfouqueron opened this issue · 1 comments
vfouqueron commented
What went wrong?
What happened:
- I cannot start the engine because of an issue accessing our external Redis in TLS, the following error occurs :
2024-04-23T12:21:48.100947879Z 2024-04-23 12:21:48 source=engine:app google_trace_id=none logger=apps.social_auth.middlewares SocialAuthAuthCanceledExceptionMiddleware.process_exception: Error 1 connecting to redis-headless.grafana-on-call.svc.cluster.local:26379. [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002).
2024-04-23T12:21:48.119202639Z 2024-04-23 12:21:48 /usr/local/li
b/python3.11/site-packages/django/views/debug.py:487: ExceptionCycleWarning: Cycle in the exception chain detected: exception 'Error 1 connecting to redis-headless.grafana-on-call.svc.cluster.local:26379. [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002).' encountered again.
2024-04-23T12:21:48.119239310Z warnings.warn(
2024-04-23T12:21:48.188197833Z 2024-04-23 12:21:48 source=engine:app google_trace_id=none logger=django.request Internal Server Error: /startupprobe/
2024-04-23T12:21:48.188243272Z Traceback (most recent call last):
2024-04-23T12:21:48.188249144Z File "/usr/local/lib/python3.11/site-packages/django_redis/cache.py", line 29, in _decorator
2024-04-23T12:21:48.188252789Z return method(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-23T12:21:48.188260051Z File "/usr/local/lib/python3.11/site-packages/django_redis/cache.py", line 99, in _get
2024-04-23T12:21:48.188264456Z return self.client.get(key, default=default, version=version, client=client)
2024-04-23T12:21:48.188267989Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django_redis/client/default.py", line 260, in get
2024-04-23T12:21:48.188276968Z raise ConnectionInterrupted(connection=client) from e
2024-04-23T12:21:48.188283515Z django_redis.exceptions.ConnectionInterrupted: Redis ConnectionError: Error 1 connecting to redis-headless.grafana-on-call.svc.cluster.local:26379. [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002).
During handling of the above exception, another exception occurred:
2024-04-23T12:21:48.188304105Z Traceback (most recent call last):
2024-04-23T12:21:48.188307713Z File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
2024-04-23T12:21:48.188311705Z response = get_response(request)
2024-04-23T12:21:48.188316055Z ^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
2024-04-23T12:21:48.188324131Z response = wrapped_callback(request, *callback_args, **callback_kwargs)
2024-04-23T12:21:48.188348423Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
2024-04-23T12:21:48.188356143Z return self.dispatch(request, *args, **kwargs)
2024-04-23T12:21:48.188359560Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-23T12:21:48.188363029Z File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
return handler(request, *args, **kwargs)
2024-04-23T12:21:48.188370338Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-23T12:21:48.188374250Z File "/etc/app/engine/views.py", line 46, in get
if cache.get(AlertChannelDefiningMixin.CACHE_KEY_DB_FALLBACK) is None:
2024-04-23T12:21:48.188381920Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-23T12:21:48.188385603Z File "/usr/local/lib/python3.11/site-packages/django_redis/cache.py", line 92, in get
2024-04-23T12:21:48.188389144Z value = self._get(key, default, version, client)
2024-04-23T12:21:48.188392535Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django_redis/cache.py", line 36, in _decorator
2024-04-23T12:21:48.188417587Z raise e.__cause__
File "/usr/local/lib/python3.11/site-packages/django_redis/client/default.py", line 258, in get
2024-04-23T12:21:48.188425896Z value = client.get(key)
2024-04-23T12:21:48.188429382Z ^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/commands/core.py", line 1829, in get
2024-04-23T12:21:48.188436499Z return self.execute_command("GET", name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-23T12:21:48.188444006Z File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 533, in execute_command
2024-04-23T12:21:48.188447626Z conn = self.connection or pool.get_connection(command_name, **options)
2024-04-23T12:21:48.188451177Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-23T12:21:48.188454627Z File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 1291, in get_connection
2024-04-23T12:21:48.188457875Z connection.connect()
2024-04-23T12:21:48.188461310Z File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 270, in connect
2024-04-23T12:21:48.188464656Z raise ConnectionError(self._error_message(e))
2024-04-23T12:21:48.188468520Z redis.exceptions.ConnectionError: Error 1 connecting to redis-headless.grafana-on-call.svc.cluster.local:26379. [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002).
2024-04-23T12:21:48.188734658Z 2024-04-23 12:21:48 source=engine:app google_trace_id=none logger=root inbound latency=0.671993 status=500 method=GET path=/startupprobe/ user_agent=kube-probe/1.26 content-length=0 slow=0
What did you expect to happen:
- The engine starts
How do we reproduce it?
Here is our Grafana On Call chart :
cert-manager:
enabled: false
externalGrafana:
url: <URL to our Grafana>
externalMysql:
db_name: grafana-on-call
host: <URL to our database>
password: <MySQL Password>
port: <Database port>
user: <Username>
externalRabbitmq:
host: rabbitmq.common-rabbitmq.svc.cluster.local
password: <RabbitMQ password>
port: 5672
user: <RabbitMQ user>
vhost: grafana-on-call
externalRedis:
host: redis-headless.grafana-on-call.svc.cluster.local
password: <Redis Password>
port: 26379
protocol: rediss
ssl_options:
ca_certs: |
<Our internal CA certificate>
cert_reqs: cert_none
enabled: true
grafana:
enabled: false
ingress:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-production
cert-manager.io/common-name: <URL to our instance>
hostname: <URL to our instance>
ingressClassName: nginx
ingress-nginx:
enabled: false
mariadb:
enabled: false
rabbitmq:
enabled: false
redis:
enabled: false
Grafana OnCall Version
v1.4.3
Product Area
Helm
Grafana OnCall Platform?
Kubernetes
User's Browser?
No response
Anything else to add?
- I can PING the redis instance with the cli using the command
redis-cli -u rediss://default:$REDIS_PASSWORD@redis-headless.grafana-on-call.svc.cluster.local:26379 --cacert /opt/bitnami/redis/certs/ca.crt PING - It is using sentinel
vfouqueron commented
I was using Redis with sentinel, which is not supported. I disabled sentinel and all works fine.