Refreshing probe list return an empty list
PostNoctem opened this issue · 5 comments
Hello guys,
I am tryharding to make IRMA work more than one day.
Indeed, it is working only less than 24h in a row, then the probe list is randomly emptied because brain and probes do not communicate anymore.
All my supervisor tasks are in debug mode, but I have no logs explaining why the connection between probes and brain is timed out.
Here is an example of brain supervisor.log showed.
brain log.txt
If you have an idea of where I can find the origin of the issue... Thank you.
This seems to be purely rabbitmq related, brain is looking for active queues and detects none, You should see why probes are no longer connected to the rabbitmq server, by activating debug.
You could also try rabbitmqctl commands to see active queues and see if probes are connected or not.
btw supervisord is no longer used in IRMA (we swiched to systemd recently) but it does not help for your problem
I guess it is from rabbitmq indeed.
I put the debug mode on, tried the web interface of rabbitmq and spied the logs. All looks great except a single line in my syslog:
WARNING:celery.worker.consumer:consumer: Connection to broker lost. Trying to re-establish the connection...
But il looks like it never reconnects.
I do not know where to investigate :)
Check settings in /opt/irma/irma-{frontend,probe}/current/config/{frontend,probe}.ini maybe brain address (default 172.16.1.30) is wrong and could be replaced by 127.0.0.1 if you are running an allinone environment.
That could also help:
https://irma.readthedocs.io/en/latest/troubleshooting/check_celery.html
Addresses are okay as celery checks :/
The probes are connected once, then randomly times-out and cannot re-connect anymore.