Pysmurf controller check_state failing out with ChannelAccessGetFailure

[jlashner]

Pysmurf controller is sometimess failng out of the check_state process due to ChannelAccessGetFailure, as shown in logs below:

2023-11-20T17:24:35+0000 Warning! Not making output directories!This will break may things!
2023-11-20T17:24:35+0000 check_state:0 Status is now "running".
2023-11-20T17:24:40+0000 Could not connect to epics server! Waiting and then trying again
2023-11-20T17:24:55+0000 Could not connect to epics server! Waiting and then trying again
2023-11-20T17:25:10+0000 Could not connect to epics server! Waiting and then trying again
2023-11-20T17:25:25+0000 Could not connect to epics server! Waiting and then trying again
2023-11-20T17:25:40+0000 Could not connect to epics server! Waiting and then trying again
2023-11-20T17:25:55+0000 Could not connect to epics server! Waiting and then trying again
2023-11-20T17:26:10+0000 Could not connect to epics server! Waiting and then trying again
2023-11-20T17:26:25+0000 Could not connect to epics server! Waiting and then trying again
2023-11-20T17:26:40+0000 Could not connect to epics server! Waiting and then trying again
2023-11-20T17:26:56+0000 [ 2023-11-20 17:26:56 ]  Command failed: smurf_server_s7:AMCc:SmurfProcessor:Downsampler:DownsamplerMode
2023-11-20T17:26:56+0000 [ 2023-11-20 17:26:56 ]  Retry attempt 1 of 5
2023-11-20T17:27:01+0000 [ 2023-11-20 17:27:01 ]  Retry attempt 2 of 5
2023-11-20T17:27:08+0000 Could not connect to epics server! Waiting and then trying again
2023-11-20T17:27:23+0000 Could not connect to epics server! Waiting and then trying again
2023-11-20T17:27:38+0000 Could not connect to epics server! Waiting and then trying again
CA.Client.Exception...............................................
    Warning: "Virtual circuit unresponsive"
    Context: "localhost:45831"
    Source File: ../tcpiiu.cpp line 919
    Current Time: Mon Nov 20 2023 17:27:53.958065110
..................................................................
2023-11-20T17:27:53+0000 check_state:0 CRASH: [Failure instance: Traceback: <class 'epics.ca.ChannelAccessGetFailure'>: Get failed; status code: 192
/usr/lib/python3.8/threading.py:932:_bootstrap_inner
/usr/lib/python3.8/threading.py:870:run
/usr/local/lib/python3.8/dist-packages/twisted/_threads/_threadworker.py:49:work
/usr/local/lib/python3.8/dist-packages/twisted/_threads/_team.py:192:doWork
--- <exception caught here> ---
/usr/local/lib/python3.8/dist-packages/twisted/python/threadpool.py:269:inContext
/usr/local/lib/python3.8/dist-packages/twisted/python/threadpool.py:285:<lambda>
/usr/local/lib/python3.8/dist-packages/twisted/python/context.py:117:callWithContext
/usr/local/lib/python3.8/dist-packages/twisted/python/context.py:82:callWithContext
/usr/local/lib/python3.8/dist-packages/socs/agents/pysmurf_controller/agent.py:308:check_state
/usr/local/src/pysmurf/python/pysmurf/client/command/smurf_command.py:6256:get_channel_mask
/usr/local/src/pysmurf/python/pysmurf/client/command/smurf_command.py:208:_caget
/usr/local/lib/python3.8/dist-packages/epics/pv.py:484:get
/usr/local/lib/python3.8/dist-packages/epics/pv.py:47:wrapped
/usr/local/lib/python3.8/dist-packages/epics/pv.py:563:get_with_metadata
/usr/local/lib/python3.8/dist-packages/epics/ca.py:641:wrapper
/usr/local/lib/python3.8/dist-packages/epics/ca.py:1396:get_with_metadata
/usr/local/lib/python3.8/dist-packages/epics/ca.py:641:wrapper
/usr/local/lib/python3.8/dist-packages/epics/ca.py:1537:get_complete_with_metadata
]

In the code I am catching RuntimeErrors, which I believe are thrown when the agent cannot connect to the epics-server whatsoever, but not ChannelAccessGetFailures, which must be caused by something else... potentially the epics server existing but being unresponsive, or not having the requested variable