Get Tasks up and Running
sjmf opened this issue · 2 comments
Some changes need to be made to the docker configuration to get tasks to run correctly. Currently the task calcuations.dailyUpdateModel fails with a number of errors which need to be worked through involving some re-configuration of the Dockerfile.
[2022-09-15 14:03:49,365: ERROR/ForkPoolWorker-2] Task calculations.dailyModelUpdate[e0393d68-3ed2-44a4-b6e2-82ee964ce8a6] raised unexpected: DoesNotExist('ZentraDevice matching query does not exist.')
Traceback (most recent call last):
File "/opt/conda/envs/ManyFEWS/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/conda/envs/ManyFEWS/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
return self.run(*args, **kwargs)
File "/app/calculations/tasks.py", line 118, in dailyModelUpdate
location = ZentraDevice.objects.get(device_sn=settings.STATION_SN).location
File "/opt/conda/envs/ManyFEWS/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/conda/envs/ManyFEWS/lib/python3.9/site-packages/django/db/models/query.py", line 650, in get
raise self.model.DoesNotExist(
calculations.models.ZentraDevice.DoesNotExist: ZentraDevice matching query does not exist.
To address this, a Zentra device matching the device in the environment variable needs to be created. Could this be automated somehow?
[2022-09-15 14:25:25,656: ERROR/ForkPoolWorker-2] Task calculations.dailyModelUpdate[d347d411-6456-4c79-b7ad-d543d20de202] raised unexpected: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
File "/opt/conda/envs/ManyFEWS/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/conda/envs/ManyFEWS/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
return self.run(*args, **kwargs)
File "/app/calculations/tasks.py", line 156, in dailyModelUpdate
updateInitialData = runningGenerateRiverFlows(
File "/app/calculations/generate_river_flows.py", line 518, in runningGenerateRiverFlows
riverFlowsData = GenerateRiverFlows(
File "/app/calculations/generate_river_flows.py", line 364, in GenerateRiverFlows
X = np.loadtxt(open(parametersFilePath), delimiter=",", usecols=range(4))
FileNotFoundError: [Errno 2] No such file or directory: '/Data/RainfallRunoffModelParameters.csv'
To address this, the Data directory needs to be either embedded into the image (+28MB) or mounted as a volume on the deployment host.
However, there is a larger issue here around feedback of task status and failures to the user UI which will need to be tracked in a separate issue. There is currently no display of whether a task has succeeded or failed in the UI.
I have fixed the above errors in commit 8dedb87. However, a new error is thrown:
Traceback (most recent call last):
File "/opt/conda/envs/ManyFEWS/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/conda/envs/ManyFEWS/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
return self.run(*args, **kwargs)
File "/app/calculations/tasks.py", line 156, in dailyModelUpdate
updateInitialData = runningGenerateRiverFlows(
File "/app/calculations/generate_river_flows.py", line 518, in runningGenerateRiverFlows
riverFlowsData = GenerateRiverFlows(
File "/app/calculations/generate_river_flows.py", line 376, in GenerateRiverFlows
modelfunOutputData = ModelFun(qp, Ep, dt, CatArea, X, F0)
File "/app/calculations/generate_river_flows.py", line 34, in ModelFun
S0 = F0[n, 0] # initial storage level for PDM (mm)
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed
This is within the model code, specifically line 156, in the function dailyModelUpdate
in file /app/calculations/tasks.py
, it looks like the parameter being passed to the function runningGenerateRiverFlows
is incorrect somehow.
@Abel-Durham Perhaps you have some insight? I am wondering if this is because the GEFS data is missing as the task to download it has not run (or failed?). The variable F0 doesn't seem to contain whatever line 34 in /app/calculations/generate_river_flows.py
was expecting.
@Abel-Durham The solution to the above error is to run the task calculations.initialModelSetUp
first!
I suggest some parameter sanity-checking in the runningGenerateRiverFlows function, and possibly also handling the IndexError and generating a verbose error message that F0 is not populated correctly and to run initialModelSetup first.
Closing this issue as the errors are fixed. See issue #83 for ongoing work.