Missing core log line for some SOOP pipelines
Closed this issue · 5 comments
This a bit weird, but doesn't seem to be blocking anything, so I'll just mention it here, rather than in issues...
In the task.X.log file for most pipelines, a new task always begins with a few lines like this:
2020-02-13 17:15:49,113 SYSINFO tasks.ANMN_NSW[cdea0aac-0eaf-42c2-8160-a0f25fe508ff] PipelineTask.success_exit_policies -> ['DELETE_CUSTOM_REGEXES_FROM_ERROR_STORE']
2020-02-13 17:15:49,129 SYSINFO tasks.ANMN_NSW[cdea0aac-0eaf-42c2-8160-a0f25fe508ff] PipelineTask.error_exit_policies -> []
2020-02-13 17:15:49,129 SYSINFO tasks.ANMN_NSW[cdea0aac-0eaf-42c2-8160-a0f25fe508ff] IncomingFileStateManager.state -> 'FILE_IN_INCOMING'
2020-02-13 17:15:49,143 INFO tasks.ANMN_NSW[cdea0aac-0eaf-42c2-8160-a0f25fe508ff] IncomingFileStateManager.error_broker -> LocalFileStorageBroker(prefix='/mnt/ebs/error/ANMN_NSW')
2020-02-13 17:15:49,143 INFO tasks.ANMN_NSW[cdea0aac-0eaf-42c2-8160-a0f25fe508ff] IncomingFileStateManager.move_to_processing -> '/mnt/ebs/processing/ANMN_NSW/cdea0aac-0eaf-42c2-8160-a0f25fe508ff/SYD140_057_1911
.zip'
2020-02-13 17:15:49,184 SYSINFO tasks.ANMN_NSW[cdea0aac-0eaf-42c2-8160-a0f25fe508ff] IncomingFileStateManager.state -> 'FILE_IN_PROCESSING'
2020-02-13 17:15:49,218 INFO tasks.ANMN_NSW[cdea0aac-0eaf-42c2-8160-a0f25fe508ff] running handler -> MooringsHandler({'input_file': '/mnt/ebs/processing/ANMN_NSW/cdea0aac-0eaf-42c2-8160-a0f25fe508ff/SYD140_057_1911.zip', ... })
In particular, the last line above shows the handler class being called, and all its arguments/attributes at the start.
For some reason, this line is always missing from the logs of these pipelines:
SOOP_ASF_FMT
SOOP_ASF_MT
SOOP_SST
Any ideas why? Does this point to anything that might have more significant consequences?
(I only noticed because I've been using a script to grep the module versions out of the logs for the last run of each pipeline, and it gets nothing for those pipelines.)
@lbesnard @lwgordonimos Let me know if you happen to have any ideas.
No ideas. It logs the line in unit tests, so not sure why it wouldn't output it in a running environment.
BTW, those three pipelines did have that log line until 2019-11-07 (running aodndata 0.5.8, aodncore 0.31.0, python 2.7.1), then they all stopped writing it.
I suspect it's related to the ship callsign list. This is the only handler that is storing this as a class attribute, so might be failing to serialise it when it hits that attribute. I haven't found the smoking gun or fix, but it's very suspect.