Add perftype to filenames to prevent unfulfilled dependency errors
Closed this issue · 5 comments
For some time now, I've been getting spurious "unfulfilled dependency at runtime" errors, and I think I've finally figured it out: A task starts because its self.complete function returns False, but then during the runtime of that task, another task starts that writes to the exact same file, and thus makes the first task's self.complete function return True.
I think the above is the case, but I need to investigate closer. If so, the culprit is not adding perftype to the filenames that all tasks write to - I run both blitz and bullet for a handful of users, and if one of them contains more games than the other, it'll "run over" to the next task's allotted time, thus creating the above error.
Obviously, if the entire pipeline runs at the same time that another instance of the pipeline is running, then the temp files will get purged... which breaks the longer-running pipeline.
Hopefully closed with 7f84fb3 . This commit makes a CopyWrapper only remove the targets that are involved in its completion, not completely clean the temp files.
(Testing this tonight by playing some bullet games!)
I neglected the fact that PostgresTargets won't have a .path attribute. Fixed this in f4993ea .
I also neglected the fact that CopyWrapper's inputs are all PostgresTargets, and I should actually be looking at the jobs list itself and iterating over the functions there. Fixed in 34fce4d .
Hopefully this is the actual end of this issue's saga. The lesson: I need a better testing, probably through a staging/test environment.
Alright, now I think I've actually fixed it. The method is: look at the job list, iterate over each function and (recursively) get the local files. This is finished with commit ab6728c .