ncsa/Swift-T-Variant-Calling

logging function causing race conditions

Opened this issue · 0 comments

During testing on iForge, when testing jumping into the CombineChromosomes stage, we ran into situations that complained that a file X in the tmp/timinglogs folder could not be removed, as it doesn't exist.

After looking around for a long time, we hypothesized that the logging function was causing it.

This is how the logging function is called:

doSomething() =>
logging()

doSomethingElse() =>
logging()

If the doSomethingElse() function finishes very quickly (in a small test case, for example), it is possible for the second call to logging function to start before the first call has finished.

The logging function does three things:

  • Step 1. Gather the info from all of the files in the tmp/timinglog directory
  • Step 2. Save this information in the permanent Timing.log file located in the output/deliverables folder
  • Step 3. Delete the original files that were gathered in the tmp/timinglog directory

This means that the second call could gather the names of files in Step 1 before the first call has finished successfully deleted all of the files it was supposed to handle in its Step 3. But eventually one of the functions will finish Step 3. After this, when the other function call gets to its Step 3, the files that it is supposed to delete no longer exist, and a fatal error occurs.

To try and fix this, I am going to add a 'void' output to the logging() function, and make sure that all calls to logging have to be waited on before moving on to the next step.

We'll see if it works