Race condition found
Opened this issue · 0 comments
using linux 64
There is a race condition in gdal2tiles_parallel at line ~1866
# Create directories for the tile
if not path.exists(path.dirname(tilefilename)):
makedirs(path.dirname(tilefilename))
During the generation of the overview tiles, two of the sub processes could run into a well known race condition and cause a FileExistsError exception. There are two possible solutions... Since our private fork has some other changes and we are python 3 only, we fixed this with
makedirs(path.dirname(tilefilename), exist_ok=True )
It is debatable that you could also use something like
if not path.exists(path.dirname(tilefilename)):
try:
makedirs(path.dirname(tilefilename))
except OSError as exception:
if exception.errno != errno.EEXIST or not os.path.isdir(path):
raise
There is a long discussion here about the pros and cons of various solutions. Since I am a neophyte python programmer, I defer to those more knowledgeable:
NOTE This will also allow for a single execution of the main path by removing the second call to Main()
if __name__ == '__main__':
main(None)
main(None)
becomes
if __name__ == '__main__':
main(None)