Unusing frames hang in memory
Opened this issue · 8 comments
i can't process 20 minutes video because all processed frames remain in memory and there is no free space for a new frame.
as you say "Only VRAM using in DAINAPP" wtf?
The pytorch model use Vram, of course some parts of the core require some ram.
This is a leak in some part of the dataset generation, and this message is from the pytorch part of the code.
https://github.com/pytorch/pytorch/search?q=buy+new+ram
I'm still looking for a way to fix this leak.
Confirmed. Getting this leak too.
I was able to keep it running, despite this huge leak. Here`s how.
- Disable Windows memory compression. This service tries to compress frames in memory, but instead of helping it is just ballooning inside your RAM, thus leaving no more free space for allocation.
- Manually set your virtual memory paging file size to something huge. I had to use 100GB paging file to interpolate 3700 4k frames.
- Optionally you can use app like Process Lasso to periodically manually flush DAIN RAM artifacts to virtual memory.
Pytorch return error string "buy new ram? " lmao
I found the leak, it is in DainDataset.py line 49.
my_combo["original"] = numpy.array(PIL.Image.open(self.combos[index]["f1"]).convert(self.frameFormat))
The dict where this image is added is still referenced by member variable self.combos, so you effectively store the entire raw dataset in RAM when iterating over the DataLoader. Just deleting this line should fix the leak. I didn't see combo["original"] being used anywhere. Another possible fix would be changing line 48 from
my_combo = self.combos[index]
to
my_combo = self.combos[index].copy()
I found the leak, it is in DainDataset.py line 49.
my_combo["original"] = numpy.array(PIL.Image.open(self.combos[index]["f1"]).convert(self.frameFormat))
The dict where this image is added is still referenced by member variable self.combos, so you effectively store the entire raw dataset in RAM when iterating over the DataLoader. Just deleting this line should fix the leak. I didn't see combo["original"] being used anywhere. Another possible fix would be changing line 48 from
my_combo = self.combos[index]to
my_combo = self.combos[index].copy()
Can you please upload the modified program as a package? Because there are many people, including me, who are not familiar with Python. Thank you very much.
@lmichalke File "Dain-App/my_DAIN_class.py", line 1102:
cleanFiles.append( {"data" : combo['original'][tt].numpy(), 'timestamp' : start})
Deleting line 49 in DainDataset.py will cause an error.