Running out of disk space due to enormous dat_0 file
lorenzoromani1983 opened this issue · 2 comments
Hi, I am running onionscanner on a digitalocean droplet, but I am running out of disk space because the dat_0 file stored in /onionscandb/relationships gets huge: 16 gb!
I have tryed deleting it but this is not working.
how can I solve the issue?
tnx!
lorenzo
Currently OnionScan snapshots everything it finds for later processing, this means the database grows pretty big (for reference on larger onionscan scans I've had the database grow over 100GB) - there is no current way to not have this happen without modifying the code. In the future there might be a version of OnionScan that doesn't do this but I currently don't have the time or funding to work on it to make it publicly available.
See also: #128
Thank you. However, I managed to delete the dat_0 file in order to free up disk space: i only need .json files for further network analysis, so I guess it's not wrong to delete the snapshots. I deleted the dat_0 file in /crawls.