Wrong deletion algorithm on windows
ariccio opened this issue · 4 comments
This is a problem that I've seen in every program that I've ever checked for it. And I suspect it's the root cause of all the vscode-typescript files filling my temp folder.
Basically, it has nothing to do with closing handles (as comments in your source suggest), and will almost certainly fail SOMETIMES when the system is under heavy load. The windows kernel is actually very highly asynchronous, way more than POSIX systems, and that file deletion is not what it seems. Deleting a file on windows only marks the file for deletion. The kernel can take an arbitrarily long time under load to actually delete it. This is a good design decision, but makes the obvious algorithm to delete files entirely but insidiously wrong.
See Niall Douglas's presentation on Boost AFIO/LLFIO at around 8:50 for details on the correct (insane) way to do it: https://youtu.be/uhRWMGBjlO8
Wow, that is really helpful! I'm feeling like it's high time for a rewrite of this, to make it use promises instead of callbacks, and remove the (sometimes convenient, but often not so much) glob handling, and use Node.js's native fs.rm when available. I'll add the insane temp file renaming stuff as well :D
Link with timecode: https://youtu.be/uhRWMGBjlO8?t=457
What's funny is I was doing something entirely unrelated, forgotten I had created this issue, and started poking around rimaf just now wondering what the deletion algorithm was. I'm consistent, that's for sure :)
v4 just shipped.
- move-remove is hella slow, even in cases where it's not needed.
- some official Microsoft sources I was pointed to don't use it except as a last resort.
- rimraf now uses move-remove as a last resort, and deletes plain files before directories, which dodges the issue most of the time.
- seems the best practice here really is exponential backoff on EBUSY, so that's what it's doing most of the time.