InputFileStream bufferSize is always 8 bytes
Closed this issue · 5 comments
Dealing with large files I noticed a very slow read operation of the InputFileStream. By looking at the buffer init code it looks like the buffer size is always 8 bytes or less:
_buffer = Uint8List(min(bufferSize, 8));
By changing min to max the issue can be easily fixed. ;)
Well, that's embarrassing :-)
Thanks for catching that, fix coming shortly.
Thanks for looking into it.
I would like to suggest a larger default buffer size. Maybe 0,1% of the file size. This would always result in max 1000 I/O operations.
I'm working on a file that is 60GB large and it took over 1h to process the file with this 8 byte buffer. I then increased the buffer size to 50MB and the whole file was processed within a few minutes.
Yeah, file IO in Dart is really slow.
A 60GB file processed in Dart! I definitely didn't imagine people doing that when it was written :-). But I'm definitely seeing people dealing with larger files now.
I pushed the fix to 3.4.3. I increased the default buffer size to 1MB for now, until I can have a better sense of how much memory is too much memory for Dart.
There were some issues with the last version, so I pushed a new version to git and will publish soon. The old version had a file buffer for every InputFileStream, even the sub-file streams, so a zip with 100k files would have 100k * 1MB cache. The new version uses a shared file cache.
Published in 3.4.5.