Asynchronous read-ahead for Go readers
This package will allow you to add readhead to any reader. This means a separate goroutine will perform reads from your upstream reader, so you can request from this reader without delay.
This is helpful for splitting an input stream into concurrent processing, and also helps smooth out bursts of input or output.
This should be fully transparent, except that once an error has been returned from the Reader, it will not recover.
The readahead object also fulfills the io.WriterTo interface, which is likely to speed up io.Copy and other code that use the interface.
See an introduction: An Async Read-ahead Package for Go
To get the package use go get -u github.com/klauspost/readahead.
Here is a simple example that does file copy. Error handling has been omitted for brevity.
input, _ := os.Open("input.txt")
output, _ := os.Create("output.txt")
defer input.Close()
defer output.Close()
// Create a read-ahead Reader with default settings
ra := readahead.NewReader(input)
defer ra.Close()
// Copy the content to our output
_, _ = io.Copy(dst, ra)You can finetune the read-ahead for your specific use case, and adjust the number of buffers and the size of each buffer.
The default the size of each buffer is 1MB, and there are 4 buffers. Do not make your buffers too small since there is a small overhead for passing buffers between goroutines. Other than that you are free to experiment with buffer sizes.
On this project contributions in terms of new features is limited to:
- Features that are widely usable and
- Features that have extensive tests
This package is meant to be simple and stable, so therefore these strict requirements.
The only feature I have considered is supporting the io.Seeker interface. I currently do not plan to add it myself, but if you can show a clean and well-tested way to implementing it, I will consider to merge it. If not, I will be happy to link to it.
This package is released under the MIT license. See the supplied LICENSE file for more info.