pipebuffer
A simple command line program for buffering stdin
/stdout
between piped processes when the operating system's pipe buffer is insufficient. Modern linux limits the buffer between pipes in a pipeline to 64kb
, which can cause bottlenecks when working with processes with "lumpy" IO profiles.
pipebuffer
is particularly useful when dealing with large volumes of data and a mix of processes that work in "chunks" of data and more stream-oriented processes.
Usage
To use, you just replace:
foo | bar
with
foo | pipebuffer | bar
or
foo | pipebuffer --size=512m | bar
And, of course, you can use many pipebuffer
s together:
foo | pipebuffer --size=128m | bar | pipebuffer --size=64m | splat | pipebuffer --size=1g | whee
License
pipebuffer
is open source software released under the MIT License.
Building
pipebuffer
is written in Rust and currently requires Rust 1.7 (beta) or greater. You'll need Rust installed. You can download here, or run the first command below:
// Optionally install Rust: if 1.7 is stable, which it should be after ~3rd March 2016, omit '--channel=beta'
curl -sSf https://static.rust-lang.org/rustup.sh | sh -s -- --channel=beta
// Clone the repo
git clone https://github.com/tfenne/pipebuffer.git
// Build and run the tests
pushd pipebuffer && (cargo test; cargo build --release); popd
// Produces executable at ./pipebuffer/target/release/pipebuffer