emk/rust-streaming

EXPERIMENTAL: Various hacks for zero-allocation stream parsing in Rust.

EXPERIMENTAL: Zero-Allocation Streaming Parsers in Rust

Here there be hacks. No APIs are stable. Code may not do what the comments claim.

Key goal:

• Build a StreamingIterator type that can return references to internal state, including as I/O buffers and the output buffers of libraries like flate2. This prevents implementing collect, but why can't we have map, filter and fold?

Target applications:

• rust-csv.
• Multicore map/reduce of Snappy-compressed records.
• Anybody else who needs to iterate over a data stream without allocating.

Random useful things to read:

• Higher-kinded types and why they're important.
• Emulating higher-kinded types with proposed associated types & lifetimes.
• Iterating short-lived objects
• Borrow scopes should not always be lexical (aka, "why we have one line of unsafe")
• Borrow checker gets confused by a conditionally returned borrows (same as above, but clearer)
• Iterator returning items by reference, lifetime issue (Iterator works the way it does for reasons explained here)

We beg for help::

• Can I write zero-copy parsers with Iterator? It looks like it might get me a 47x speedup here.
• Rust struct can borrow “&'a mut self” twice, so why can't a trait?