Panic in bytes_exchange.rs:180 self.buffer.empty().len() >= capacity

[antiguru]

Got the following panic when running timely in multi-process setup. It seems to happen when large amounts of fine-grained data (migration experiment with large number of bins during migration).

thread 'worker thread 0' panicked at 'assertion failed: self.buffer.empty().len() >= capacity', /home/moritzho/.cargo/registry/src/github.com-1ecc6299db9ec823/timely_communication-0.7.0/src/allocator/zero_copy/bytes_exchange.rs:180:9
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:227
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:475
   5: std::panicking::begin_panic
             at /checkout/src/libstd/panicking.rs:409
   6: <timely_communication::allocator::zero_copy::push_pull::Pusher<T, P> as timely_communication::Push<timely_communication::message::Message<T>>>::push
             at ./<panic macros>:3
             at /home/moritzho/.cargo/registry/src/github.com-1ecc6299db9ec823/timely_communication-0.7.0/src/allocator/zero_copy/push_pull.rs:51
   7: <timely::dataflow::operators::input::Handle<T, D>>::flush
             at /home/moritzho/.cargo/registry/src/github.com-1ecc6299db9ec823/timely_communication-0.7.0/src/lib.rs:123
             at /home/moritzho/.cargo/registry/src/github.com-1ecc6299db9ec823/timely-0.7.0/src/dataflow/channels/mod.rs:47
             at /home/moritzho/.cargo/registry/src/github.com-1ecc6299db9ec823/timely-0.7.0/src/dataflow/channels/pushers/tee.rs:24
             at /home/moritzho/.cargo/registry/src/github.com-1ecc6299db9ec823/timely-0.7.0/src/dataflow/channels/pushers/counter.rs:26
             at /home/moritzho/.cargo/registry/src/github.com-1ecc6299db9ec823/timely-0.7.0/src/dataflow/channels/mod.rs:47
             at /home/moritzho/.cargo/registry/src/github.com-1ecc6299db9ec823/timely-0.7.0/src/dataflow/operators/input.rs:260
   8: <timely::dataflow::operators::input::Handle<T, D>>::close_epoch
             at /home/moritzho/.cargo/registry/src/github.com-1ecc6299db9ec823/timely-0.7.0/src/dataflow/operators/input.rs:269
   9: <timely::dataflow::operators::input::Handle<T, D>>::advance_to
             at /home/moritzho/.cargo/registry/src/github.com-1ecc6299db9ec823/timely-0.7.0/src/dataflow/operators/input.rs:322

[frankmcsherry]

I believe this should be fixed by this commit, which resulted from this issue posted through other means.