TimelyDataflow/timely-dataflow

Why exchange by timestamp is not support in 0.7.0 ?

Closed this issue · 1 comments

bmmcq commented

as title.

codes of 'Exchange' in the version of 0.7.0 :

/// Exchange records between workers.
pub trait Exchange<T, D: ExchangeData> {
    /// Exchange records so that all records with the same `route` are at the same worker.
    ///
    /// # Examples
    /// ```
    /// use timely::dataflow::operators::{ToStream, Exchange, Inspect};
    ///
    /// timely::example(|scope| {
    ///     (0..10).to_stream(scope)
    ///            .exchange(|&x| x)
    ///            .inspect(|x| println!("seen: {:?}", x));
    /// });
    /// ```
    fn exchange(&self, route: impl Fn(&D)->u64+'static) -> Self;
}

where is the "exchange_ts" method?

It's not there. Historically, it was added for one use case, and that use case stopped needing it. Arguably, it shouldn't have been added as an exposed method, but prototyped as user code first. My understanding is that if it is important, you should be able to copy/paste it back in locally (the PACT is still present but would need some clean-up). If you would like help with this, let me know.

There were several conceptual issues with the exchange, the main one that I recall being that it isn't clear that timely dataflow wants to commit to timestamps as data so much as capabilities. When e.g. messages may get segmented or boxcar'd, the system may want the ability to conservatively change the timestamp capability, and if this changes the semantics of your program there is a problem. I remember thinking there was a clear case where it had ill-specified semantics, but it isn't coming to me right now.

Anyhow, it should be prototypable as user code, and if there are problems there we should fix up the PACT interface so that it is possible. I would expect it to break again in the future though, especially if we pivot to multiple capabilities per message.