Improving ergonomics of events with Observable
benlesh opened this issue · 117 comments
Observable has been at stage 1 in the TC-39 for over a year now. Under the circumstances we are considering standardizing Observable in the WHATWG. We believe that standardizing Observable in the WHATWG may have the following advantages:
- Get Observable to web developers more quickly.
- Allow for a more full-featured proposal that will address more developer pain points.
- Address concerns raised in the TC-39 that there has not been sufficient consultation with implementers.
The goal of this thread is to gauge implementer interest in Observable. Observable can offer the following benefits to web developers:
- First-class objects representing composable repeated events, similar to how promises represent one-time events
- Ergonomic unsubscription that plays well with AbortSignal/AbortController
- Good integration with promises and async/await
Integrating Observable into the DOM
We propose that the "on" method on EventTarget should return an Observable.
partial interface EventTarget {
Observable on(DOMString type, optional AddEventListenerOptions options);
};
[Constructor(/* details elided */)]
interface Observable {
AbortController subscribe(Function next, optional Function complete, optional Function error);
AbortController subscribe(Observer observer); // TODO this overload is not quite valid
Promise<void> forEach(Function callback, optional AbortSignal signal);
Observable takeUntil(Observable stopNotifier);
Promise<any> first(optional AbortSignal signal);
Observable filter(Function callback);
Observable map(Function callback);
// rest of Array methods
// - Observable-returning: filter, map, slice?
// - Promise-returning: every, find, findIndex?, includes, indexOf?, some, reduce
};
dictionary Observer { Function next; Function error; Function complete; };The on method becomes a "better addEventListener", in that it returns an Observable, which has a few benefits:
// filtering and mapping:
element.on("click").
filter(e => e.target.matches(".foo")).
map(e => ({x: e.clientX, y: e.clientY })).
subscribe(handleClickAtPoint);
// conversion to promises for one-time events
document.on("DOMContentLoaded").first().then(e => …);
// ergonomic unsubscription via AbortControllers
const controller = element.on("input").subscribe(e => …);
controller.abort();
// or automatic/declarative unsubscription via the takeUntil method:
element.on("mousemove").
takeUntil(document.on("mouseup")).
subscribe(etc => …);
// since reduce and other terminators return promises, they also play
// well with async functions:
await element.on("mousemove").
takeUntil(element.on("mouseup")).
reduce((e, soFar) => …);We were hoping to get a sense from the whatwg/dom community: what do you think of this? We have interest from Chrome; are other browsers interested?
If there's interest, we're happy to work on fleshing this out into a fuller proposal. What would be the next steps for that?
Thanks @benlesh for starting this. There've been mumblings about a better event API for years and with Chrome's interest this might well be a good direction to go in.
Apart from browser implementers, I'd also be interested to hear from @jasnell @TimothyGu to get some perspective from Node.js.
Another thing that would be interesting to know is what the various frameworks and libraries do here and whether this would make their job easier.
From what I remember discussing this before one problem with this API is that it does not work with preventDefault(). So if you frequently need to override an action (e.g., link clicks), you can't use this API.
From what I remember discussing this before one problem with this API is that it does not work with preventDefault(). So if you frequently need to override an action (e.g., link clicks), you can't use this API.
Couldn't this be covered with EventListenerOptions options?
element.on('click', { preventDefault: true })
.filter(/* ... */)
// etc.Cool! I propose to call them DOMservables
I think @annevk is remembering an async-ified version, perhaps based on async iterators. But observables are much closer to (really, isomorphic to) the EventTarget model already, so don't have this problem. In particular, they can call their next callback in the same turn as the event was triggered, so e.preventDefault() will work fine.
It really is just A Better addEventListener (TM). ^_^
(It seems my concern indeed only applies to promise returning methods, such as first().)
@benlesh Can you speak more to why the subscribe(observer) signature isn't valid? That's how all the observable implementations I've seen currently work, but they aren't written in C or Rust.
In web APIs, per the rules of Web IDL, it's not possible to distinguish between a function and a dictionary. (Since functions can have properties too.) So it's just disallowed in all web specs currently. Figuring out how or whether to allow both o.subscribe(fn) and o.subscribe({ next }) is the TODO.
To be clear, the tricky case is
function fn() { console.log("1"); }
fn.next = () => { console.log("2") };
o.subscribe(fn);Which does this call? Sure, we could make a decision one way or another, but so far in web APIs the decision has been to just disallow this case from ever occurring by not allowing specs that have such overloads, So whatever we do here will need to be a bit novel.
This is all a relatively minor point though, IMO. Perhaps we should move it to https://github.com/heycam/webidl/issues.
Another thing that would be interesting to know is what the various frameworks and libraries do here and whether this would make their job easier.
I can't speak for frameworks, directly. Perhaps @IgorMinar or @mhevery can jump in for that, but for RxJS's part, whenever anyone goes to build an app using only RxJS and the DOM, one of the most common things they need from RxJS is fromEvent, which this would completely replace. I would also definitely love to see an RxJS that was simply a collection of operators built on top of a native Observable we didn't always have to ship.
Thanks for the clarification.
It's interesting to me that TC39 and WHATWG both have the ability to add JS APIs, but with different constraints. The TC39 proposal decides what to do based on if the first param is callable. If the TC39 proposal was stage 4, the browsers would be implementing that behavior, right? (Or maybe the TC39 proposal was supposed to be in WebIDL too and violated this. I hadn't heard about that, but I'm not a TC39 member either).
FWIW in practice the DOM already makes the distinction of function-vs-object in the case of addEventListener:
const handle = e => console.log('main!', e)
handle.handleEvent = e => console.log('property!', e)
document.body.addEventListener('click', handle)
// Logs with `main!`(Not suggesting WebIDL can't make the distinction, just pointing out there is a precedent here)
@annevk concerns about the ability to call preventDefault() when using Promise returning methods are valid. Mutation use cases could be addressed with a do method which allows side effects to be interleaved.
button.on(“click”).do(e => e.preventDefault()).first()This method is included in most userland Observable implementations.
@jhusain it could also be handled with map, although it would drive some purists crazy:
button.on('click').map(e => (e.preventDefault(), e)).first()The biggest thing I see missing from this proposal is an official way to use userland operators. @benlesh, do you think pipe is ready to be included?
To the question about frameworks/libraries, as the author of a userland implementation, there are three core types that I'm interested in:
ObservableMemorylessSubject- It's both anObserverand anObservable. Calls to itsnextmethod are passed to all the observers who havesubscribed.Subject- In addition to being both anObserverand andObservable, it remembers its most recent emission and calls new subscribers with it immediately.
If Observable is standardized without the Subjects, userland libraries will still ship them. Still, any standardization is good for the whole ecosystem - it means operator authors know what interface they must implement to be interoperable with the ecosystem.
Furthermore, if Observable is standardized, I expect it will become common knowledge among web authors (just like promises, generators, etc.) That will make it easier for frameworks to depend on it without having to worry about intimidating their users with a steep learning curve - understanding observables becomes part of the job.
@benlesh Using map would be less ergonomic for this use case, because developers would be obligated to also return the function argument.
button.on(“click”).map(e => {
e.preventDefaut():
return e;
}).first()The do method would allow preventDefault() to be included in a function expression, and would consequently be more terse. Given how common this use case may be, the do method may be justified.
@appsforartists Nitpick, but a Subject should not retain its current value. Rather than having a Memoryless subject, I think it should be the other way around: having a Subject that can retain its latest value, like BehaviorSubject in rxjs. Adding functionality on top of the common base, instead of removing it.
Not to bikeshed, but the name of that method seems to keep flipping between do and tap. I don't know the reasoning for the changes, but I suspect tap is easier for users to disambiguate from do blocks and do expressions.
@johanneslumpe Agree that the information architecture is a bit wonky. I think I originally learned Subject has a value, so I think of the other as MemorylessSubject, but that doesn't mean we should standardize that way.
@jhusain If I may, I believe you're nitpicking a bit too much on that point.
First, the benefits and ergonomics of having the listener removed due to the first() operator are much greater than the possible ergonomics lost on preventDefault(). In fact, it's the ergonomics for adding+removing event listeners which would make this API so rich.
Second, calling preventDefault() would probably not happen quite as you mention in your example. I believe it would be more like
button.on('click').first().map(e =>
e.preventDefault();
... do more stuff with event
})
@jhusain I completely agree. I was just demonstrating that if there was a concern over method proliferation, it's possible with existing proposed methods. (And I know about the return requirement, notice the sly use of the comma in my example)
Not to bikeshed, but the name of that method seems to keep flipping between do and tap.
@appsforartists, that's an RxJS implementation thing. Unrelated to this issue.
For Node.js to accept (and implement) this, the on() method name must be changed, given that the Node.js EventEmitter class has on() as an alias for addEventListener(). People who program both for Node.js and for the web will just get more confused between the different behaviors on different platforms.
@appsforartists this proposal is really meant to meet needs around events in the DOM, and happens to ship with a nice, powerful primitive. We should keep it to that and not over-complicate it.
@TimothyGu Node could switch on the number of parameters provided to the on() method. That is, assuming Node wants to do direct integration into their EventEmitter like the web platform does.
Congrats all for bringing this proposal 🎉. I just want to address a point that I think the current implementation of RxJs is missing.
Currently, both Promises and Observables can only be typed on the value, not on the error. And I know talking about types (TypeScript or flow) is a strange thing to do in an Observable proposal, but the underlying reason is a subtle one, and the behaviour of how Observables handle different error situations is paramount.
The problem, as stated here, arrise when handling functions that throws errors, for example
Observable
.of(1)
.map(_ => {throw 'ups'})
.fork(
x => x,
err => err // The error will be the string 'ups', but the type checker can only type it as any
);
We can't avoid to handle these types of error as either your functions or functions that you call may throw, but on the other hand this type of error handling goes against of how we should do it functionally. It would be better to use a chainable method like .switchMap and Observable.throw or in the case of promises .then and Promise.reject.
For that reason, both Promises and Observables can only be typed on error as any and sadly it is the correct type. Luckly I think there are at least two possible solutions, one that is relevant to the Observable proposal.
One solution would be to try catch all methods that may throw and wrap the possible error into a class that extends from Error, for example
class UncaughtError extends Error {
}
which would make the following example possible
Observable
.of(1)
.map(_ => {throw 'ups'})
.switchMap(_ => Observable.throw(new DatabaseError())
.fork(
x => x,
err => err // The error type would be UncaughtError | DatabaseError
);
Note that UncaughtError is always a posibility both if you have a function that throws or not but DatabaseError could be infered from the types of Observable.throw and switchMap.
Very recently I created a Promise like library called Task (WIP) that takes this into account and allow us to type both success and error cases. So far the results where more than satisfying.
The other solution I can think of would be if TypeScript or flow implements typed exceptions, but I don't think is the path their plans.
I hope you take this situations under consideration and thanks for the great work
Cheers!
Node could switch on the number of parameters provided to the on() method.
The IDL proposed in the OP allows a second options-style parameter.
But I'd also like to point out jQuery's on() as evidence for that the name simply has too much historical burden to it.
At the risk starting the bikeshedding wars around the API; would .observe() be a reasonable name for this?
At the risk starting the bikeshedding wars around the API; would .observe() be a reasonable name for this?
@keithamus Certainly! But there's already a huge amount of industry momentum around subscribe as the primary API here. There's also some interesting nuance around what this method does. In all cases, it does observe, but in many cases it not only observes, but it also sets up the production of values. In this particular use case (EventTarget), it's only really observing, so the name "observe" makes total sense, but we would be sacrificing a name that makes a little more sense for other use cases of the Observable type as a primitive.
To be clear, anything other than on would be fine with me. I’ll leave y’all to determine what’s best other than that :)
I don't think we should change the name for Node.js concerns; Node.js already has differently-named APIs (e.g. addListener/removeListener vs. addEventListener/removeEventListener) and the objects passed would be different (Node.js arbitrary, DOM Event instances).
In the end, both jQuery objects and Node.js EventEmitters are different APIs, so can have different entry points without there being a problem. What's more interesting is whether they would make use of the Observable primitive itself, in one way or another.
Btw, any reasoning on why this should be a DOM API and not core language feature?
@TimothyGu Also worth considering: Node and JQuery chose on because it happens to be a really good name for an event-related method. It's almost an argument for why it's the right name. It would be a shame if we didn't consider using it.
@YurySolovyov there is a related proposal in the TC39 to add it as a feature to JavaScript: https://github.com/tc39/proposal-observable/ It's basically in the same shape as this, but this particular proposal is more about improving the DOM eventing API with a proven primitive that matches up with EventTarget well.
@YurySolovyov there's a lot of history here. This proposal has repeatedly failed to advance at TC39 (the committee in charge of JS standardization) precisely because it's not a great fit as a language feature. It doesn't expose any fundamentally new capabilities, it doesn't tie in to any syntax, and by staying at the language level it doesn't integrate well with popular platform APIs that would use it. As such there would be no advantage to putting it in the language, as opposed to just continuing to let people use libraries like RxJS. That's why it's been at stage 1 for over a year, unable to advance to stage 2 where "The committee expects the feature to be developed and eventually included in the standard ".
The committee's feedback was to do precisely what @benlesh has done, and propose working with the DOM community to create a feature worth shipping in engines because it ends up with deep integrations with the platform.
Note that being specified in the DOM spec doesn't prevent it from being used across the different JS-using ecosystems, like Node.js. As we've seen so far with URL, TextEncoder/TextDecoder, and the performance.* APIs, there are a lot of APIs specified outside the core language which are available across many different JS platforms. I would expect Observable to become one of these, and indeed you can see the beginnings of that discussion happening above :).
This proposal has repeatedly failed to advance at TC39 (the committee in charge of JS standardization) precisely because it's not a great fit as a language feature. It doesn't expose any fundamentally new capabilities, it doesn't tie in to any syntax, and by staying at the language level it doesn't integrate well with popular platform APIs that would use it.
Out of curiosity - what was different in promises vs. observables that the former found their way in the core language then? Was async-await already in TC39 minds when promises were getting standardized?
Yep, that and module loading.
It doesn't expose any fundamentally new capabilities, it doesn't tie in to any syntax
Is is out of the question that a native Observable type could be await-ed? I think one of the core advantages of Observable in the first place is that they are an effective superset of Promise.
@rauchg There might be a better venue for that question (e.g. https://github.com/tc39/proposal-observable/), since I think this issue is trying to stay focused on implementor's perspectives on the EventTarget use case.
Observable has been at stage 1 in the TC-39 for over a year now
It's worth noting that the only meeting @jhusain has even attempted to advance the proposal beyond stage 1 at TC39 in the last few years is, according to the agendas, May 2017. There was a non-advancement update in September 2016 regarding cancellation; prior to that, May 2016 and on 3 meeting agendas in 2015.
I don't think it's a fair characterization that "TC39 is going too slow"; the committee has only been given 6 chances in the entire multi-year history of the proposal to discuss it, only one of which asked for stage 2 (it's likely that stage advancement was brought up in more than one meeting, of course; I'm not checking the notes, only the agendas - but either way, it's still not been frequently brought up)
@jhusain are you planning on adding an agenda item in January to talk about Observables, and hopefully advance them?
Observable would be as a core primitive as Promise is, and thus, imo, it should be defined at the language level.
Please spend your strengths pushing TC39 to advance on it.
I don't think the WHATWG should restrain itself from improving the ergonomics of the web platform just because TC39 might, theoretically, be interested in a feature at some unknown future time (despite not being that interested in it for several years).
What exactly is it about Observable that is a language feature that, for example, ReadableStream is not? I think it was absolutely the right choice to design that feature in WHATWG.
@benlesh is it your intention not to define an Observable constructor at this time? Or is that something you want to add immediately?
@matthewp i'm not sure where "not being that interested in it" comes from; many of us are very interested in it. See #544 (comment) - TC39 simply hasn't been given many meeting opportunities to discuss it.
I wish you had chosen to respond to the rest of my comment and not just the side-note that I put in parentheses. I wish I had omitted that as it obviously distracts from the point I was trying to make; that this specification doesn't appear to need language-level support and can be implemented to suit each platform's needs, the same way that ReadableStream is not a EMCA api.
@matthewp Specifically, I think that the language needs a primitive for "multiple temporal values", in the way that an array/iterable is a primitive for "multiple scalar values". Specifically, the language needs an answer to the fourth quadrant in https://github.com/kriskowal/gtor#a-general-theory-of-reactivity ("plural" + "temporal"). I don't feel strongly that an Observable necessarily needs to be it; but one needs to exist, and it would be ideal if, like Promises, future web and node APIs were interoperable with it.
The language has async iterators for that quadrant; those make more sense at the language level, given their integration with async generators and for-await-of. Observables are much more of a library-level feature (including the DOM as a popular library here).
Definition of AbortController, for newbs like me who don't know: https://dom.spec.whatwg.org/#interface-abortcontroller
A few questions:
- Does the Promise return of first() mean that event dispatch to a client that registered with first() takes an extra trip around the event loop? If so, wouldn't that hurt performance? What is the benefit?
- Do the subscription methods that take an Observable potentially have the same issue as 1?
- Could someone provide an explanation of the semantics of takeUntil()? It is not clear to me from the name and example.
- Does the map/filter stuff really provide meaningful syntactic sugar? For example, take this case:
element.on("click").
filter(e => e.target.matches(".foo")).
map(e => ({x: e.clientX, y: e.clientY })).
subscribe(handleClickAtPoint);
What if on() was just a very short synonym for addEventListener instead? Then you could write:
element.on("click",
e => if (e.target.matches(".foo")) {
handleClickAtPoint({x: e.clientX, y: e.clientY})
})
This seems equally clear, and likely would be more efficient, since only one JS function is called per dispatch instead of 3. What's the win?
It's not totally clear to me why the first version is better. If anything, a bigger win to ergonomics would be adding Point Event.clientPoint.
@othermaciej Thanks for the link to AbortController. To see an example of how it's used elsewhere, here's a blog post @jakearchibald wrote showing how abort() works with fetch().
Here are the docs for takeUntil. In short, it aborts the subscription when its argument emits a value. Something like this:
Observable.prototype.takeUntil = function (cancellation$) {
return new Observable(
(observer) => {
const upstreamSubscription = this.subscribe(observer);
const cancellationSubscription = cancellation$.subscribe(
() => {
upstreamSubscription.abort();
cancellationSubscription.abort();
}
);
}
}
}
element.on('mousemove').takeUntil(element.on('mouseup')).subscribe(console.log);would log all the mousemoves until the next emission from mouseup. Then, both the subscriptions to mousemove and to mouseup would be aborted.
@benlesh has written more about this approach. Note: in that article, abort is called unsubscribe.
@appsforartists thanks for answering my question 3. I can see how takeUntil is handy. It makes me wonder how subscription handlers can subscribe themselves conditionally otherwise. Do they need to be a closure capturing the AbortController return from subscribe?
Still interested in the answers to 1, 2 and 4.
Quick note for @appsforartists. Just want to point a common misunderstanding regarding takeUntil as you didn't add it to your code snippet.
It doesn't only unsubscribe but also completes the Observable. takeUntil will trigger the complete callback if present. I think is worth mentioning for accuracy.
Wonderful proposal, to a good degree it also reminds me of the v-stream directive in vue-rx, which was essentially build with my encouragement. Please make this happen and take a look at vue-rx for a directive based reference implementation. Once RxJS commands the DOM your in stream-land and everything opens up. Rock "on"
@othermaciej Glad you found it helpful. As neither a browser implementer nor a champion, I'm trying to leave this conversation to people more-qualified-than-me to have it, but I'll also try to help where I can.
Observable operators and the event loop
If I'm understanding your second question correctly, you're asking if takeUntil needs to wait for a turn around the event loop? No, subscription is synchronous. Here's a simplified implementation of Observable (without the error and complete channels, and without AbortController).
As you can see, subscribe just passes its argument to the function that was provided to the constructor. It's all synchronous, so there shouldn't be any additional passes through the event loop.
Promise-returning operators and the event loop
I'm unfamiliar with the promise-returning functions, so I'll let @benlesh and @domenic speak to them.
Is this really useful?
As for your last question, yes, many of us do find value in this style of programming (as you can see by how many positive reactions have appeared on Ben's proposal just today). I'm afraid that this can very easily turn into a tangent on the value of reactive programming, and I'm not sure if this is the appropriate venue for that conversation, so I'll try to be brief.
I find that reactive programming is more modular, testable, composable, and maintainable than the same logic expressed imperatively. One property of reactive programming is that the data flow through the system is immutably defined at definition time. This makes it easier to reason about as the application becomes more complex (hence, better maintainability). For example:
const movePoint$ = element.on('mousemove').map(
({ pageX, pageY }) => ({ x: pageX, y: pageY })
);will always represent the mouse's last known location in element. Any code that depends on movePoint$ can trust that no other code can change that. Moreover, any code that takes a stream of points can consume movePoint$ without needing to know anything about mousemove.
As a more concrete example, suppose that you had an application that understood PointerEvent and you wanted to add support for TouchEvent. Here's how you might do that with Observables:
function convertTouchEventsToPointerEvents(touchEvent$) {
return touchEvent$.map(
({ type, targetTouches, changedTouches }) => Array.from(
['touchend', 'touchcancel'].includes(type)
? changedTouches
: targetTouches,
({ pageX, pageY, identifier }) => (
{
pageX,
pageY,
pointerId: identifier,
type: TOUCH_TYPE_TO_POINTER_TYPE[type]
}
)
)
}).flatten();
}
function getPointerEventStreamsFromElement(element) {
if (typeof PointerEvent !== 'undefined') {
return {
down$: element.on('pointerdown'),
move$: element.on('pointermove'),
up$: element.on('pointerup'),
cancel$: element.on('pointercancel'),
};
} else {
return {
down$: element.on('mousedown').merge(
convertTouchEventsToPointerEvents(element.on('touchstart'))
),
move$: element.on('mousemove').merge(
convertTouchEventsToPointerEvents(element.on('touchmove'))
),
up$: element.on('mouseup').merge(
convertTouchEventsToPointerEvents(element.on('touchend')
),
cancel$: convertTouchEventsToPointerEvents(element.on('touchcancel')),
};
}
}That's code pulled from our codebase. The ability to add support for TouchEvent to our entire library with just the two functions above demonstrates the power of reactive programming, of which observables are the cornerstone. Unlocking that power in platform APIs like EventTarget is why folks are so excited to see this proposal.
It might be hard to see a meaningful difference between the imperative and reactive styles in the simple examples included in the proposal, but the reactive model has many benefits when writing real applications.
I'm not sure if this is the right place for this discussion, but should we consider how this might look when using event delegation or is there already a simple and robust pattern for handling event delegation to improve performance that I'm not aware of?
I would like to see Observable be the de facto type for events but I'd also like to see a more clear and consistent vision for how it will work with Promises and async iterators.
For the former consider the proposed takeUntil, it seems highly surprising to me that takeUntil is specified to take Observable not Observable | Promise (not sure if WebIDL would even accept that). Plenty of existing APIs (both provided by the browser and custom written) use Promises widely and these often make perfect sense at end conditions.
And it's pretty much just the same thing with interoperability with async iterables, is there going to be large API fragmentation where some utilities only work with async iterables and others only with observables, will we need to write two versions of every operator, one for observable and one for async iterables? Do both promises and async iterables all need to implement the proposed [Symbol.observable] in order to have a good path for interchanging the two.
Now I decided a while ago to experiment with writing a library that can treat event sources as async iterables, it has an API almost borrowed from Observable except written in terms of async iteration and returns an async iterator (that is also iterable) and in my experience I could write the same code as with the Observable form but it was also a lot easier to work with existing Promise based code because it's natural to write operators like .map that also respect async operations.
For example this is an example of a line drawing application using my library:
import Stream from "@jx/stream"
import takeUntil from "./operators/takeUntil.mjs"
/* Stream uses a queue internally to store events, it's slightly worse
than Observable in this regard, but it seems likely that Observable
would need to implement similar in it's [Symbol.asyncIterator] method
anyway
*/
function makeEventStream(element, eventName) {
return new Stream(stream => {
element.addEventListener(eventName, stream.yield)
return _ => element.removeEventListener(eventName, stream.yield)
})
}
/* event is a tiny utility to convert a single event into a Promise */
function event(element, eventName) {
return new Promise(resolve => {
element.addEventListener(eventName, resolve, { once: true })
})
}
/* events is an async iterable of events of the given name from the source
element, because it only creates the stream on iteration it's effectively
equivalent to an unsubscribed observable because it can both be "subscribed"
to multiple times (by looping or other operators)
it can be cancelled via `.return` on the async iterator like other
iterators so this is effectively "unsubscribe"
*/
function events(element, eventName) {
return {
[Symbol.asyncIterator]() {
return makeEventStream(element, eventName)
}
}
}
async function main() {
// Everytime the mouse is clicked down on the canvas
for await (const mouseDown of events(theCanvas, 'mousedown')) {
let previousEvent = mouseDown
// Repeatedly render line segments between the current and
// previous event, (all mouse moves except the first mousedown)
const mouseMoves = takeUntil(
events(theCanvas, 'mousemove'),
event(theCanvas, 'mouseup'),
)
for await (const mouseMove of mouseMoves) {
drawLineSegmentBetween(theCanvas, previousEvent, mouseMove)
previousEvent = mouseMove
}
}
}
// You could even write it using operators as are familiar to
// most Observable users
function main() {
// No side effects or event listeners are added until the for-await-of
// loop just like Observables
const mouseDowns = events(theCanvas, 'mousedown')
const lineStreams = map(mouseDowns, mouseDown => {
const mouseMoves = events(theCanvas, 'mousemove')
return startWith(
mouseDown,
takeUntil(mouseMoves, event(theCanvas, 'mouseup')
)
})
// Pair each element with it's previous inside a single stream
// so that each event is paired with the event we last saw
const eventPairs = flatMap(lineStreams, lineStream => {
return pairWise(lineStream)
})
// This is effectively Observable.subscribe where side effects happen
for await (const [previousEvent, currentEvent]) {
drawLineSegmentBetween(theCanvas, previousEvent, currentEvent)
}
}
main()I just wanted to leave this an example that there's nothing particularly special about Observable's ability to represent composable and lazy async sequences. I want to see how Observable will work with existing types instead of what I often see where everything needs to be hammered to work with Observable and vice versa.
For everyone new here, welcome! I'd like to recommend reading through https://whatwg.org/working-mode and https://whatwg.org/faq to get a sense of how the WHATWG operates.
@WillsB3 lets say you wanted to track all link clicks within a container element:
container.on('click').filter(e => e.target.closest('a')).subscribe(e => {
// …
});Due to the failure of the TC39 CancelToken proposal, the web has a standardized cancellation primitive (AbortController) and JavaScript does not. At least one implementer has already expressed concerns about adding an additional cancellation primitive to the language and consequently to the web platform.
That's what happens when the Web platform goes ahead and defines things not harmonized with TC39 — it ends up impeding TC39 from, eventually, doing it right (the way they would have wanted to or that ultimately would be best for the web).
The same will happen, imo, with Observable if you pursue with this effort independently of TC39 — observables have too many connections with the concepts of events, promises and async to not be tightly integrated with the design of a language.
I also perceive, somewhat, by what I read in this thread, that it is your (of many of you) opinion that the friction and inertia of TC39 is unwarranted. I happen to see it as a great filter and guarantee that things aren't standardized unless they have met/satisfied all technical interests, and maybe even some political ones. Convincing people, fine tuning designs takes time, after all — I think that this is inevitable.
It is taking long, most probably, because it is such a hard problem, in many dimensions (not just, technically, as the standalone API). Just like classes and modules did take long to settle on, and are very hard problems; despite all the difficulties, TC39 finally reached a consensus. The latter are more obviously language features, but, for me, events, and its thin connection with observables are also obvious language features.
This is independent of whether these end up having any special syntax. It is true, imo, even if the result is a bunch of standardized interfaces that every API then seamlessly uses.
Does the Promise return of first() mean that event dispatch to a client that registered with first() takes an extra trip around the event loop?
Think of it like:
function first(el, type) {
return new Promise(resolve => {
el.addEventListener(type, resolve, { once: true });
});
}
first(link, 'click').then(event => {
// …
});Since microtasks are processed per event callback (if the stack is empty), it's all within the same turn of the event loop. This also happens before unsetting the event's passive listener flag, so preventing default should be fine for spec-dispatched events (cc @annevk).
Events dispatched via JS won't be cancellable from .first and other promise-returning methods, as the microtask checkpoint is skipped because the stack isn't empty. This could become a bit of a gotcha.
Does the map/filter stuff really provide meaningful syntactic sugar? For example, take this case:
I think the main benefit is in providing an observable that's pre-filtered to another piece of code. I agree that there isn't much benefit in the one-liner.
@othermaciej I think most of your questions have been answered by now, but I'll try and summarize my responses quickly.
What is the benefit of adapting Observables to Promises?
The primary benefit of adapting Observables into Promises is that makes it possible to await events inside of async functions. Consider the following example which collects a signature on a canvas:
async function getSignature(signatureCanvas, abortSignal) {
await.cancelToken = cancelToken;
const toPoint = e => ({ x: e.offsetX, y: e.offsetY });
const sigMouseDowns = signatureCanvas.on('mousedown').map(toPoint);
const sigMouseMoves = signatureCanvas.on('mousemove').map(toPoint);
const sigMouseUps = signatureCanvas.on('mouseup').map(toPoint);
while(true) {
let lastPointClicked = await sigMouseDowns.first(abortSignal);
await sigMouseMoves.takeUntil(sigMouseUps).
forEach(
point => {
strokeLine(signatureCanvas, lastPointClicked.x, lastPointClicked.y, point.x, point.y);
lastPointClicked = point;
},
abortSignal);
}
}Does the map/filter stuff really provide meaningful syntactic sugar?
Its worth noting the same question could be asked of Array’s map and filter methods. The value of map/filter is not necessarily that using these methods is always terser, but rather that they enable composition. This means that one module can filter an Observable, and another that accepts that filtered Observable and map over it, and so on.
Events dispatched via JS won't be cancellable from
.firstand other promise-returning methods, as the microtask checkpoint is skipped because the stack isn't empty. This could become a bit of a gotcha.
fwiw this is a pretty big deal, especially now EventTarget is constructible, we'll see a lot more event dispatches from JS.
@jakearchibald Sync dispatch in userland EventTargets is already a pretty big footgun IMO (ie reentrancy). There are already good reasons why EventTarget vendors should wait until the job queue is drained before looking for a cancellation. Perhaps this can be encouraged by providing a convenience asyncDispatchEvent method on EventTarget that returns a Promise? Failing that, the aforementioned do method will allow developers to synchronously cancel the minority of EventTargets that do sync dispatch (#544 (comment))
There are already good reasons why EventTarget vendors should wait until the job queue is drained before looking for a cancellation
What reasons? I'm only aware of the microtask one.
Besides preventDefault(), events also have cancel() to prevent further propagation. It seems like the Promise issue could interfere with that too (assuming that is a useful feature that we still want).
Are Observers intended to have the Promise-like behavior of always running the callback at micro task time?
stopPropagation and stopImmediatePropagation will have the same issues if the JS stack isn't empty.
Are Observers intended to have the Promise-like behavior of always running the callback at micro task time?
No, the non-promise callbacks are sync.
el.on('click').subscribe(() => console.log('One'));
el.on('click').first().then(() => console.log('Three'));
el.click();
console.log('Two');
// Logs "One" "Two" "Three"@jhusain the async/await code looks neat. Seeing that code and in light of @jakearchibald's comments, it's too bad there isn't a way to use async/await without having to defer the callback to microtask time.
On map/filter, I think they make a bigger difference for arrays, because they let you not write out a loop at all. But in the case of Observable, the looping is implicit whether you use map/filter or your mapping and filtering right into your callback.
On the composition angle, can you point to some examples of app code using userspace implementations of Observable that pass around mapped and/or filtered Observables and subscribe them somewhere else? I think I'd have to see an example to understand why this is useful.
It's interesting so many people are focusing on first(). I guess that makes sense given that it was a prominent example. But it isn't really as big a part of the proposal as you might assume.
Rather, I think of it in terms of the observable <-> array analogy. Arrays have a few combinators that produce other arrays, such as map, filter, flatMap, and slice. They also have other combinators that produce scalar values, such as a[i], every, find, includes, and reduce. first() is just a particular case of the a[i] combinator, translated to observables. Indeed, it's really a special case of the observable.at(i) combinator, which is omitted here because in practice it's not very useful apart from observable.at(0), for the common cases where you end up with only a single event (or other object) in the observable. So instead what's proposed is observable.first() directly.
It's true that once you convert your observables to a scalar values, you enqueue a microtask. But by that time you're usually "done", and not processing these things as events, but instead as scalar values converted from events. This is more obvious with cases like
// Find the maximum Y coordinate while the mouse is held down.
const maxY = await element.on("mousemove")
.takeUntil(element.on("mouseup"))
.map(e => e.clientY)
.reduce((y, soFar) => Math.max(y, soFar), 0);So I at least am not too worried about first(); it just is part of this general family. I think most people will understand that once you await something, you're treating it more as something that happened in the past, not something that is ongoing and you have a chance to cancel or stop propagation of.
Besides dispatching custom events by hand, I'm pretty sure there are Web Platform APIs that will synchronously dispatch an event from a method called from JavaScript. For example element.click() or element.focus(). In this case, the micro task queue won't be drained until after returning to the outer level, so preventDefault() and stop[Immediate]Propagation() won't work properly. In addition, timing may be weird, if an event listener was counting on happening before the default action is performed. For example, if you click() a forms submit button, your Promises won't get to run before the form is submitted so they can't last-minute update form fields.
Additional example: @jakearchibald mentioned on IRC that abortController.abort() causes the "abort" event to fire on the related signal without a turn of the event loop too.
Just to reiterate, I don't expect the primary way for people to use observables to subscribe to events to be promises. That is only when you want to convert into a promise-using ecosystem (e.g. service worker's waitUntil). Most of the time you'll just do e.g. element.on("click").subscribe(e => e.preventDefault()).
As an implementer, I'm focusing on things that could be hard to implement, that could cause performance issues, or that may cause weird behavior inconsistencies that are not obvious. The Promise-returning methods seem to be the most likely to cause strangeness because of the tricky timing rules for when Promises fire. That's true even if they won't be the most popular part of this API. The challenge with Promises in the context of event propagation is that they are sometimes "things that happened in the past" but other times they are happening in the middle of event dispatch just where you expect.
Because the syntax is concise, I expect people will want to use this even if they don't intend to do sequence operations on the event stream.
For example, an author might reasonably expect form.on("submit").first(e => /* do stuff that alters form fields before submission */) to work, and it will sometimes work, but other times it won't. And on the flip side, it's also not safe to assume in that first() callback that form has already been submitted so it's safe to make post-submission changes. Either could be true. It depends on whether the form was submitted from JavaScript or directly by a user action.
Is there a way to avoid the indeterminate timing?
Is there a way to avoid the indeterminate timing?
All the ways I think of are pretty deep-cutting, because this is a preexisting problem. For example:
form.addEventListener("submit", e => {
console.log(1);
Promise.resolve().then(() => console.log(2));
});In this case if the user clicks submit, this will always log 1, then 2, with no JS code running in between them (besides potentially other enqueued microtasks). Whereas if you do
form.submit();
console.log(3);then this will log 1, then 3, then 2.
We could "fix" this case, and thus also the observable case, by e.g. inserting a microtask checkpoint after all event firing, even if the stack is not empty. That breaks a lot of assumptions about promise-using code though, that promise callbacks only run with an empty stack.
My and @jhusain's take is instead to say that this existing feature of the platform is reasonable already, or at least something we'd want to solve separately. Instead, if we really believe that first() is a footgun, we can provide mitigations like allowing el.on("submit").do(e => e.preventDefault()).first().then(doMoreStuff) or (new proposal) el.on("submit").first(e => e.preventDefault()).then(doMoreStuff). In both cases the idea is that when converting to a scalar you get a chance to intercede and do something sync before going into async-promise-land.
As far as practical debugging, browsers should at least issue a warning when code calls event.stopPropagation() or other dispatch-changing methods on a "stale" event that has already been fully dispatched. There's already such advice in the spec for passive but really it would be better for that term to have been reserved for an event that affects neither the default action nor propagation. In that sense, today's passive is more like "passive aggressive". 😸 I wonder if there's any benefit of having an EventOptions flag indicating full passivity.
I'm worried about the cases where people will use this for purposes other than converting to a scalar. form.on("submit").first().then(myFunc) looks nicer than form.addEventListener("submit", myFunc, {once: true}). It's a lot more appealing to write than sticking Promise.resolve().then() into your event handler function. But it's not obvious from the syntax that it may create weird timing issues. I totally believe that thoughtful web developers can use this well. But if it becomes the cool new best way to handle events all he time, then its use will not be limited to thoughtful web developers.
As far as where to fix this, if we cared to: there are other options than altering the timing of microtask dispatch in general. For example, one could instead imagine creating a new kind of "eager Promise" that is usable with async/await and other Promisey things, but which calls the then() callback synchronously instesad of at microtask time when resolved. Using that instead of a regular Promise would remove the timing hazard without messing with all uses of microtasks. It's not clear to me that the deferral to microtask time is doing any work here. Alternately, one could imagine that this form of Promise is still always async, but gets extra opportunities to run at times we wouldn't fire other microtasks when other JS is on the stack.
(Apologies if I have butchered Promise-related terminology or concepts here).
Yet another option: create an "extra lazy Promise" that will never be processed before the end of event dispatch. That also provides consistency, regardless of whether JS is on the stack. (Though it is probably a less useful behavior.)
Note: I'm not saying these changes are necessarily worth it. Just trying to think through whether the consistent timing issue is patchable.
It's not clear to me that the deferral to microtask time is doing any work here.
Well, it's guaranteeing that promise reactions only run on an empty stack, which is a fairly important invariant that's ingrained into a lot of promise-using code. I personally don't think that introducing a new promise type whose timing is different from all others is a good idea; I think it increases the chances of unexpected behavior, instead of decreasing it.
If you really think first() is an attractive nuisance whose downsides outweigh its benefits, then perhaps we should just take that as implementer feedback and remove it, but I think it would be a shame, as I see lots of useful ways to use it which won't ever encounter this problem.
I'm not sure if the downsides outweigh the benefits. It should be pretty obvious that I'm not familiar with this API, so I'm not in a position to give a considered opinion yet.
I do note that this issue is titled "Improving ergonomics of events with Observable", which implies to me that it's meant to make things easier for everyday use of events by typical developers, not just for advanced use cases where it is possible to be careful and get things right. Based on that, I expect it needs to be designed to be minimally dangerous to non-expert web developers. In that context, "there are some use cases where it's convenient and safe" is not a good answer to "this seems to create the risk of unexpected or unpredictable behavior".
There might be ways of getting events into the Promise ecosystem that create less risk of mistakes. Even something as simple as giving first() a different name might do it. Alternately, if this API is meant only for expert use, there's the possibility of giving on() a more off-putting name like observableForEvent().
It might be better if first is named toPromise. This is how we handle conversion of observable to promise in RxJS (although the implementation is a bit different). It's useful as an interop point for APIs that expect promises, such as async-await, however the name is either just unattractive enough or just specific enough that people don't often use it as a means to subscribe to observable unless they actually need it. That wouldn't put us in much different of a situation than someone organically using Promise.resolve inside of an addEventListener. The name is fairly explicit, so people will definitely know they're dealing with a Promise.
Then again, the presence of a then method is a solid cue that a Promise is being used, and that microtask scheduling is at play. So the behavior will not likely be too surprising to anyone familiar with promises.
I am with @jhusain that the promise-related methods are not the most important part of this proposal. However, it would be really nice to get some interop for async-await and other api's that expect promises.
Should takeUntil() accept AbortSignals as well?
(Sorry if this was already brought up.)
@matthewwithanm you could use takeUntil(signal.on('abort'))
@benlesh that misses the case where the signal has already aborted, but I guess that's easy to work around?
In terms of first(), @domenic's callback idea works, or it could return an observable:
el.on('submit').first().subscribe(event => …);It could be a thenable observable so it works with await? Maybe that's too hacky.
maybe for awaiting there could be a .promise() method?
await el.on('click').subscribe().promise();or even
await el.on('click').promise();@othermaciej this is perfectly legal (and seems like it just might work) syntax though:
myDiv.addEventListener('click', async (e) => {
await loadMyWebsite();
e.preventDefault();
}, { once: true });I feel that's a way bigger (existing) footgun than .first() would be. Though I also agree that if you pass a function to .first(<fn>) it should be called sync just like the other observable methods, so that you explicitly have to do .then(<fn>) to get into the microtask queue.
It’s likely that we can’t provide APIs which allow Observables to be adapted into Promises without introducing footguns. Observable and Promise have very different semantics around dispatch (for good reasons), which creates impedance mismatches.
Champions believe that there are compelling use cases to adapt Observables to Promises, and developers will likely do this anyway. However there is an argument to be made that making adaptation more arduous increases the likelihood that it will only be used when necessary. For example, removing first() will increase the likelihood that developers don’t cavalierly adapt to a Promise instead of using obs.take(1).subscribe(...) when the latter is more appropriate. If we accept this argument, we have the following options:
- Remove Promise-returning methods from the Observable prototype.
- Modify the Promise-returning methods to return Observables which notify only once.
Personally I favor the second. Developers are already used to the idea of subscriptions which notify once (ie { once: true }). However I think there are valid arguments to be made that developers might be confused why functions that return scalars values on Array return a vector type on an Observable. This concern has been expressed when the option was raised in the past.
@jakearchibald Note that making the Observable returned by first awaitable is still a footgun, because scheduling will still happen on await.
@benlesh that misses the case where the signal has already aborted, but I guess that's easy to work around?
@jakearchibald In that case you could either synchronously check the signal prior to subscription, or write a simple adapter around it using the Observable constructor. If you're interested in what what might look like, let me know.
Note that making the Observable returned by first awaitable is still a footgun
@jhusain is correct here. Because Promises are always eager and Observables will lazily start observation, Observables can have side-effects. Which would mean that calling a "then" directly on an Observable would mean creating side-effects, when in Promise's case it would never do that. It would be very surprising behavior for developers.
Modify the Promise-returning methods to return Observables which notify only once.
Personally, I like this idea as it already exists in this form in RxJS and I know it works out for most use cases. I still think we should have a more obvious method like toPromise as well though, because I really believe we should try to provide interop points where we can. It makes the type more powerful.
Oops... accidental close, just missed with the mouse, sorry.
I'm definitely not an expert here. My gut feeling is that adding primitives by introducing new browser APIs might be a challenge performance wise later and might lead to (potentially unnecessary) copy&paste on the Node side later. Would it make sense to resurrect the idea of providing Observable primitives on the VM layer and have browser and Node specify APIs in terms of those primitives? Much like with Promises?
@bmeurer #544 (comment) explains it somewhat. I would love to hear more about this as well, but I do feel like the discussion on where to spec this should be moved to a separate thread somewhere.
@mathiasbynens not sure if that was brought up, but strictly speaking, it does not have to be syntax-driven feature, just adding new global/API to spec might work fine too, like with Promise
@mathiasbynens I'm aware of that, and I remember some discussions in the V8 team whether or not it makes sense to have Observables as primitives, similar to Promises.
I'm not speaking as a V8 representative or some TC39 implementor here. Just my own personal opinion:
When we say that Observables are perfectly fine as a user land feature and have a single, canonical implementation of that primitive, i.e. RxJS, that makes total sense to me. However adding Observables as primitives to the platform via some embedder API (be it something in the browser or something in node, or both in the end) doesn't seem like it's the best approach long-term, as you might end up with a lot of copy and paste, and due to performance constraints you'll probably want to have the core part implemented inside the JavaScript engine eventually, which is not covered by the TC39 specification then.
So my question is really: If we think it is useful as a native platform feature, shouldn't the primitive then end up in ECMAScript, and have platform APIs (node/browser) define more complex logic in terms of the primitive? Like we did with Promises? What makes Observables as a primitive different from Promises in this regard?
@bmeurer the difference between the two cases was discussed upthread, as @mathiasbynens links to.
@bmeurer there's already precedent for features defined outside TC39 ending up in the JavaScript engine, e.g., https://streams.spec.whatwg.org/. I don't think that has resulted in a lot of copy-and-paste.
One difference (though not sure if this is relevant): Promises require special platform hooks to implement correctly due to the timing requirements on resolving a promise. But Observables don't. They would have almost exactly the same behavior whether implemented in the JS engine, in the browser engine (or other JS hosting environment), or in pure JS.
(To be clear: I have no particular opinion on whether they should be specified in an ES-like way or a DOM-like way, what venue this should be done in, or if they should be standardized at all.)
It's time for another rookie question. Someone posted a link to a 2x2 chart like this:
spatial temporal
scalar value Promise
vector Array ???
Where Observable would fill in the missing concept. Why isn't ReadableStream the missing concept? I can see many differences in details of the API, but what is the difference conceptually between a Stream and an Observable?
@othermaciej The primary difference would be that ReadableStream, AsyncIterator, et al, are pull-based, where Promise and Observable are push-based. Certainly ReadableStream and AsyncIterator would fall into that generic quadrant, but it doesn't mean they can be used in the same way. I'd almost argue that Promise misses the mark in the "temporal" space, because it isn't able to act synchronously, only asynchronously, therefor not all timings could work with it as a primitive. Observables, on the other hand, are much more primitive and do not prescribe any scheduling for values emissions. ReadableStream and AsyncIterator are implemented on top of promise, meaning that they're also tied to the same scheduling restrictions. Restrictions that make them not viable as an abstraction for eventing, as events can be subscribed to, dispatched and torn down synchronously.
@annevk Streams didn't end up in the JavaScript engine. Intuitively I'd have used that as a counter example.
@domenic I probably misread that comment. Let me ask instead: Once you have hooked up Observables to the DOM, would you go back to TC39 and propose the primitive (based on the experience with DOM) to JavaScript again?
Streams didn't end up in the JavaScript engine. Intuitively I'd have used that as a counter example.
This is likely because of AsyncIterator, which is basically the same thing (pulling promises out of an API), only more primitive. (I'm speculating)
It's time for another rookie question. Someone posted a link to a 2x2 chart like this:
For more detail, see https://stackoverflow.com/questions/39439653/events-vs-streams-vs-observables-vs-async-iterators/47214496#47214496
Streams didn't end up in the JavaScript engine.
They did in Gecko and, I believe, WebKit. They arguably did in Blink, although via the abstraction that is V8 extras, which you might not count.
would you go back to TC39 and propose the primitive (based on the experience with DOM) to JavaScript again?
I don't see any motivation for doing this, perhaps because I don't view them as a primitive.
They did in Gecko and, I believe, WebKit. They arguably did in Blink, although via the abstraction that is V8 extras, which you might not count.
WebKit implements Streams at the browser engine (WebCore) level, not the JS engine (JavaScriptCore) level. We do try to make them look more like JS builtins than other bindings do, though perhaps not with full fidelity. This is in part because the having an ECMA add-on spec is unusual and mildly confusing; and in part because nothing else is implemented as a DOM binding that's supposed to look like a JS builtin.