OffscreenCanvas support?
Opened this issue · 6 comments
Is there any plan to support OffscrenCanvas with these new apis?
My read is that the element being placed has to be a "direct child of the canvas element". I think this does limit the api to the main thread, since you cannot pass a reference to an element across to workers, and the OffscreenCanvas, although transferable, is not an element that would have access to the actual <canvas> children.
Is that correct?
I think it would be interesting to support some sort of (even limited) use case for Web workers, such as just supplying dynamic data for an html template / Web Component / DOM Parts, etc.
One of the uses cases I have could be summarized as:
- I have a web component that supplies UI. It could be instantiated on the main thread and attached to the canvas as a child.
- This component is completely isolated from the rest of the page, can be in a shadow root.
- Component adjusts itself visually based on custom props passed to it.
- The values of these props are dynamically computed and updated by a web worker.
- All the features the component needs are already supported by web workers-- except for the ability to render a DOM element.
Besides just moving the cost of the compute for props off the main thread, a requirement is to asynchronously display visual updates decoupled from the main thread animation frame rate. So just a convenience wrapper around postMessage is not sufficient for this use case.
Is it viable to consider something like this in the future? Or does it fundamentally require main thread?
Cheers.
(Apologies, I know I asked this question elsewhere but am failing to find the source now...)
I think you asked the question on the original proposal doc that was circulating.
Anyways, the issue is running style/layout/paint on the worker to determine the display list for the element. I suppose you could somehow pass that display list to the worker, but for the worker to do anything other than just draw it you would need a whole new parameterized display list concept (I think).
Some more thinking on this:
The drawElement method is not supported for offscreen contexts, but there are some things we could do. The attached test case demonstrates something that plausibly could work, and there are other things that could work too. The same functionality might allow support for detached elements also.
In particular, for an offscreen context transferred from a DOM canvas, the offscreen has access to the host canvas and could in theory get the child element from there. I would expect this to work provided the offscreen is not itself transferred to a worker. There is not a particularly compelling use case for this, but it might be worth doing.
For contexts in a worker thread, we have access to the document that owns the JS realm that the worker is running in. There are obvious threading implications to using that document, but in theory it is possible to find the target element in the document and draw it. Or somehow pass the element as an argument to the worker.
Alternatively, we could theoretically construct an entire detached document fragment in the worker context (or maybe not). This would run into all sorts of problems because we can't layout a detached document as far as I know. So maybe just wishful thinking.
This doesn't really solve the OP's problem, but I wonder if it would make sense to split the API up into two parts: taking a snapshot, then importing that snapshot into a <canvas>. The snapshot could be sent across the worker boundary, either as an ImageBitmap or as some other type (maybe a display list, as you mentioned, to move as much work as possible off of the main thread). With this approach, there would also be only one place to add new parameters that might affect the snapshot (Element.snapshot), as opposed to two places in the current proposal (CanvasRenderingContext2D.drawElement and WebGLRenderingContext.texElement2D).
It would be amazing if an offscreen canvas could somehow create their own HTML documents completely separated from the main thread with their own virtual window size either based on the canvas size or user defined. This would allow to create HUDs or project a UI into a 3d world with its own dimension.
It feels a bit limiting that this proposal is based on the page document and does not take a deeper more modular approach that would allow for more advanced use cases.
I work on Whimsical, and our diagramming and white-boarding features use canvas rendering heavily. Our current approach is to render individual board elements (like each shape shown below) via svg in their own offscreen canvases, read back the image, and then composite those images onto a larger HTML canvas element. This lets us update individual pieces without having to re-render every object from scratch.
Because the SVG spec doesn't support text wrapping, we have our own text measurement and wrapping code that we maintain to handle that. This proposed functionality would be fantastic for us, as we're currently having to keep both DOM rendered versions (used when editing) and SVGs-with-custom-text-measurements versions in sync.
Our use case would require offscreen rendering though, having to attach canvas elements to the DOM for an arbitrary number of non-visible renderings of the component images probably wouldn't be worthwhile. We'd also be working with elements that aren't attached to the DOM, so they wouldn't be inheriting styling or context in the same way as elements under an attached canvas element would.
I hope that makes sense, please let me know if you want more context.
Thanks so much for your use case. It's remarkable similar to another that I am aware of, which also uses customized SVG text layout. I have an action item, not directly related to the HTML-in-Canvas work, to explore ways of reproducing HTML text layout and styling in SVG with less effort than is currently required.
Meanwhile I'll also put some more thought into how we can support offscreen HTML-in-canvas.