JS feature support

Question

JS feature support

StoneCypher opened this issue a year ago · 7 comments

Lots of the compelling things one wants to work with in a browser aren't obligatory. For planning's sake, it might be easier for new users if your onboarding documentation mentioned whether facilities were available for developers.

At this time it is not yet clear to me whether it's better to think of Ultralight in the fashion that one thinks of embedding mshtml as a COM control (that is, as a renderer,) or in the fashion of Electron/Tauri (that is, as a platform)

By example, In the MSHTML approach, your storage lives in the exterior C++-or-whatever app, but in Electron, you do your storage with something like the node fs api

I kind of have the impression that it's MSHTML style. But, I should check first, because any of this that's inside JS is suddenly way, way more portable.

For each of these I'm curious whether these things are available, and if so, on which platforms. Some things I'm interested in:

WebGL
fetch, xmlhttprequest, webtransport, or anything similar
The joystick API
The pointer API
The <audio> element or any alternative
The window.localStorage (or .sessionStorage, lol) API
The FileSystem API

I kind of expect most of those to be missing, tbh? But I have high hopes about WebGL, and moderate hopes about the Joystick and pointer APIs.

Thanks for your time

Answer 1 · 2024-01-24T19:07:52.000Z

Hello!

Thanks for the well-written set of questions.

It's best to think of Ultralight as a write-once, run-everywhere web renderer that can be used either:

Standalone (simple web app display, ala Electron)
Integrated into other apps (web UI + other code)

We have a few major goals at this point (warning: long, rambling rationale ahead):

Platform-agnostic and portable.
- We've tried to decouple the library from low-level OS/platform as much as possible. This allows us to port to new platforms/hardware faster and insulate the library from issues involving platform-specific API.
- We've also exposed abstract Platform interfaces to allow users to provide their own implementations for OS-level functionality (file system, clipboard, font loading, etc.) so that the library can be custom-fit for complex integrations (for example, games typically need to sandbox web API access and/or provide their own file system).
Consistent rendering across targets (write once, run everywhere).
- We wanted to support the case where designers could write HTML/CSS/JS and deploy it across all targets and devices without worry that it will perform or look differently. We initially built the library with a pure-GPU renderer but quickly found that this brought a number of platform differences (GPU hardware performance varies wildly, shader support is inconsistent). We've since added a CPU-only renderer (SIMD / multi-threaded) which has closed the gap on rendering inconsistencies without leaving too much performance on the table.
- Other libaries (Tauri, et al) may be lightweight since they use OS WebViews but what you trade for that is tight-coupling on those same OS WebViews which may be different engines or different versions with unknown long-term compatibility.
Easy to integrate and customize.
- I've been working on embedding and modifying browser engines a long time (about 20 years) and have been one of the most ardent supporters of using HTML as a UI language. During that time I've frequently run into the same basic roadblock which is that these engines are primarily designed to build a standalone Browser in which the engine developers have control over the entire stack. Most design decisions are centered around the "browser" use-case and less on the "embeddable web renderer" use-case.
- Out of all of the codebases, WebKit is the least-frustrating to manipulate to this end (it is the OG after all, forked from KHTML) but still has a lot of rough spots with tight-coupling to OS API (designed for iOS, macOS) and multi-process (they've papered over many design issues by throwing process isolation at problems, which we can't always support across all platforms).
- Ultralight is really born out of the basic goal of "wouldn't it be nice if WebKit was actually an elegant, modular web-rendering library that ran everywhere and was easy to embed?"

Regarding storing persistent state, we actually do support HTML5 IndexedDB and LocalStorage APIs-- in the backend we have some SQLite code that synchronizes the state to a local file directory. For more info see Renderer::CreateSession().

You can also, of course, store your own state in the C++/native side and load that in during runtime (call into JS from C++ and vice-versa, we support the full JavaScriptCore C API).

Regarding support for various features:

WebGL
- (not yet, have plans to enable, have to build out an expansion to the GPUDriver interface first)
fetch, xmlhttprequest, webtransport, or anything similar
- (yes, we are current with latest WebKit/JavaScriptCore)
The joystick API
- (yes, we support the full gamepad API)
The pointer API
- (not yet, have plans to enable)
The element or any alternative
- (not officially, video/audio works is enabled in some experimental builds but we are re-working the whole stack for 1.4 and so it's disabled in official builds)
The window.localStorage (or .sessionStorage, lol) API
- (yes, as long as persistent sessions are used)
The FileSystem API
- (not yet, we're building out more platform interfaces to allow users to write / browse local file systems. we do, however, support customizable file:/// access)

Hope this helps, let me know if you need more clarification. 🙂

Answer 2 · 2024-01-24T22:31:16.000Z

Without WebGPU, I have to start to consider sprite performance.

Is your experience that your CPU renderer performs reasonably competitively with Chrome? (I think I saw skia somewhere but tbh I don't remember clearly.)

Do you have a rule of thumb how many sprites one can expect to work on tablets and phones?

Answer 3 · 2024-02-05T00:42:55.000Z

@StoneCypher I understand what you're asking, because I too have wondered about it before. But I think you're misunderstanding what Ultralight is for. Like me, it sounds very much like you're hoping it might be a way to get an html5 game running on desktop/mobile/console, using a faster renderer than Chrome (and thus Electron) provides, with a smaller memory footprint to boot.

In reality, and please confirm @adamjs if this is the case - I don't get the impression it's a suitable match for this at all. When you look at what's missing (audio, pointer events, webgl/canvas/webgpu) it makes it a really quite poor choice for running an html5 game! Unless its rendering performance is so fast that you could handle all the sprites via css/dom manipulation, but I'd be dubious about that, because I really don't think that is what it's meant for.

When you launch, for example, a AAA game like Battlefield 2042 and you activate the in-game character / inventory screens, or pull down the communication rosette to chat to another player - that is what Ultralight feels like it has been designed for. To allow big studio game devs to craft all of that complex UI using the most natural and flexible layout tools possible: HTML and CSS. Ultralight will handle converting all of that into a super-fast texture (or GPU rendering) that the game can overlay as you play. In short, if you need to display complex UI or web content within your game, that is a perfect match.

However, that still doesn't answer the question: just how performant would it actually be at throwing around hundreds or indeed thousands of <div> elements and using the Javascript core to control the whole lot, and barely a line of C at all. I've no idea and would love to know, although I can't help but feel that the licensing model still prices it out of the range of most html5 game devs anyway.

Answer 4 · 2024-02-05T17:32:44.000Z

@photonstorm In my mind Ultralight does have the potential for writing html5 based games that could run on desktop/mobile/console. The most important thing is having webgl/webgpu support. Any missing JS features a game would need could supplemented by just having the C++ code pass up data to the WebView when needed. this would pretty much be the same model as most embeddable WebView frameworks (capactiorJS, Electron, Tauri).

I believe indie game CrossCode was an HTML5 game that got ported to consoles.

Answer 5 · 2024-02-05T20:16:13.000Z

@RealCosmik potential, sure. But it really needs one of the Ultralight team to chime in here. It simply may not be what their business model even looks like (and I'd argue the pricing reflects this)

Answer 6 · 2024-02-05T20:54:52.000Z

Ultralight has everything I need right now except for audio

Answer 7 · 2024-02-05T23:34:05.000Z

@StoneCypher: UltralightCore's CPU renderer is a mix of Skia's rasterizer + custom SIMD compositor + parallel dispatch to make use of all available cores.

It should be fast enough for displaying most web content but, as mentioned in the docs, you might consider using the GPU renderer if you're going to be rendering lots of constantly-animated elements on-screen.

We actually are working on a hybrid compositor in 1.4 that more closely matches Chrome's behavior so you can get the best of both (renders static content via CPU, uses a heuristic to automatically promote certain content to be rendered via GPU).

Re: video/audio, we actually did implement HTML5 video/audio cross-platform via GStreamer/FFmpeg but the Windows port of GStreamer was too buggy for us to ship in official builds. It has since gotten a new maintainer and so this work will be revisited again soon (hopefully re-enabled later this Spring).

@RealCosmik: The library has indeed focused more attention lately on being the best solution to embed HTML inside games and other applications (primarily for HUD and UI purposes), but that's not to say we have decided to cut out WebGL, WebGPU, Video/Audio.

The vision actually is to allow developers the freedom to develop on Chrome and then ship content via Ultralight everywhere (without having to fit some proprietary spec).

The main issue limiting deployment of WebGL has actually been more philosophical than technical-- which is that we strive not to make any deep calls to OS APIs or third-party libs under the hood, instead offering users the choice to handle those calls or let AppCore handle it for you.

The WebGL and WebGPU specs are rather large and so it hasn't been very feasible to offer such interface handlers elegantly. This is an area where we plan to let the standard third-party libraries (ANGLE, GLES) handle it by default (as is typical in browsers) but offer APIs to hook into the implementation (to access GL textures for compositing, for instance).