DISTRHO/DPF

WebView UI: More customization?

Opened this issue · 30 comments

sevenc-nanashi commented

I'm porting my VST with choc's WebView into DPF.
I used bind, navigate, and fetchResource in that VST, but it seems like I can't use those APIs in DPF's WebView UI.

It would be nice if those were accessible.

falkTX commented

The webview for DPF is still very primitive and work in progress.
I dont plan to add such features though (except maybe navigate), the webview is meant to be as simple as possible and use the same APIs as a regular UI does.

The webview wont talk to the plugin side directly but instead use state changes to notify host and plugin side that something happened.

I think juce allowing "bind" functions is misguided, it comes from how they always have UI and DSP coupled together but on DPF side we always meant to separate them as much as possible.
The "fetch resource" I might still change my mind on, but since we can load local filesystem assets it does not seem as important to me. That is, one can fetch the plugin bundle location and use "file://" style URLs to reference local file resources.

lucianoiam commented

Great to see official webview UI support is coming to DPF, a couple of months ago I scaled down my project, made it ISC instead of GPL and renamed it to something that hints at DPF; the fancy name was unnecessary and regret it.

  • How do you handle the GTK webview on Linux? I had to create a helper binary making the Linux version 2x more complex than the other platforms, but maybe this dont-link-plugin-to-gtk rule is outdated?

  • Is the scope described somewhere? as a recap my implementation started with file:// support, then added http, then added remote UIs by adding a websockets server for state & parameter messaging. Network messaging would be cool to have, it means distributed UIs for free on the client side.

  • I see JUCE released something similar but seeing fugly names like WebComboBoxParameterAttachment and WebSliderRelay.

  • ❤️1
falkTX commented
  • How do you handle the GTK webview on Linux? I had to create a helper binary making the Linux version 2x more complex than the other platforms, but maybe this dont-link-plugin-to-gtk rule is outdated?

there is some fancy silly things being done on the Linux side, where we create a specially executable shared object if building as plugin. that way we do not need to ship with external tools, the only "tool" needed is ldlinux to make the magic work.
also everything related to gtk (and also a Qt5 and Qt6 implementation) is not linked into the plugin but uses dlopen/dlsym.

but similar to your implementation there is IPC to talk between native UI class process and the new child process that does web things via gtk/qt.
not everything is complete but the important building blocks are in place.

  • Is the scope described somewhere? as a recap my implementation started with file:// support, then added http, then added remote UIs by adding a websockets server for state & parameter messaging. Network messaging would be cool to have, it means distributed UIs for free on the client side.

not really, it was initially just an experiment made for https://github.com/moddevices/mod-desktop which needed a webview inside the plugin view.
then slowly making it work generic enough so that it could be ported over to DPF.

still dealing with some life things so no time for this right now, to be continued later in the year

  • I see JUCE released something similar but seeing fugly names like WebComboBoxParameterAttachment and WebSliderRelay.

Yeah we will have no such things on DPF side.

my idea so far is simple: web views will have the same exact API as the DPF UI.
So they have stuff like parameterChanged and stateChanged callbacks for when stuff changes on DSP side, and can call setParameter, editParameter and setState to change stuff on DSP side.

That is how far it will go, intentionally. That way the implementation can be done in a quick way, and one that mimics existing APIs and (the eventual) documentation for those.

Some basic goals:

  • I dont want to have network code inside DPF (besides the liblo stuff needed for DSSI).
  • We need to support Linux in a way that does not involve linking to gtk3, and allows more than just gtk3
  • Windows is a mess, let CHOC do that for us, but patch it so it comes with the minimum baggage possible
  • webviews should able to be usable standalone as a regular widget on top of GL or cairo (with of course the limitation that they are an OS level above, so you cant paint on top of webviews)
  • advanced features can be supported if there is a good reason to do it, but those wont be exposed when using webview as direct plugin UI (they are reserved for when using webview as a child widget)
  • Minimum Linux/mac/windows support, support for BSD would be nice but not a must for now (in theory the same ldlinux magic is possible on BSD, but didnt look into it)
  • no use of external build tools, like a gtk wrapper
  • no extra dependencies should be needed (which is the case right now)

PS: if someone wants to build a real (and opensource) plugin using webviews I would gladly collaborate on that. I would do the build system and project maintenance, but leave the web graphics to you.

lucianoiam commented

there is some fancy silly things being done on the Linux side, where we create a specially executable shared object if building as plugin. that way we do not need to ship with external tools, the only "tool" needed is ldlinux to make the magic work. also everything related to gtk (and also a Qt5 and Qt6 implementation) is not linked into the plugin but uses dlopen/dlsym.

but similar to your implementation there is IPC to talk between native UI class process and the new child process that does web things via gtk/qt. not everything is complete but the important building blocks are in place.

Cool, need to do some research it looks like my stuff can be simplified on Linux.

  • Is the scope described somewhere?
    not really, it was initially just an experiment made for https://github.com/moddevices/mod-desktop which needed a webview inside the plugin view. then slowly making it work generic enough so that it could be ported over to DPF.

still dealing with some life things so no time for this right now, to be continued later in the year

Thanks

  • I see JUCE released something similar but seeing fugly names like WebComboBoxParameterAttachment and WebSliderRelay.

Yeah we will have no such things on DPF side.

🙏🏻

my idea so far is simple: web views will have the same exact API as the DPF UI. So they have stuff like parameterChanged and stateChanged callbacks for when stuff changes on DSP side, and can call setParameter, editParameter and setState to change stuff on DSP side.

That is how far it will go, intentionally. That way the implementation can be done in a quick way, and one that mimics existing APIs and (the eventual) documentation for those.

100%. Barebones interface and UI/DSP decoupling then the programmer can build any desired abstraction on top of it. Web should mimic the API as much as possible.

Some basic goals:

  • I dont want to have network code inside DPF (besides the liblo stuff needed for DSSI).
  • We need to support Linux in a way that does not involve linking to gtk3, and allows more than just gtk3
  • Windows is a mess, let CHOC do that for us, but patch it so it comes with the minimum baggage possible
  • webviews should able to be usable standalone as a regular widget on top of GL or cairo (with of course the limitation that they are an OS level above, so you cant paint on top of webviews)
  • advanced features can be supported if there is a good reason to do it, but those wont be exposed when using webview as direct plugin UI (they are reserved for when using webview as a child widget)
  • Minimum Linux/mac/windows support, support for BSD would be nice but not a must for now (in theory the same ldlinux magic is possible on BSD, but didnt look into it)
  • no use of external build tools, like a gtk wrapper
  • no extra dependencies should be needed (which is the case right now)

Could you elaborate on the reason for #1 ? all points are centered about minimalism which is desirable and I fully agree, but if there is an exception for DSSI there can be a second exception for the webview? I think webviews and network are highly intertwined and when used together lots of additional things can be done for almost free, which would be very hard/complex otherwise.

Naturally, (nearly) everything optional can be built as a layer on top.

PS: if someone wants to build a real (and opensource) plugin using webviews I would gladly collaborate on that. I would do the build system and project maintenance, but leave the web graphics to you.

Nice!

In my opinion webviews are not only useful for plugins with very complex UIs, but also open the door to programmers outside to the audio community for contributing which is cool.

falkTX commented

Could you elaborate on the reason for #1 ? all points are centered about minimalism which is desirable and I fully agree, but if there is an exception for DSSI there can be a second exception for the webview? I think webviews and network are highly intertwined and when used together lots of additional things can be done for almost free, which would be very hard/complex otherwise.

I just dont want to have to maintain network related code. the liblo dep was okay because:

  1. DSSI is only relevant (if at all) on Linux systems, so the dependency is more acceptable
  2. pretty much all DSSI plugins also use liblo, so having the similar codebase is nice
  3. I also use liblo in Carla so I am already used to it and its API

Writing something like a websocket is out of scope for DPF due to its complexity, but bringing a whole dep just for it also seems overkill.
The plugins that want it can deal with that themselves, and leave the framework to remain simple.

  • 👍1
sevenc-nanashi commented

I dont plan to add such features though (except maybe navigate), the webview is meant to be as simple as possible and use the same APIs as a regular UI does.

I see;
(It would be nice if there is a HWND/NSView/XView Component like juce does, btw)

falkTX commented

I dont plan to add such features though (except maybe navigate), the webview is meant to be as simple as possible and use the same APIs as a regular UI does.

I see; (It would be nice if there is a HWND/NSView/XView Component like juce does, btw)

that is already possible in the develop branch if you include "extra/WebView.hpp" and set DISTRHO_UI_WEB_VIEW to 1 in the plugin config header.

API is in https://github.com/DISTRHO/DPF/blob/develop/distrho/extra/WebViewImpl.hpp
So you would do something like...

WebViewHandle handle = webViewCreate("http://127.0.0.1:8080/", getNativeWindowId(), 500, 300, getScaleFactor());

Then call webViewIdle at regular intervals.
The webview create call has optional settings to pass a function callback, so the javascript side is able to talk back to the C++ side, the global JS function postMessage is used for this, uses a single string as argument.
The reverse is already possible with the webViewEvaluateJS API.

  • 👍1
sevenc-nanashi commented

Thanks!

Are there access to something like fetchResource, which creates dummy host (juce.backend, choc.localhost iirc) for some secure-only apis?

falkTX commented

as already mentioned, no. and there might never be, still debating on that.
what is the usecase for this? arent you able to just load files from the plugin bundle? is having the files locally accessible that big of a pain point?

sevenc-nanashi commented

Oh I found that file protocol is considered as safe, but there are some problems that is not solved:

Premise: I'm porting a electron+vite app into VST, and I'd like to use DPF because of license (GPL is suck).

Reasons:

  • Our electron app uses vite, and vite uses / for asset urls.
    • Requesting / on file:// protocol requests to drive root.
  • I'd like to zip assets to reduce the size of my plugin.
    • My app includes large assets such as fonts.
falkTX commented

you can zip the contents on the installer and still have the full size when installed. when dealing with browser-based plugins I think disk space is better used vs more ram having to load things in memory every time (browsers tend to be memory hogs, best to not make it worse)

cant really about the file:// stuff, but I suspect web frameworks tend to have a way to deal with this.

sevenc-nanashi commented

iirc vite has option to use /something instead of /, but I don't think that it can use relative urls. (as every html is mounted on static route in most cases)

Few more things:

  • Our electron app also assumes the root is always /, and I'd like to avoid editing existing code as much as possible.
  • Simply, I'd like to avoid file:// protocol (in other words: I'd like to avoid allowing access to outside of assets)
falkTX commented

ok fair, but it is not really something I have in mind to support for a first MVP.
after everything is in place and verified to be stable and working, perhaps we can see about extending to support that.

you can try to contribute to make this happen, which then doesn't depend on me doing it.
roughly speaking we need:

  • a base API that doesn't rely on new C++ features for its declaration (so a regular callback and not lambda function)
  • code adapted to dpf that works on Linux (gtk3 and Qt), macOS webkit and windows through CHOC (I removed that part from the header file, would need to be reimported with that on)
  • an example plugin that makes use of this, so we can easily test it
  • 👍1
lucianoiam commented

file:// just won't do it for lots of modern web stuff, though it suits basic hand written JS well, a.k.a. small plugins with basic UIs. But then having a webview is overkill in those cases.

falkTX commented

wouldnt that then imply that there is a webserver running?

lucianoiam commented

Yes, in fact found this myself the hard way. I was hoping for file:// to be enough but it turns out some things are artificially restricted because of security (eg whatwg/html#8121), and for some of the webviews I couldn't find all the needed workarounds (cannot recall exactly now).

The same libwebsockets library provides sockets and file server, it is plain C, mature and actively maintained.

I would expect "complaints" about this over and over...

falkTX commented

I looked a bit on how to do this on Qt side, since I need to get that updated first anyway.
Looks doable to have a callback for web side requesting access to arbitrary resource files. Because the reply is on native code we can even give back dynamic runtime data to the browser. I like that this does not involve any HTTP or network code at all, it is just functions and passing data around, the browser receives it as if it was a regular network request.

So I am more open to this idea, seems sensible enough. But I would leave it up to the plugin C++ code to decide what data to return, maybe only having a convenience class/code to return data based on a filesystem path but anything else would need to have custom code written by the developer.

One thing I am not 100% sure of, because I only investigated the Qt side so far, this works by having a custom URL scheme handler on JUCE side too right? Or is it a custom domain that redirects to local resources?
I am fine with having like dpf://myfile.html but not with http://specialdpfdoman/myfile.html
The 1st is quite simple and obvious that is a local resource, the 2nd feels hacky and prone to issues

If we are ok with the custom URL scheme, I would try to get an API for it after finalizing my current Qt update.

  • 👍1
sevenc-nanashi commented

I looked into choc'c implemetation.
It looks like:

I think it's better to have dummy host (choc.choc in choc), for consistency.

falkTX commented

choc is only used for windows, we can ignore the linux and mac side of it.
weird though that it cannot use custom scheme...

need to investigate what is that about

sevenc-nanashi commented

Oh I meant: I think it's okay to use custom url scheme, but there should be a dummy host for consistency (like choc does)

falkTX commented

well if all are able to work the same way there is no need for http hostname.

thirtythreeforty commented

@falkTX and @lucianoiam I just want to say that between y'all, the web UI support for DPF is shaping up to be excellent. I have previously implemented my own wrapper around webview and getting it working cross-platform is quite difficult (especially on Linux). Thank you so much.

Let me know if I can pitch in. Among other things, I have a setup where CMake knows how to bundle a helper binary into the .so as a blob, which can be run directly from a memfd file descriptor. It would help dpfwebui, but I think DPF's method of launching the .so as a helper executable might be cleaner than my approach. But if this sounds useful I am happy to contribute.

  • 👍2
falkTX commented

Let me know if I can pitch in. Among other things, I have a setup where CMake knows how to bundle a helper binary into the .so as a blob, which can be run directly from a memfd file descriptor. It would help dpfwebui, but I think DPF's method of launching the .so as a helper executable might be cleaner than my approach. But if this sounds useful I am happy to contribute.

how does that setup work? my current linux approach was focused on not needing extra files to be shipped with the plugin in order to activate webview. so if I understood you right, you would:

  1. build an extra binary as part of the makefile/cmake setup
  2. append that binary to the main shared lib somehow
  3. when a webview is needed, the binary is copied out of the shared lib into tmpfs
  4. execute the binary that now exists separately

or perhaps you really had an external binary needed for this, in that case the actual implementation does not really matter

the tricky part on the linux side is that we need to run the webview in a separate process, because we are loading gtk or qt, likely conflicting with host symbols. so there needs to be an fork/exec in the whole thing (or posix_spawn if we want to be fancy). the "exec" part needs an actual executable program to run, which on my approach is the ld linux loader. would be nice to not have to rely on that, so we can be more resilient to changes and also work on BSDs.

lucianoiam commented

Hi @thirtythreeforty, just wanted to clarify that both implementations were developed independently and I would favor @falkTX 's for new projects. Back then when I started working on my project, there was no CHOC WebView (or at least I wasn't aware of its existence). But nowadays it just makes sense to use CHOC which is a better supported solution.

Some things that can be still rescued from the (unofficial) dpfwebui:

  • Built-in http/websockets server, that is useful for creating distributed UIs (the project's original goal, and out of DPF's scope)
  • Alternate CEF based implementation on Linux in addition to the GTK webview.

Some things I don't like now:

  • No network-less transfer protocol except for file://, which could cause some problems
  • Custom code instead of popular library for critical component
  • No CMake

Regarding multiple files in a plugin, highly likely the plugins are distributed with a bunch of HTML/JS/CSS files so an extra helper binary doesn't seem to hurt. But yes it would be cool to have everything self-contained.

@falkTX maybe it would be possible to make the dpf:// protocol ( I think that was the name given to it on another dicussion ) read "files" appended to the plugin binary? like if it was a tarball.

On the other hand, modern plugin formats already require a filesystem structure...

falkTX commented

@falkTX maybe it would be possible to make the dpf:// protocol ( I think that was the name given to it on another dicussion ) read "files" appended to the plugin binary? like if it was a tarball.

if you look into the current qt code I added, there are already some steps for that. from my tests it is doable and not complicated to do at all. but a bit time consuming for supporting it on all frameworks and OSes.
and it works without needing a webserver or websockets, though sure we could call this call a sorta ad-hoc webserver (but it simply doesnt use HTTP protocol, it bypasses the whole network layer)

  • 🎉1
lucianoiam commented

I was referring to how to pack the web files into the plugin, rather than how to access them which I agree it is better to skip the network. Anyways, fancy non-critical stuff.

I haven't had a look at the code yet but everything sounds like going on the right track.

thirtythreeforty commented

so if I understood you right, you would:

  1. build an extra binary as part of the makefile/cmake setup
  2. append that binary to the main shared lib somehow
  3. when a webview is needed, the binary is copied out of the shared lib into tmpfs
  4. execute the binary that now exists separately

Mostly correct, but in (2) the binary is "blobbed" into a char* array. The CMake helper code exports a C symbol and understands that the blob depends on the specified executable, so it's a single line to add another blob. (I use the same mechanism to blob the JavaScript bundle that runs in the webview.) And in (3) the blob is copied into, and executed directly from, a Linux memfd, so it never hits a filesystem at all. (There's some other prep in the forked child to set env vars such as GDK X11 setup, and closing file descriptors the DAW may have had open.)

thirtythreeforty commented

How does the DPF solution plan to handle embedding files in the plugin?

falkTX commented

How does the DPF solution handle "serving" files/embedding them in the plugin?

there is no way to serve files yet, the default approach for now is using file:/// that points to the bundle/resources of the plugin

embedding files is left to the developer to do, there is a tool to convert a file into C code https://github.com/DISTRHO/DPF/blob/main/utils/res2c.py but this can be done in many other ways. newer C++ standards even allow to #embed "somefile.ext" directly now.

but this is all very new stuff, there is not yet a single plugin that makes use of webgui from dpf.
https://github.com/mod-audio/mod-desktop is going to use it for the next release though, the webgui implementation started from there actually, and that is the main reason it exists as I need it for MOD Audio.

and yeah it works for that already, it does not use files but provides its own webserver

image

thirtythreeforty commented

The missing piece for an html file bundled into the plugin is the ability to either:

  • give up and place index.html separately into the plugin bundle :)
  • tell the UI superclass to load an HTML string (https://github.com/webview/webview supports this with webview::set_html(std::string) and it's dirt simple if you can tell your framework to emit a single file - Svelte is happy enough to do this
  • "Serve" the file in custom code as you mention in response to a "network request" which is really just passing data around

Since it's looking like the last bullet is how the consensus is leaning, what needs to be done to implement it?