Wayland layer shell support

[simplejack-src]

Hello,

I'm looking into adding support in the Wayland video backend for the wlr_layer_shell protocol. I think the best route for this implementation would be to add two new Wayland-specific hints:

SDL_HINT_VIDEO_WAYLAND_SHELL_LAYER (default: "") ["background", "bottom", "top", "overlay"]
SDL_HINT_VIDEO_WAYLAND_SHELL_EXCL (default: 0) [-int, +int]

• The video hint SDL_HINT_VIDEO_WAYLAND_SHELL_LAYER is a string used to set the layer that wlr_layer_shell windows will be on.
• The video hint SDL_HINT_VIDEO_WAYLAND_SHELL_EXCL indicates the exclusion zone for wlr_layer_shell windows. This value has multiple meanings depending on if an anchor is set or not (see the relevant set_exclusive_zone function in the wlr_layer_shell protocol specification).

When the first hint SDL_HINT_VIDEO_WAYLAND_SHELL_LAYER is set (any other value then ""), wlr_layer_shell semantics are used for subsequent window/video operations (on wlr_layer_shell windows, in places where functionality would differ from the standard XDG semantics). Namely:

// If hint set, create layer-shell window (instead of XDG)
SDL_CreateWindow();

// Stretch size to anchors (i.e. max_width/max_height)
SDL_MaximizeWindow();

// Set anchors (and optionally a margin)
// SDL_WINDOWPOS_UNDEFINED = unanchor, left+right
// SDL_WINDOWPOS_UNDEFINED = unanchor, top+bottom
SDL_SetWindowPosition(
    x - >0 = anchor left margin-1, 0 = anchor left+right, <0 anchor right margin-1
    y - >0 = anchor top margin-1, 0 = anchor top+bottom, <0 anchor bottom margin-1
);
SDL_SetWindowKeyboardGrab(
    grabbed - (FALSE = 'on_demand' KB interactivity, TRUE = 'exclusive' KB interactivity)
);

Does anything look glaringly wrong before I look to start implementing? I'd also be a first-time contributor to SDL so any advice before I start changing code would also be appreciated. I don't know the SDL ranks, so I'm just referencing that last couple committers on Wayland code @Kontrabant @1bsyl @slouken.

EDIT: Updated implementation to what was actually implemented.

[Kontrabant]

Hi,

Questions about supporting this protocol have come up before, and the main question remains: is there a legitimate use case for it? This protocol is mainly intended for things like desktop widgets and panels, which are outside the domain of a general purpose application framework like SDL.

If there is some real, legitimate use for it then it can be considered, but I think the arguments against it in the linked issue by @dos1 and @flibitijibibo still stand, as this generally isn't a protocol that the typical application has any business using, particularly if the primary goal is to bypass the restrictions imposed on normal windows/applications, like keeping a window on top of all others.

[simplejack-src]

My legitimate use case for it will be in the construction of a desktop (hell maybe mobile at some point) panel. I find SDL meets my requirements with the exception of lacking this protocol. I'd agree that it's not the default window method (that's currently held by the XDG protocol), which is why my recommendation is for this to be influenced by the above HINT, indicating that the window being created will be for a more specific purpose.

[dos1]

I don't think implementing layer-shell in SDL makes a lot of sense. What may actually make some sense is giving the user an ability to implement custom surface role, so applications with special needs could implement layer-surface (or any other role they want) by themselves.

[simplejack-src]

@dos1 - The issue with exporting this work onto the calling client is a matter of interoperability. Specifically, trying to parent xdg_popup to a wlr_layer_shell surface and having fun hooking into the other Wayland callbacks without interfering with the SDL video Wayland backend.

I would reiterate that my proposed additions are strictly opt-in (done by setting the first HINT value). If the calling code doesn't set this hint nothing changes from what is being done now.

[simplejack-src]

Well I'm going to start the above implementation. I'm not sure what the SDL stance is Wayland protocol stability, but the wlr_layer_shell protocol is the development stage (new Wayland protocols terminology), so I'll put up a draft PR when I'm done.

[Consolatis]

If usual applications (vs desktop shell components like panels / notification daemons / backgrounds ..) start to make use of the layershell protocol to enforce an always-on-top feature, exclusive keyboard grabs or similar, compositors may start to restrict the usage of the protocol much sooner than is planned anyway. With all the complications that this would bring to existing legitimate users of the protocol.

Using SDL to create desktop components seems totally fine, providing that same API to usual client applications which might completely ignore its intended use-case (or are not even aware of it) sounds problematic.

[jadahl]

If usual applications (vs desktop shell components like panels / notification daemons / backgrounds ..) start to make use of the layershell protocol ...

If they did, they would already be unusable on many systems, as the availability of any of the layer shell protocols already varies between deliberately unavailable, available to privileged (desktop component) clients only, and available to all. The support requested here and the API added in #7303 is inherently unportable, so while my suggestion too would be what @dos1 suggested, to avoid adding API that are unreliable.

[simplejack-src]

SDL is a library like any other, how applications use it is entirely within the applications domain. Misuse and/or other restrictions are meant to be moderated by the server/compositor in accordance with their domain logic (e.g. GNOME doesn't wish to support this protocol as they consider the shell to be within their compositors domain, KDE, sway and some other wlroots compositors disagree). This protocol is generally well supported within the Wayland community at large (and I do expect it will be promoted to the staging status within Wayland-protocols). Much like any other Wayland protocol it may/or may not be supported by a Wayland compositor (such is the nature of any protocol outside of Core Wayland).
There might be value in having a hint feedback mechanism in SDL (which I don't believe currently exists), to allow an application to determine what protocols are actually supported by the compositor, but thats another SDL issue.

[Kontrabant]

Along with what has already been said, the implementation uses hints that significantly change the behavior of the Wayland backend, to the extent that toggling them basically changes it into a separate backend all its own, which has its own rules and doesn't behave like any other. Hints are used to toggle optional features and parameters, not radically change the behavior of the library; if this were to be activated on a non-supporting application (which can happen externally), you basically have undefined behavior. Likewise, what happens if someone relies on it and support is missing?

The suggestion regarding being able to create a surface with no role so the application can do what it will with it is probably the best long-term solution, and the work would dovetail with what is needed for foreign window/surface support via CreateWindowFrom(). This would allow for the creation of a separate helper library like SDL_layer_shell that applications can use, like the ones that exist for GTK and Qt.

[flibitijibibo]

There's a lot of technical discussion going on but on the side I'd point out that in the process of maintaining the Wayland backend I would absolutely break this feature at some point, since I won't be testing this path and I don't think the other maintainers will either... so technical details aside, if you want to proactively audit any and all commits as they come in to ensure this feature works indefinitely for the remainder of SDL's existence, I guess I can't argue with that!

(But yeah this probably shouldn't be a hint, that could bleed into other applications pretty easily.)

[simplejack-src]

@Kontrabant While your description of the hints subsystem seems logical I would contend that there are numerous cases where similar hints being enabled can radically change the behavior of the library (either explicitly or implicitly due to effecting the behavior of an external component). For example, some hints in a similar vein as the two I added:

SDL_HINT_X11_WINDOW_TYPE
SDL_HINT_X11_FORCE_OVERRIDE_REDIRECT
SDL_HINT_VIDEO_FORCE_EGL
SDL_HINT_VIDEO_WAYLAND_ALLOW_LIBDECOR
...

I would also disagree that this forms a new video backend. wlr_layer_shell is a Wayland protocol, to which it creates Wayland windows, it abides by all the same constraints that any Wayland protocol is bound by. In addition, despite many Wayland protocols being poorly named, this one does fall in line with the shell metaphor (with others being XDG and wl_shell). The semantics that it allows for are specific to it's protocol (just as the semantics to XDG are to it).
The PR (#7303) is up, so you can see that in the same vein as libdecor not being present, the fallback is to use a regular window. This could be changed to an error condition, however, I went with the graceful fallback method as this seems to be more prevalent in the Wayland backend. Conversely, it should also be asked what happens when someone is relying on libdecor or XDG support and its missing?

@flibitijibibo I don't know what SDL maintainership implies, but offhand, I would agree with you that I may not be up to the lifelog/cycle task (especially if it would be solo endeavor). Nevertheless, I do require this feature for my own use, so if necessary I will fork SDL and maintain it myself (which may be the path of least resistance).

I'll take a deeper look into the CreateWindowFrom() route and see if looks feasible (I did do a broad overview during design and besides looking like it would be a misuse of this interface, it also looked like a tough fit).

[simplejack-src]

Well my contract that I needed this on expired. Since its not wanted by the current SDL developers I'll leave it up as a personal fork.