Fixing windows ink tablet events on SDL3

[Susko3]

Preface

For best performance and customizability, the built in tablet drivers should be used. But windows ink tablets are quite common and plug-and-play on windows, so it makes sense to support them.

Supporting tablets that generate fake mouse events (likely includes external OpenTabletDriver) is out of scope, if it works then great, but if it doesn't, we'll ask users to disable relative mouse mode.

Related issues:

• SDL3
  • ppy/osu#28223
• Before SDL3
  • ppy/osu#23011
  • ppy/osu#23839
  • ppy/osu#27227

SDL changes

SDL3 already handles absolute raw input (i.e. tablets/windows ink), but it forcefully converts the events to relative motion. So the fix is to report the events as absolute input. Importantly, the mouseID is SDL_PEN_MOUSEID to allow discerning those events framework-side. (By default, the tablet inputs are marked with mouseID=0, same as touchpad inputs.)

Unsure if upstream will accept this. We would hide this behind a hint, eg. SDL_HINT_WINDOWS_TABLET_ALWAYS_ABSOLUTE_MOTION, but using SDL_PEN_MOUSEID is really hacky. There is already a way to discern tablet events from touches that is arguably just as hacky.

diff --git a/src/video/windows/SDL_windowsevents.c b/src/video/windows/SDL_windowsevents.c
index eb0131b2b..52d3ca2eb 100644
--- a/src/video/windows/SDL_windowsevents.c
+++ b/src/video/windows/SDL_windowsevents.c
@@ -578,13 +578,7 @@ static void WIN_HandleRawMouseInput(Uint64 timestamp, SDL_VideoData *data, HANDL
                 }
             }
         } else {
-            const int MAXIMUM_TABLET_RELATIVE_MOTION = 32;
-            if (SDL_abs(relX) > MAXIMUM_TABLET_RELATIVE_MOTION ||
-                SDL_abs(relY) > MAXIMUM_TABLET_RELATIVE_MOTION) {
-                /* Ignore this motion, probably a pen lift and drop */
-            } else {
-                SDL_SendMouseMotion(timestamp, window, mouseID, SDL_TRUE, (float)relX, (float)relY);
-            }
+            SDL_SendMouseMotion(timestamp, window, SDL_PEN_MOUSEID, SDL_FALSE, (float)(x - window->x), (float)(y - window->y));
         }
 
         data->last_raw_mouse_position.x = x;

Framework changes

Framework-side we'd discern tablet from relative mouse events, similar to how touch events are handled on windows.

osu-framework/osu.Framework/Platform/Windows/WindowsWindow.cs

Lines 218 to 238 in 7fe8bcb

	if (evtTfinger.TryGetTouchName(out string? name) && name == "pen")
	{
	// Windows Ink tablet/pen handling
	// InputManager expects to receive this as mouse events, to have proper `mouseSource` input priority (see InputManager.GetPendingInputs)
	// osu! expects to get tablet events as mouse events, and touch events as touch events for touch device (TD mod) handling (see https://github.com/ppy/osu/issues/25590)
	TriggerMouseMove(evtTfinger.x * ClientSize.Width, evtTfinger.y * ClientSize.Height);
	switch (evtTfinger.type)
	{
	case SDL_EventType.SDL_EVENT_FINGER_DOWN:
	TriggerMouseDown(MouseButton.Left);
	break;
	case SDL_EventType.SDL_EVENT_FINGER_UP:
	TriggerMouseUp(MouseButton.Left);
	break;
	}
	return;
	}

Adding sensitivity is fairly simple, basically copying the old WindowsMouseHandler logic.

osu-framework/osu.Framework/Platform/Windows/WindowsMouseHandler.cs

Lines 129 to 142 in 66e8ea1

	if (Sensitivity.Value != 1)
	{
	// apply absolute sensitivity adjustment from the centre of the screen area.
	Vector2 halfScreenSize = (screenSize / 2);
	position -= halfScreenSize;
	position *= (float)Sensitivity.Value;
	position += halfScreenSize;
	}
	// map from screen to client coordinate space.
	// not using Window's PointToClient implementation to keep floating point precision here.
	position -= new Vector2(window.Position.X, window.Position.Y);
	position *= window.Scale;

WIP branch & testing

Here's the WIP framework branch: https://github.com/ppy/osu-framework/compare/master...Susko3:win-fix-sdl3-tablet?expand=1. To run, release SDL3-CS locally with the SDL3 changes.

I've tested this with https://github.com/Teages/vTablet using an android phone as a tablet. The code works really well for positioning the cursor, but there are some issues with tablet pen down events not registering. I've not investigated those yet.

Please give it a spin on a real tablet. And if it works fine, I'll start with the upstream SDL changes.

osu! input settings

We should probably have high precision mouse and sensitivity if no tablets are detected. Unsure about the names. It's also possible to have mouse sensitivity be a different setting from tablet sensitivity.

[peppy]

The suggested settings text makes little sense to me. "Using windows ink?" should be removed, or the part after it should be written to make sense.

I can't suggest better copy because I don't know what it's trying to convey.