Scrcpy Mask

中文介绍

To achieve computer control of Android devices, I developed a cross-platform desktop client using Tarui + Vue 3 + Rust. This client provides visual mouse and keyboard mapping configuration, enabling multi-touch operations similar to Android emulators through key mapping, with millisecond-level response time. This tool can be widely used for controlling Android devices from computers to play mobile games, providing a smooth touch experience.

I express my deep respect and gratitude to the developers of the Scrcpy project. Scrcpy is a powerful and efficient open-source tool that greatly facilitates control over Android devices. This project is built upon the excellent architecture of Scrcpy, with optimizations and adjustments for mouse and keyboard control.

This project does not provide Scrcpy's screen mirroring feature! It only implements Scrcpy's control protocol.

Because screen mirroring may involve latency and blurriness issues, this project takes a different approach by directly abandoning screen mirroring and instead using a transparent mask to display the content behind the window (which can be AVD, low-latency screen mirroring provided by your phone manufacturers, etc.), Completely eliminates the problem of poor screen casting experience inherent in Scrcpy.

If you really need screen mirroring and don't mind the latency and performance issues, you can use the Android app ScreenStream for LAN screen mirroring. Scrcpy MAsk has been adapted to work with ScreenStream since version v0.5.0. While its performance may leave something to be desired, it is ready to use out of the box.

Furthermore, to better support interaction between Scrcpy Mask and Android devices, I have made some modifications to the scrcpy-server, leading to the creation of a separate branch project called scrcpy-mask-server.

Features

Demonstration video

Implementation principle

Screenshot

Device control

Key mapping setting

Mask above game

Basic using

Install software suitable for your system platform from releases
Identify your Android device type
1. For physical devices like phones or tablets
  1. You need to solve the problem of screen casting on your own. Recommend using the official screen mirror method of your device brand to achieve the minimum delay. Since v0.5.0 version, it can be used with ScreenStream to cast screen under the same LAN.
  2. Enable ADB debugging on your device via USB or wirelessly, then connect it to your computer.
2. For emulator, you don't need screen mirror, and emulator generally default to enabling ADB wired debugging. So this is the best way for game, I think.
Launch the software and navigate to the Device page.
1. Find your device among the available devices (if not found, please search for how to enable ADB debugging for your device).
2. Right-click on your device again and choose "Control this device".
Navigate to the Settings page -> Mask Settings, set the width and height of the mask to the same ratio of the device screen size and ensure that the mask size is appropriate.
Navigate to the Mask page where you can see a transparent mask. Next, adjust and move your emulator window or screen mirroring window to align the displayed content area with the transparent mask area.
Navigate to the Key mapping page and switch or edit the key mapping configs.
Return to the Mask page and start enjoying.

About Macros

Currently, the structure of macros is simply a JSON object with limited functionality, serving as a transitional solution. Please refrain from investing too much time in writing macros, as the specifications for macro creation may change with version updates.

An example of macros can be found in the async function execMacro function in hotkey.ts file.

For instance, the key-input-mode macro can switch from key mapping mode to key input mode, commonly used for text input. An example is as follows:

[{ "args": [], "type": "key-input-mode" }]

Error Report

When asking a question, please provide as much detailed information as possible regarding the issue, including the operating system and software version. Specifically, if there is an error output, please be sure to include the relevant logs.

There are two sources of logs that might help in identifying and solving the error. Generally, the error output can be found in the Web logs.

Web Logs: Open Developer Tools by pressing Ctrl+Shift+I or Cmd+Opt+I, click on the console tab, and check the information output in the console.
Rust Logs:
1. On macOS or Linux, navigate to the installation directory, use the terminal to run scrcpy-mask, and you can see the program's output in real-time in the terminal.
2. On Windows, you need to clone the project and run it yourself to view the Rust output.

Contribution.

If you are interested in this project, you are welcome to submit pull request or issue. But my time and energy is limited, so I may not be able to deal with it all.

AkiChase/scrcpy-mask