Hi From CodeVideo - we're interested in writing a "human-like mouse movement" plugin for nut-js!

[princefishthrower]

Short overview

I found this library from your comments on robotjs, and this is exactly the library I've been looking for (after fighting multiple bugs and patches with robotjs, I've gone fully nut-js 😄 )

Use case

At CodeVideo, we provide a declarative way to make realistic software coding videos (for use by educators / etc.). Eventually, part of our product will include a full fledged keyboard / mouse emulator in the editor environment, and we'll want actions that mimic human movements via bezier curves (or perhaps bezier curves with noise / error correction). Things like 'circling' certain variables or things on the screen with the mouse, or 'underlining' by 'painting' with the mouse - if that makes sense.

Detailed feature description

We've got some simple TypeScript code as a start of this implementation, an attempt at 'arcing' style mouse movements, (still untested):

import { mouse, Point, straightTo } from "@nut-tree/nut-js";

type BezierCurveType =
  | "arc-above"
  | "arc-below"
  | "arc-left"
  | "arc-right"
  | "straight-line";

export const moveMouseInHumanLikeWay = async (
  startPoint: Point,
  endPoint: Point,
  curveType: BezierCurveType,
  jitter: boolean = false
) => {
  // Speed up the mouse.
  mouse.config.mouseSpeed = 20; // pixels per second

  if (curveType === "straight-line") {
    // Move the mouse in a straight line.
    const pointsCount = 100;
    const xIncrement = (endPoint.x - startPoint.x) / pointsCount;
    const yIncrement = (endPoint.y - startPoint.y) / pointsCount;

    for (let i = 0; i <= pointsCount; i++) {
      let x = startPoint.x + i * xIncrement;
      let y = startPoint.y + i * yIncrement;

      if (jitter) {
        // Add slight random deviations to the path.
        const deltaX = (Math.random() - 0.5) * 5;
        const deltaY = (Math.random() - 0.5) * 5;

        x += deltaX;
        y += deltaY;
        await straightTo(new Point(x, y));

        // Compensate by applying the opposite adjustment after each jitter.
        x -= deltaX;
        y -= deltaY;
        await straightTo(new Point(x, y));
      }
    }
    return;
  }

  // Calculate the control point for the bezier curve based on the curveType.
  let controlPoint: Point;
  if (curveType === "arc-above" || curveType === "arc-below") {
    controlPoint = { x: (startPoint.x + endPoint.x) / 2, y: startPoint.y };
  } else if (curveType === "arc-left" || curveType === "arc-right") {
    controlPoint = { x: startPoint.x, y: (startPoint.y + endPoint.y) / 2 };
  } else {
    console.error(
      "Invalid curve type. Please provide a valid curve type: 'arc-above', 'arc-below', 'arc-left', 'arc-right', or 'straight-line'."
    );
    return;
  }

  // Calculate the bezier curve points.
  const bezierCurvePoints: Point[] = [];
  for (let t = 0; t <= 1; t += 0.01) {
    let x =
      Math.pow(1 - t, 2) * startPoint.x +
      2 * (1 - t) * t * controlPoint.x +
      Math.pow(t, 2) * endPoint.x;
    let y =
      Math.pow(1 - t, 2) * startPoint.y +
      2 * (1 - t) * t * controlPoint.y +
      Math.pow(t, 2) * endPoint.y;

    if (jitter) {
      // Add slight random deviations to the path.
      const deltaX = (Math.random() - 0.5) * 5;
      const deltaY = (Math.random() - 0.5) * 5;

      x += deltaX;
      y += deltaY;
      bezierCurvePoints.push({ x, y });

      // Compensate by applying the opposite adjustment after each jitter.
      x -= deltaX;
      y -= deltaY;
      bezierCurvePoints.push({ x, y });
    }
  }

  // Move the mouse along the bezier curve points
  await mouse.move(bezierCurvePoints);
};

Additional content

If this is something you'd think is beneficial to have in the ecosystem, I'd be happy to put some more effort in with more functions and test cases / examples.

Anyways, cheers for this great library and thanks again!

[s1hofmann]

Hi @princefishthrower 👋

The great thing about nut.js and its architecture is that you don't have to get things like this merge into the core.

I specifically designed nut.js in a way that you can build your own utilities for it, without having to wait for a PR to get merged etc.
If you want to keep working on these things, feel free to create your own library and I'm happy to link it.

[princefishthrower]

Hi @s1hofmann - thanks for the information. Alright, sounds good. And thanks again for this library!