vladmandic/human

Questions on using the face meshes to make masks

mantzaris opened this issue · 1 comments

mantzaris commented

I am using within ElectronJS:

    "@tensorflow/tfjs": "^4.17.0",
    "@tensorflow/tfjs-node": "^4.17.0",
    "@vladmandic/human": "^3.2.1",

the human config object:

const human_config = {
  backend: "webgl", // Specify the backend to use, wasm alternative
  modelBasePath: model_path,
  face: {
    enabled: true,
    detector: { rotation: true, maxDetected: 5, return: true },
    mesh: { enabled: true, return: true },
    meshRaw: { enabled: true, return: true },
    iris: { enabled: true, return: true },
    distance: { enabled: true, return: true },
    description: { enabled: true },
    emotion: { enabled: false },
  },
  gesture: { enabled: true },
};

I have a question on how to use the face meshes to create 'masks'. I can get the points of the face mesh and draw them via:

const result = await human.detect(videoElement);

  // You can access various results, e.g., face landmarks
  if (result.face.length > 0) {
    const canvas = document.getElementById("face-mesh-canvas");
    const ctx = canvas.getContext("2d");
    ctx.clearRect(0, 0, canvas.width, canvas.height);

    result.face.forEach((face) => {
      drawMesh(face.mesh, ctx); 
      //drawFilledMesh(face.mesh, ctx);
    });
...

function drawMesh(mesh, ctx) {
  const scaleX = videoElement.offsetWidth / videoElement.videoWidth;
  const scaleY = videoElement.offsetHeight / videoElement.videoHeight;

  mesh.forEach((point) => {
    const size = 2;
    const x = point[0] * scaleX; 
    const y = point[1] * scaleY; 

    ctx.fillStyle = "blue"; 
    ctx.fillRect(x, y, size, size);
  });
}

This simply produces the points but not the mask of the face showcased on the right in
https://github.com/vladmandic/human-motion/blob/main/assets/screenshot-face.jpg

  1. how can I produce that face mask?

  2. There is also an image of a 468-Point Face Mesh with an image for the point markers to help 3D rendering at
    https://github.com/vladmandic/human/blob/main/assets/facemesh.png
    On the result from Human, the face has a 'mesh' of 478 points, are we supposed to produce a new map and create a mapping to a new image is there another way with some function helpers?

  3. Would it be easier to use the annotations to create mask mappings?

4 How can a mapping to another modified mapping be possible, eg, a 'cat' face? Is there process for creating the 'mesh' with the numbering of the 'keypoints' that can then be drawn from some standard approach?

any assistance is appreciated

(sponsoring)

vladmandic commented

you've posted a link to image on human-motion repo - that repo uses human library to for analysis, but then it also uses a 3d framework babylonjs to draw a mask. and yes, human does give you helper functions for masks - they are called uvmaps.
see human-motion repo for examples, e.g. https://github.com/vladmandic/human-motion/blob/8e1ccbb9b53443360a070b2f7342b7c4501976bb/src/mesh.ts#L137

for 3d framework - i choose babylonjs in human-motion, but you could go with any of your choice, for example three.js`.

regarding mesh having 468 or 478 points, it simply depends if mesh is just a normal face mesh or its augmented with details from iris model for extra points for each eye. this is covered in docs.

btw, in the future please create a discussion topic - this is a general question, not a product issue.

  • 👍1