Internoise 2021: Head tracker using webcam for auralization

Support files for the Internoise 2021 paper "Head tracker using webcam for auralization".

Further updates to this software are published at this repository.

Description

Head tracker via camera face tracking and communication via UDP protocol.

Built on top of the Google's MediaPipe face_mesh (python release).

Folder structure:

/src: Contains the source code for the HeadTracker as published in the paper.
/test: Presents auralization experiments in MATLAB using the HeadTracker.
/audios: The raw files for the audio examples in the paper.
/videos: The images related to the head movements that produced the audios in the paper.
/presentation: PDF conference presentation of the paper.

System support

OS	Support
Windows	Tested on Windows 10
macOS	Tested on v10.15 and v11.2.1 (amd_64)
Linux	Tested on Ubuntu 18.04.5 LTS

-It also works on Raspberry Pi.

Setup python environment

This application only requires you to run pip install mediapipe==0.8.3.1. However for the sake of good practices, we recommend you create a new python enviroment and install the required libraries with:

cd internoise2021-headtracker/src/

conda create --name headtracker python=3.8

conda activate headtracker

pip install -r requirements.txt

MediaPipe supports Python 3.6 to 3.8.

Raspberry Pi

Install OpenCV and mediapipe from the sources bellow:

OpenCV: https://pimylifeup.com/raspberry-pi-opencv/
Mediapipe: https://pypi.org/project/mediapipe-rpi4/

Using the HeadTracker

The application can be initialized with:

python HeadTracker.py

Alternatively you may use the Windows executables distributed here. Notice that you don't need to setup an environment, or install anything else, in order to use the .exe standalones.
Connect to any plataform that accepts UDP/IP connection using the address: IP:'127.0.0.1' and PORT:50050 .
In order to close the app, mouse clicking "quit the window" might not work in all operating systems, as a general rule use "Esc" to finish the process, while we work in a more elegant solution for this issue.

Interpreting received data

The HeadTracker application currently sends to the server yaw, pitch and roll information in degrees, where downwards pitch and counterclockwise roll and yaw are denoted with negative angles, such that the full rotation is bounded between -180° and 180°, as illustrated bellow.

The sent data are strings encoded into bytes, for e.g. if the sent/received message is: b'-5,10,0', the corresponding coordinates are yaw=-5°, pitch=10° and roll=0° — depending on the application the data needs to be decoded for proper use.

Example: reading HeadTracker output data with Matlab

Bellow you can find a snippet of how to connect to the UDP address and convert the binary data to matlab array.

% Open the HeadTracker application (make sure the file path is added to matlab path variables)
open('HeadTracker.exe')   

% Connect to the local server
udpr = dsp.UDPReceiver('RemoteIPAddress', '127.0.0.1',...
                       'LocalIPPort',50050); 

% Read data from the head tracker
while true   
    py_output = step(udpr);
    if ~isempty(py_output)
        data = str2double(split(convertCharsToStrings(char(py_output)), ','));
        disp([' yaw:', num2str(data(1)),...
             ' pitch:', num2str(data(2)),...
             ' roll:', num2str(data(3))])
    end
end

Other examples of the connection to matlab are posted here.

Issues

So far the code defaults to the main camera, if you experience difficulties, you can either enable only the desired one in your system or directly select it in the code by changing the index in cv2.VideoCapture(0).

Cite us

D. R. Carvalho; W. D’A. Fonseca; J. Hollebon; P. H. Mareze; F. M. Fazi. Head tracker using webcam for auralization. In 50th International Congress and Exposition on Noise Control Engineering — Internoise 2021, pages 5071–5082(12), Washington, DC, USA, Aug. 2021. doi: 10.3397/IN-2021-2956.

Bibtex:

@InProceedings{headtracker:2021,
  author    = {Davi Rocha Carvalho and William {\relax D'A}ndrea Fonseca and Jacob Hollebon and Paulo Henrique Mareze and Filippo Maria Fazi},
  booktitle = {{50th International Congress and Exposition on Noise Control Engineering --- Internoise 2021}},
  title     = {Head tracker using webcam for auralization},
  year      = {2021},
  address   = {Washington, DC, USA},
  month     = {Aug.},
  pages     = {5071--5082(12)},
  doi       = {10.3397/IN-2021-2956},
}

eac-ufsm/internoise2021-headtracker