/nitec

NITEC: Versatile Hand-Annotated Eye Contact Dataset for Ego-Vision Interaction (Accepted at WACV24)

Primary LanguagePythonMIT LicenseMIT

NITEC: Versatile Hand-Annotated Eye Contact Dataset for Ego-Vision Interaction (Accepted at WACV24)

animated

Citing

If you find our work useful, please cite the paper:

@InProceedings{Hempel_2024_WACV,
    author    = {Hempel, Thorsten and Jung, Magnus and Abdelrahman, Ahmed A. and Al-Hamadi, Ayoub},
    title     = {NITEC: Versatile Hand-Annotated Eye Contact Dataset for Ego-Vision Interaction},
    booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
    month     = {January},
    year      = {2024},
    pages     = {4437-4446}
}

Paper

Thorsten Hempel, Magnus Jung, Ahmed A. Abdelrahman and Ayoub Al-Hamadi, "NITEC: Versatile Hand-Annotated Eye Contact Dataset for Ego-Vision Interaction", accepted at WACV 2024.

Abstract

Eye contact is a crucial non-verbal interaction modality and plays an important role in our everyday social life. While humans are very sensitive to eye contact, the capabilities of machines to capture a person's gaze are still mediocre. We tackle this challenge and present NITEC, a hand-annotated eye contact dataset for ego-vision interaction. NITEC exceeds existing datasets for ego-vision eye contact in size and variety of demographics, social contexts, and lighting conditions, making it a valuable resource for advancing ego-vision-based eye contact research. Our extensive evaluations on NITEC demonstrate strong cross-dataset performance, emphasizing its effectiveness and adaptability in various scenarios, that allows seamless utilization to the fields of computer vision, human-computer interaction, and social robotics. We make our NITEC dataset publicly available to foster reproducibility and further exploration in the field of ego-vision interaction.

Quick Usage: 

pip install face_detection@git+https://github.com/elliottzheng/face-detection
pip install nitec

Example usage:

from nitec import NITEC_Classifier, visualize
import cv2

nitec_pipeline = NITEC_Classifier(
    weights= CWD / 'models' / 'nitec_rs18_e20.pth',
    device=torch.device('cuda') # or 'cpu'
)

cap = cv2.VideoCapture(0)

_, frame = cap.read()    
# Process frame and visualize
results = nitec_pipeline.predict(frame)
frame = visualize(frame, results, confidence=0.5)

Train / Test

NITEC Dataset

Prepare the dataset as explained here.

Snapshots

Download from here: https://drive.google.com/drive/folders/1zc6NZZ6yA4NJ52Nn0bgky1XpZs9Z0hSJ?usp=sharing

Train

 python train.py \
 --gpu 0 \
 --num_epochs 50 \
 --batch_size 64 \
 --lr 0.0001 \

Test

 python test.py \
 --snapshot models/nitec_rs18_20.pth \
 --gpu 0 \