ScanDy
is a modular and mechanistic computational framework for simulating realistic scanpaths in dynamic real-world scenes. It is specifically designed to quantitatively test hypotheses about eye-movement behavior in videos.
Specifically, it can be used to demonstrate the influence of object-representations on gaze behavior by comparing object-based and location-based models.
For a visual guide of how ScanDy
works, have a look at the interactive notebook (also on Colab) and the example usecases.
The structure of ScanDy
is inspired by the neurolib
framework, which is also used for parameter optimization and exploration.
You can install ScanDy
as pypi package using pip
:
pip install scandy
We however reccomend that you clone (or fork) this repository and install all dependencies with
git clone https://github.com/rederoth/ScanDy.git
cd ScanDy/
pip install -r requirements.txt
pip install .
This gives you more freedom to modify the existing models and run the examples.
CAVEAT: There is currently an incompatability with Python 3.11 and the numba
package (required by neurolib
), see numba/numba#8304. We therefore recommend using Python <=3.10 or manually installing numba
/neurolib
.
The scanpath models require precomputed maps of the video data. We use the VidCom dataset (Li et al., 2011), for which we provide all the required data on OSF (https://www.doi.org/10.17605/OSF.IO/83XUC).
To prepare the dataset, we used the following resources:
- VidCom - Video and eye-tracking data
- deep_em_classifier - Eye movement classification
- detectron2 - Frame-wise object segmentation
- deep_sort - Object tracking
- dynamic-proto-object-saliency - Low-level saliency maps
- TASED-Net - High-level saliency maps
- PWC-Net - Optical flow calculation
If you only want to play around with a single video, we uploaded a version of the dataset only containing the "field03" video to Google drive.
We prepared a number of IPython Notebooks for you to explore the framework.
To get started with ScanDy
, have a look at our interactive guide, where you can explore the effect of individual model parameters.
Additionally, we show instructive usecases, including:
- Example 1, on Colab: Scanpath simulation and visualization for a single video
- Example 2, on Colab: Evolutionary optimization of model parameters
- Example 3, on Colab: Extending on existing models: Location-based model with object-based sensitivity
All figures from our manuscript (Roth et al., 2023) can be reproduced with this notebook, which is also executable on Colab.
If ScanDy
is useful for your research, please cite our paper:
Roth, N., Rolfs, M., Hellwich, O., & Obermayer, K. (2023). Objects guide human gaze behavior in dynamic real-world scenes. PLOS Computational Biology 19(10): e1011512.
@article{roth2023objects,
title = {Objects Guide Human Gaze Behavior in Dynamic Real-World Scenes},
author = {Roth, Nicolas and Rolfs, Martin and Hellwich, Olaf and Obermayer, Klaus},
journal = {PLOS Computational Biology},
publisher = {Public Library of Science},
year = {2023},
month = {10},
volume = {19},
url = {https://doi.org/10.1371/journal.pcbi.1011512},
pages = {1-39},
number = {10},
If you have feedback, questions, and/or ideas, feel free to send a mail to Nico.
Nicolas Roth, PhD Student at Science of Intelligence; Neural Information Processing Group, Fakultaet IV, Technische Universitaet Berlin, MAR 5-6, Marchstr. 23, 10587 Berlin
This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC 2002/1 "Science of Intelligence" – project number 390523135.