- Towards Passive Labeling in 3D via Gaze Observation
- File structure
- Gaze observations
- MATLAB code
- Not yet investigated but implemented
This repository contains the material for my Master's Thesis at Universität Bern. The project aims at generating segmentations in medical images using eye-tracking data from clinicians. This has a great potential to reduce time costs for gaining training data for supervised learning techniques in medicine, as manual labeling is very time-consuming. Various approaches have been implemented in MATLAB and described in the thesis. The problem of finding training data using eye-tracking has been reformulated. A way to solve it using Gradient Boosting has been suggested and implemented.
The MATLAB-code assumes the data structure as described below. For further explanation of the content of the data/Dataset[X]/-folders, see also the README.txt files inside the folders. The data can be found along this git repository on the ARTORG server, but is not uploaded to the git repository.
- matlab code/
- NIfTI_20140122/
- vlfeat/
- libsvm-3.20/
- libsvm-3.20/
- sqb-0.1/
- data/
-
Dataset1/
- Video.avi (the input video)
- gaze-measurements/ (the recorded gaze positions in csv format -- refer to Gaze observations
- (...).csv
- input-frames/ (single frames of the input video in png format)
- frame_00001.png
- frame_00002.png
- ...
- ground_truth-frames/ (ground truth segmentations for the single frames of the input video in png format)
- frame_00001.png
- frame_00002.png
- ...
- (...)-descriptors/ (generated by MATLAB code)
- frame_00001.mat
- frame_00002.mat
- ...
- trainingSet.mat (generated by MATLAB code)
-
Dataset2
- Video.avi
- ...
-
...
-
The csv file containing gaze records should have the following structure (columns):
frame; time; visible; x; y
- frame: the frame number
- time (not currently needed): the time of the measurement [ms from start of video] to detect whether some positions are 'old records' (it is not currently used in the project and could therefore be left blank)
- visible: 1, if the user pressed the space bar during the frame, 0 otherwise - in the code, this value is usually called key_pressed
- x, y: x- and y-positions as outputted by The Eye Tribe in relative screen coordinates (floats between [0,1])
So, an example file would look as follows:
0; 0; 0; 0.431286; 0.412303
1; 33; 0; 0.431181; 0.413398
2; 66; 0; 0.430746; 0.414175
...
158; 5129; 1; 0.779511; 0.382291
159; 5163; 1; 0.780631; 0.381369
The MATLAB code is organized as follows:
- main.m: the main script contains the necessary MATLAB sections to execute, if you want to try out the whole procedure
- classification/ contains functions used for the classification part; that is everything that is used in the main-script from step 5) onwards.
- prepareData/ contains a script "prepareData" that is called in the main-script, and most of the functions that are used in it. Also the functions to generate features can be found there.
- pugradboost/ contains the functions and a test-script for gradient boosting with the PU-loss function. The current implementation has decision stumps as weak learner.
- subtightplot/ contains the subtightplot toolbox to create subplots without big margins. It was published on Mathworks File Exchange
- utils/ contains simple functions that make the life easier and don't belong to one specific step
- visualizations/ contains scripts and functions used to generate all kinds of visualizations that have been used in the thesis. Some require to first load certain data into the workspace. All this is indicated in the descriptions of the single scripts of functions.
There is a script prepareData/trainSuperpixelautoencoder.m that can be used to train an autoencoder for later encoding of superpixels. Once an autoencoder has been trained and the object is stored to a file, it can be used in prepareData/getSuperpixelFeaturesBeta(...,...,3) to encode superpixels with this autoencoder (the exact filename has to be changed in getsuperpixelFeaturesBeta.
The regularized method of Zhou et al. to smooth labels has been implemented, but was only very briefly tested and no results regarding this strategy are mentioned in the thesis. The function classification/smoothFrameLabels.m can be used to smooth the resulting labels of a classifier. In classification/testSuperpixelClassifier there is a commented line to show how to use this function for all the obtained labels after the classification has been performed. Other things could be thought of (e.g. it could be also applied for only a few frames at a time instead of at the end for all samples of all frames).