AMT_Real_vs_Fake

Run "real vs fake" experiments on Amazon Mechanical Turk (AMT).

Synopsis

Runs a series "real vs fake" trials. Each trial pits a real image against a "fake" image generated by an algorithm.

Matlab

Put all images to test in a web accessible folder. This folder should have subfolders for the results of each algorithm you would like to test (names of subfolders are specified in opt.which_algs_paths). Must also contain a subfolder for the real images (path: opt.gt_path) and (optionally) a folder for vigilance test images (path: opt.vigilance_path), which can be obviously fake images used to verify that the Turkers are paying attention. Images should be named "1.jpg", "2.jpg", etc, in consecutive order up to some total number of images N.
Set experiment parameters by modifying opt in getOpts.m for experiment with name expt_name.
Run mk_expt(expt_name) to generate data csv and index.html for Turk.
Create experiment using AMT website or command line tools. For the former option, paste contents of index.html into HIT html code. Upload HIT data from the generated csv.

Can enforce that each Turker can only do HIT once (uses http://uniqueturker.myleott.com/; see opt.ut_id in getDefaultOpts.m)
If multiple algorithms are specified in opt.which_algs_paths, then each HIT tests a single algorithm randomly selected from this list. This way, each participant will only see one algorithm, but participants for each algorithm will be sampled iid from the same Turk population (and, importantly, iid w.r.t. time, so that no algorithm unfairly gets a "better" time of day).
If opt.paired is true, then "fake/n.jpg" will be pitted against "real/n.jpg"; if false, "fake/n.jpg" will be pitted against "real/m.jpg", for random n and m
See getDefaultOpts.m for documentation on more features

This tool was initially developed for Colorful Image Colorization. Feel free to use this bibtex to cite.