Digital hand-in supplementary to the thesis ======================================================= "Unsupervised learning of object categories from video" ======================================================= by Jonas Luft, as a partial fulfilment of the degree of "Master of Science" at TU Berlin, 2018. Contact the author here via github Disclaimer ---------- THIS CODE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES. Use at your own risk. The package contains various third-party parts published under varying licenses. Notably: ISM code developed at ETH Zurich in 2001-2004, 2006-2008 and at TU Darmstadt in 2004-2005, copyrighted 2001-2005 by Bastian Leibe. With slight alterations made by us for research purposes only. Motion segmentation code copyrighted 2013 by Peter Ochs. Please refer to files ISM/readme.txt and MotSeg/readme.txt respectively. Sample data from Hopkins155 dataset: R. Tron and R. Vidal. A Benchmark for the Comparison of 3-D Motion Segmentation Algorithms. IEEE International Conference on Computer Vision and Pattern Recognition, June 2007. Use of our programs and files is made available for non-commercial purposes. Important files =============== When problems occur with motion segmentation or ISM please consult the following files: code/ISM/readme.txt code/ISM/clusterer9/readme.txt code/ISM/mcmatcher3/INSTALL.txt code/ISM/mcmatcher3/README.txt code/MotSeg/readme.txt Installation ============ Dense motion segmentation requires either CUDA or sshpass and titan.cv.tu-berlin.de access. Run code/ISM/install2.sh for installing and compiling ISM binaries. Requires Qt3 To use features other than patches: From code/ISM/mcmatcher9/INSTALL.txt: "Create a symbolic link from "~/code" to the provided "code" subdirectory. This is required, so that the detector can find the feature extraction binaries. ln -s $PWD/code ~/code" $PWD being the ISM directory. Run ./code/compile.sh to install our own tools for extracting figure masks and calculating category similarity. Requires OpenCV2 Requires boost Usage ===== We included two input videos from the Hopkins155 dataset and some procossing steps to illustrate intermediary results. Preparing input files --------------------- Input files are provided at code/data/input/frames/$sequenceName/ with unique names (without containing dashes "-") and a file $sequenceName.bmf according to code/MotSeg/readme.txt as PPM files List each sequence to be processed with it's $sequenceName in data/input/frames/sequences.txt Alternatively, put your input videos as .avi files in data/input/* and run ./preparefiles.sh Running the processing chain ---------------------------- run ./trainSequences.sh with the maximal number of iterations as the first parameter Evaluation ---------- ./detectAll.sh takes a detector name as it's first argument and the IoU threshold for matching as the second parameter It will run test detections with said detector on any sequences listed in data/input/frames/sequences.txt and put it's results in data/output/detection. Running ./evalSim.sh with 1. a detector, 2. a Jaccard-index threshold (default 0.5) will compare the results of the previozus test detections with results calculated during training and present in data/output/training You can use ./changeDetectorParam.sh to change ISM params either in trainSequences.sh (for training) or detect.sh (for testing). ./ISM/mcmatcher3/readme.txt has some notes. See files/carsall.params for possible parameters. Processing Chain ================ Training -------- .avi files in data/input/ 0. preparefiles.sh: split to $sequence-%3frame.ppm frames data/input/frames/$sequence create bmf file for MotSeg/MoSeg Don't put hyphens "-" in file names! data/input/frames/$sequence list of sequences to be processed in data/input/frames/sequences.txt 1. ./MotSeg/MoSeg output: sparse segmentation in data/inter/tracking/OchsBroxMalik$subsampling_all_00000$nu.00/$sequence as ppm optical flow files in data/input/frames/$sequence/flowfiles 2. densify.sh output: dense segmentation in data/inter/denseSeg/$sequence as ppm 3. runExSeg.sh output: segmentation masks for each object in ExSeg/output/$sequence/$object as png 4. prepareISMfiles.sh output: object frames in data/inter/frames/$sequence/$object object masks in data/inter/frames/$sequence/$object/maps files with frame lists as files/$sequence.idl 5. runISM.sh output: codebook files in files/codebooks/$detector 6. createOccurences.sh output: occurence files in files/codebooks/$detector 7. generateDetectors.sh output: detector files in files/codebooks/$detector detector list in files/codebooks/detectors.txt 8. runTrainingDetection.sh runs detections on figure arey only. output: detections of detectors in files/codebooks/detectors.txt and all sequences in data/output/training/$detector-$sequence results in data/output/training/$detector-$sequence/results.txt 9. mergeCats.sh output: updated detector list in files/codebooks/detectors.txt 8. and 9. iterating. Testing ------- 1. detectAll.sh runs detections on whole frames. input: sequences, output from training.8 and $detector output: detections and results in data/output/detection 2. evalSim.sh input: output from training.8 and from 1 output: baseline comparison results in data/output/detection/1eval_$detector.txt Comments ========