This repository makes available the source code, datasets, pre-trained weights for the work, "One-shot Learning-based Animal Video Segmentation", which is accepted by IEEE Transactions on Industrial Informatics.
Our proposed approach achieved mean intersection-over-union score of 89.5% on the DAVIS 2016 animal dataset [1] with only one labeled frame each video.
The contents of this repository are released under an MIT license.
The required python packages are listed in requirements.txt
Our repository has three stages as described in paper: base training, objectness training, fine-tuning. Details will be introudced in the following.
You can download the selection results in bubbleNet_data.zip (72 MB), unzip it in "datasets" folder. Noticeably, the BubbleNet that runs on the testing set does not need any labels.
Download ImageNet pretrained model for XceptionNet imagenet_pretrain_weights.zip (585 MB), unzip it in "weights" folder
Download Pascal and its extension datasets pascal_extension_dataset.zip (1 GB), unzip it in "datasets" folder
Then you can run train_pascal_base.py to start base training. Noticeably, the dataset here is large, which requires you to have a GPU with 11GB memory or more. You can reduce the number of training images by delete items in "datasets/pretrain_benchmark_reduced.txt".
"pretrain_benchmark_and_pascal.txt" represents the full dataset and "test_algorithm_pretrain_benchmark_reduced.txt" is used to debug fast.
You can also download our pre-trained weights on pascal and its extension pascal_base_train_weights.zip (121 MB), then unzip it in "weights" folder.
Download Davis 2016 training datasets DAVIS2016_train_dataset.zip (254 MB), unzip it in "datasets" folder
Before running train_davis_objectness.py, you should ensure that pre-trained weights of pascal and its extension dataset are in "weights/pascal_base_train_weights" folder, and the iterations are matched with program here.
You can also download our pre-trained weights on DAVIS 2016 training set objectness_weights.zip (121 MB), then unzip it in "weights" folder.
Download DAVIS animal testing datasets (fine-tune and test) finetune_test_dataset.zip (71 MB), unzip it in "datasets" folder
Similar to objectness training, please note that the pretrained weights in "weights/objectness_weights" should be matched with the one in program here. After running finetune_test.py, you can find the segmentation results in "results/segmentation" folder.
If you wanna evaluate the results quantatively as we did in paper, you can refer to the DAVIS official code https://davischallenge.org/davis2016/code.html
@ARTICLE{9556571,
author={Xue, Tengfei and Qiao, Yongliang and Kong, He and Su, Daobilige and Pan, Shirui and Rafique, Khalid and Sukkarieh, Salah},
journal={IEEE Transactions on Industrial Informatics},
title={One-Shot Learning-Based Animal Video Segmentation},
year={2022},
volume={18},
number={6},
pages={3799-3807},
doi={10.1109/TII.2021.3117020}}
[1] Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M. and Sorkine-Hornung, A., 2016. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 724-732). https://davischallenge.org/davis2016/code.html