RESEARCH-ARTICLE SHARE ON

Automatic Color Scheme Extraction from Movies

https://dl.acm.org/doi/10.1145/3372278.3390685

ABSTRACT

A color scheme is an association of colors, i.e., a subset of all possible colors, that represents a visual identity. We propose an automated method to extract a color scheme from a movie. Since a movie is a carefully edited video with different objects and heterogeneous content embodying the director's messages and values, it is a challenging task to extract a color scheme from a movie as opposed to a general video filmed at once without distinction of shots or scenes. Despite such challenges, color scheme extraction plays a very important role in film production and application. The color scheme is an interpretation of the scenario by the cinematographer and it can convey a mood or feeling that stays with the viewer after the movie has ended. It also acts as a contributing factor to describe a film, like the metadata fields of a film such as a genre, director, and casting. Moreover, it can be automatically tagged unlike metadata, so it can be directly applied to the existing movie database without much effort. Our method produces a color scheme from a movie in a bottom-up manner from segmented shots. We formulate the color extraction as a selection problem where perceptually important colors are selected using saliency. We introduce a semi-master-shot, an alternative unit defined as a combination of contiguous shots taken in the same place with similar colors. Using real movie videos, we demonstrate and validate the plausibility of the proposed technique.

References Yaugiz Aksoy, Tuncc Ozan Aydin, Aljovsa Smolić, and Marc Pollefeys. 2017. Unmixing-Based Soft Color Segmentation for Image Manipulation. ACM Trans. Graph., Vol. 36, 2 (2017), 19:1--19:19.Google ScholarDigital Library Evlampios E. Apostolidis and Vasileios Mezaris. 2014. Fast shot segmentation combining global and local visual descriptors. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP. 6583--6587. https://doi.org/10.1109/ICASSP.2014.6854873Google Scholar George Awad, Asad A. Butt, Keith Curtis, Yooyoung Lee, Jonathan G. Fiscus, Afzal Godil, Andrew Delgado, Jesse Zhang, Eliot Godard, Lukas L. Diduch, Alan F. Smeaton, Yvette Graham, Wessel Kraaij, and Georges Quénot. 2019. TRECVID 2019: An evaluation campaign to benchmark Video Activity Detection, Video Captioning and Matching, and Video Search & retrieval. In Proceedings of TRECVID 2019. National Institute of Standards and Technology (NIST) .

affiliations https://dl.acm.org/conference/icmr/affiliations

MovieColor Analys

http://www.smartjava.org/content/movie-color-analysis-xbmc-boblight-java-and-d3js/

[]http://www.smartjava.org/content/movie-color-analysis-xbmc-boblight-java-and-d3js/)

Comparison of the movies

http://www.smartjava.org/examples/movie-viz/batman.html

http://www.smartjava.org/examples/movie-viz/

SmartJava posts

http://www.smartjava.org/topics/posts/

IBM: Open Video Scene Detection Dataset

https://research.ibm.com/haifa/projects/imt/video/Video_DataSet.shtml

Scene detection in Broadcast Videos

short links:

Source code on google drive:

Imagelab Shot Detector

Source code on github gist:

Caffe models from the ACMMM15 paper

Dataset on google drive:

RAI dataset https://drive.google.com/file/d/1YColUfc3ZuCbiAAHHMYRQVBF2yBikj4N/view?usp=drive_link

Dataset:

BBC Planet Earth dataset https://drive.google.com/file/d/1vr-bBxXFpz5-vyirVsW6-RF4EjXC7nfN/view?usp=drive_link

Movies- Find out colors of your favorite movies

here

Acknowledgments

We acknowledge the CINECA award under the ISCRA initiative, for the availability of high performance computing resources and support.

Publications

1	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "Recognizing and Presenting the Storytelling Video Structure with Deep Multimodal Networks" IEEE TRANSACTIONS ON MULTIMEDIA, vol. 19, pp. 955 -968 , 2017 \| DOI: 10.1109/TMM.2016.2644872 Journal
2	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "NeuralStory: an Interactive Multimedia System for Video Indexing and Re-use" Proceedings of the 15th International Workshop on Content-Based Multimedia Indexing, Florence, Italy, 19-21 June 2017, 2017 \| DOI: 10.1145/3095713.3095735 Conference
3	Baraldi, Lorenzo; Grana, Costantino; Messina, Alberto; Cucchiara, Rita "A Browsing and Retrieval System for Broadcast Videos using Scene Detection and Automatic Annotation" Proceedings of the 2016 ACM on Multimedia Conference, Amsterdam, The Netherlands, pp. 733 -734 , 15 - 19 October 2016, 2016 \| DOI: 10.1145/2964284.2973825 Conference
4	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "Scene-driven Retrieval in Edited Videos using Aesthetic and Semantic Deep Features" Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval, New York, USA, pp. 23 -29 , 6-9 Giugno 2016, 2016 \| DOI: 10.1145/2911996.2912012 Conference
5	BARALDI, LORENZO; GRANA, Costantino; BORGHI, GUIDO; VEZZANI, Roberto; CUCCHIARA, Rita "Shot, scene and keyframe ordering for interactive video re-use" Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, vol. 4, Rome, pp. 626 -631 , Feb 27-29, 2016, 2016 \| DOI: 10.5220/0005768706260631 Conference
6	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "Analysis and Re-use of Videos in Educational Digital Libraries with Automatic Scene Detection" Digital Libraries on the Move, vol. 612, Bolzano, pp. 155 -164 , Jan. 29-30, 2016 \| DOI: 10.1007/978-3-319-41938-1_16 Conference
7	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "A Deep Siamese Network for Scene Detection in Broadcast Videos" Proceedings of the 23rd ACM international conference on Multimedia, Brisbane, Australia, pp. 1199 -1202 , 26-30 October 2015, 2015 \| DOI: 10.1145/2733373.2806316 Conference
8	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "Shot and Scene Detection via Hierarchical Clustering for Re-using Broadcast Video" Computer Analysis of Images and Patterns. Part I, vol. 9256, Valletta, Malta, pp. 801 -811 , 2-4 September 2015, 2015 \| DOI: 10.1007/978-3-319-23192-1_67 Conference
9	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "Scene segmentation using temporal clustering for accessing and re-using broadcast video" Proceedings - IEEE International Conference on Multimedia and Expo, vol. 2015-, Torino, Italia, pp. 1 -6 , 2015, 2015 \| DOI: 10.1109/ICME.2015.7177476 Conference
10	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "Measuring scene detection performance" Pattern Recognition and Image Analysis, vol. 9117, Santiago de Compostela, Spain, pp. 395 -403 , 17-19 June 2015, 2015 \| DOI: 10.1007/978-3-319-19390-8_45 Conference

</div>

1	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "Recognizing and Presenting the Storytelling Video Structure with Deep Multimodal Networks" IEEE TRANSACTIONS ON MULTIMEDIA, vol. 19, pp. 955 -968 , 2017 \| DOI: 10.1109/TMM.2016.2644872 Journal
2	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "NeuralStory: an Interactive Multimedia System for Video Indexing and Re-use" Proceedings of the 15th International Workshop on Content-Based Multimedia Indexing, Florence, Italy, 19-21 June 2017, 2017 \| DOI: 10.1145/3095713.3095735 Conference
3	Baraldi, Lorenzo; Grana, Costantino; Messina, Alberto; Cucchiara, Rita "A Browsing and Retrieval System for Broadcast Videos using Scene Detection and Automatic Annotation" Proceedings of the 2016 ACM on Multimedia Conference, Amsterdam, The Netherlands, pp. 733 -734 , 15 - 19 October 2016, 2016 \| DOI: 10.1145/2964284.2973825 Conference
4	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "Scene-driven Retrieval in Edited Videos using Aesthetic and Semantic Deep Features" Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval, New York, USA, pp. 23 -29 , 6-9 Giugno 2016, 2016 \| DOI: 10.1145/2911996.2912012 Conference
5	BARALDI, LORENZO; GRANA, Costantino; BORGHI, GUIDO; VEZZANI, Roberto; CUCCHIARA, Rita "Shot, scene and keyframe ordering for interactive video re-use" Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, vol. 4, Rome, pp. 626 -631 , Feb 27-29, 2016, 2016 \| DOI: 10.5220/0005768706260631 Conference
6	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "Analysis and Re-use of Videos in Educational Digital Libraries with Automatic Scene Detection" Digital Libraries on the Move, vol. 612, Bolzano, pp. 155 -164 , Jan. 29-30, 2016 \| DOI: 10.1007/978-3-319-41938-1_16 Conference
7	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "A Deep Siamese Network for Scene Detection in Broadcast Videos" Proceedings of the 23rd ACM international conference on Multimedia, Brisbane, Australia, pp. 1199 -1202 , 26-30 October 2015, 2015 \| DOI: 10.1145/2733373.2806316 Conference
8	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "Shot and Scene Detection via Hierarchical Clustering for Re-using Broadcast Video" Computer Analysis of Images and Patterns. Part I, vol. 9256, Valletta, Malta, pp. 801 -811 , 2-4 September 2015, 2015 \| DOI: 10.1007/978-3-319-23192-1_67 Conference
9	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "Scene segmentation using temporal clustering for accessing and re-using broadcast video" Proceedings - IEEE International Conference on Multimedia and Expo, vol. 2015-, Torino, Italia, pp. 1 -6 , 2015, 2015 \| DOI: 10.1109/ICME.2015.7177476 Conference
10	Baraldi, Lorenzo; Grana, Costantino; Cucchiara, Rita "Measuring scene detection performance" Pattern Recognition and Image Analysis, vol. 9117, Santiago de Compostela, Spain, pp. 395 -403 , 17-19 June 2015, 2015 \| DOI: 10.1007/978-3-319-19390-8_45 Conference

ibanknatoPrad/boblight