/Awesome-Image-Harmonization

A curated list of papers, code and resources pertaining to image harmonization.

Awesome Image Harmonization Awesome

A curated list of resources including papers, datasets, and relevant links pertaining to image harmonization.

Contributing

Contributions are welcome. If you wish to contribute, feel free to send a pull request. If you have suggestions for new sections to be included, please raise an issue and discuss before sending a pull request.

Table of Contents

Online Demo

Try this online demo for image harmonization and have fun! hot

Leaderboard

The leaderboard of SOTA image harmonization methods can be found here.

Color Transfer

We summarize different color transfer strategies which could be used for image harmonization task here.

Papers

High-resolution image harmonization

  • Wenyan Cong, Xinhao Tao, Li Niu, Jing Liang, Xuesong Gao, Qihao Sun, Liqing Zhang: "High-Resolution Image Harmonization via Collaborative Dual Transformations." CVPR (2022) [arXiv] [dataset]
  • Jingtang Liang, Xiaodong Cun, and Chi-Man Pun: "Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization." arXiv preprint arXiv:2109.05750 (2021) [arXiv] [code]

Image harmonization using rendered images

  • Wenyan Cong, Junyan Cao, Li Niu, Jianfu Zhang, Xuesong Gao, Zhiwei Tang, Liqing Zhang: "Deep Image Harmonization by Bridging the Reality Gap." arXiv preprint arXiv:2109.06671 (2021) [arXiv] [dataset]
  • Zhongyun Bao, Chengjiang Long, Gang Fu, Daquan Liu, Yuanzhen Li, Jiaming Wu, Chunxia Xiao: "Scene Inference for Object Illumination Editing." arXiv preprint arXiv:2108.00150 (2021) [arXiv]
  • Zhongyun Hu, Ntumba Elie Nsampi, Xue Wang, Qing Wang: "NeSF: Neural Shading Field for Image Harmonization." arXiv preprint arXiv:2112.01314 (2021) [arXiv]

Supervised deep learning methods

  • Zonghui Guo, Dongsheng Guo, Haiyong Zheng, Zhaorui Gu, Bing Zheng, Junyu Dong: "Image Harmonization with Transformer." ICCV (2021) [pdf] [supp] [code]
  • Yifan Jiang, He Zhang, Jianming Zhang, Yilin Wang, Zhe Lin, Kalyan Sunkavalli, Simon Chen, Sohrab Amirghodsi, Sarah Kong, Zhangyang Wang:"SSH: A Self-Supervised Framework for Image Harmonization." ICCV (2021) [pdf] [supp] [arXiv] [code]
  • Jun Ling, Han Xue, Li Song, Rong Xie, Xiao Gu: "Region-Aware Adaptive Instance Normalization for Image Harmonization." CVPR (2021) [pdf] [supp] [arXiv] [code]
  • Zonghui Guo, Haiyong Zheng, Yufeng Jiang, Zhaorui Gu, Bing Zheng: "Intrinsic Image Harmonization." CVPR (2021) [pdf] [supp] [code]
  • Wenyan Cong, Li Niu, Jianfu Zhang, Jing Liang, Liqing Zhang: "BargainNet: Background-Guided Domain Translation for Image Harmonization." ICME (2021) [arXiv] [code]
  • Konstantin Sofiiuk, Polina Popenova, Anton Konushin: "Foreground-aware Semantic Representations for Image Harmonization." WACV (2021) [pdf] [supp] [arXiv] [code]
  • Guoqing Hao, Satoshi Iizuka, Kazuhiro Fukui: "Image Harmonization with Attention-based Deep Feature Modulation." BMVC (2020) [pdf] [supp] [code]
  • Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, Liqing Zhang: "DoveNet: Deep Image Harmonization via Domain Verification." CVPR (2020) [pdf] [supp] [arXiv] [code].
  • Xiaodong Cun, Chi-Man Pun: "Improving the Harmony of the Composite Image by Spatial-Separated Attention Module." IEEE Trans. Image Process. 29: 4759-4771 (2020) [pdf] [arXiv] [code]
  • Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, Ming-Hsuan Yang: "Deep Image Harmonization." CVPR (2017) [pdf] [supp] [arXiv] [code]

Unsupervised deep learning methods

  • Anand Bhattad, David A. Forsyth: "Cut-and-Paste Neural Rendering." arXiv preprint arXiv: 2010.05907 (2020) [arXiv] [supp]
  • Fangneng Zhan, Shijian Lu, Changgong Zhang, Feiying Ma, Xuansong Xie:"Adversarial Image Composition with Auxiliary Illumination." ACCV (2020) [pdf] [arXiv]
  • Bor-Chun Chen, Andrew Kae: "Toward Realistic Image Compositing With Adversarial Learning." CVPR (2019) [pdf]

Traditional methods

  • Shuangbing Song, Fan Zhong, Xueying Qin, Changhe Tu: "Illumination Harmonization with Gray Mean Scale." Advances in Computer Graphics. CGI (2020) [pdf]
  • Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, Alexei A. Efros: "Learning a Discriminative Model for the Perception of Realism in Composite Images." ICCV (2015) [pdf] [arXiv] [code]
  • Su Xue, Aseem Agarwala, Julie Dorsey, Holly E. Rushmeier: "Understanding and improving the realism of image composites." ACM Trans. Graph. 31(4): 84:1-84:10 (2012) [pdf]
  • Kalyan Sunkavalli, Micah K. Johnson, Wojciech Matusik, Hanspeter Pfister: "Multi-scale image harmonization." ACM Trans. Graph. 29, 4 (2010) [pdf]
  • Jue Wang, Maneesh Agrawala, Michael F. Cohen. 2007: "Soft scissors: an interactive tool for realtime high quality matting." ACM Trans. Graph. 26, 3 (2007) [pdf]
  • Jean-François Lalonde, Alexei A. Efros: "Using Color Compatibility for Assessing Image Realism." ICCV (2007) [pdf] [code]
  • Daniel Cohen-Or, Olga Sorkine, Ran Gal, Tommer Leyvand, Ying-Qing Xu: "Color harmonization." ACM Trans. Graph. 25, 3 (2006) [pdf]
  • Jiaya Jia, Jian Sun, Chi-Keung Tang, Heung-Yeung Shum: "Drag-and-drop pasting." ACM Trans. Graph. 25, 3 (2006) [pdf]
  • Patrick Pérez, Michel Gangnet, Andrew Blake: "Poisson image editing." ACM Trans. Graph. 22, 3 (2003) [pdf]

Datasets

  • iHarmony4: It contains four subdatasets: HCOCO, HAdobe5k, HFlickr, Hday2night, with a total of 73,146 pairs of unharmonized images and harmonized images. [pdf] [link]
  • GMSDataset: It contains 183 images with image resolution of 1940*1440. It consists of 16 different objects and for each object, one source image and 11 target images in different background scenes and illumination conditions are captured. [pdf] [link] (access code: ekn2)
  • HVIDIT: A dataset built upon VIDIT (Virtual Image Dataset for Illumination Transfer) dataset for image harmonization. It contains 3007 images of 276 scenes for training and 329 images of 24 scenes for testing. [pdf] [link]
  • RHHarmony: A rendered human harmonization dataset, which contains 15,000 ground-truth rendered images and has the potential to generate 135,000 composite rendered images. [pdf] [link]
  • RealHM: A Real-world HarMonization dataset, which contains 216 real composite images with manually harmonized outputs. [pdf] [link]

Related Topics

Inharmonious region localization

  • Jing Liang, Li Niu, Liqing Zhang: "Inharmonious Region Localization." ICME (2021) [arXiv] [code]
  • Jing Liang, Li Niu, Penghao Wu, Fengjun Guo, Teng Long: "Inharmonious Region Localization by Magnifying Domain Discrepancy." AAAI (2022)

Video harmonization

  • Haozhi Huang, Senzhe Xu, Junxiong Cai, Wei Liu, Shimin Hu: "Temporally Coherent Video Harmonization Using Adversarial Networks." IEEE Trans. Image Process. 29: 214-224 (2020) [pdf] [arXiv]
  • Xinyuan Lu, Shengyuan Huang, Li Niu, Wenyan Cong, Liqing Zhang: "HYouTube: Video Harmonization Dataset." arXiv preprint arXiv:2109.08809 (2021) [arXiv] [dataset]

Other Resources