/Awesome-Image-Harmonization

A curated list of papers, code and resources pertaining to image harmonization.

Awesome Image Harmonization Awesome

A curated list of resources including papers, datasets, and relevant links pertaining to image harmonization.

Contributing

Contributions are welcome. If you wish to contribute, feel free to send a pull request. If you have suggestions for new sections to be included, please raise an issue and discuss before sending a pull request.

Table of Contents

Survey

A brief review on image harmonization can be found in the following survey on image composition:

Li Niu, Wenyan Cong, Liu Liu, Yan Hong, Bo Zhang, Jing Liang, Liqing Zhang: "Making Images Real Again: A Comprehensive Survey on Deep Image Composition." arXiv preprint arXiv:2106.14490 (2021). [arXiv]

Online Demo

Try this online demo for image harmonization and have fun! hot

Leaderboard

The leaderboard of SOTA image harmonization methods can be found here.

Color Transfer

We summarize different color transfer strategies which could be used for image harmonization task here.

Papers

Supervised deep learning methods

  • Yazhou Xing, Yu Li, Xintao Wang, Ye Zhu, Qifeng Chen: "Composite Photograph Harmonization with Complete Background Cues." ACM MM (2022) [pdf]
  • Ben Xue, Shenghui Ran, Quan Chen, Rongfei Jia, Binqiang Zhao, Binqiang Zhao: "DCCF: Deep Comprehensible Color Filter Learning Framework for High-Resolution Image Harmonization." ECCV (2022) [arXiv] [code]
  • Jinlong Peng, Zekun Luo, Liang Liu, Boshen Zhang, Tao Wang, Yabiao Wang, Ying Tai, Chengjie Wang, Weiyao Lin: "FRIH: Fine-grained Region-aware Image Harmonization." arXiv preprint arXiv:2205.06448 (2022) [arXiv]
  • Junyan Cao, Wenyan Cong, Li Niu, Jianfu Zhang, Liqing Zhang: "Deep Image Harmonization by Bridging the Reality Gap." BMVC (2022) [arXiv] [dataset] (rendered images)
  • Zonghui Guo, Zhaorui Gu, Bing Zheng, Junyu Dong, Haiyong Zheng: "Transformer for Image Harmonization and Beyond." T-PAMI (2022) [pdf]
  • Jingtang Liang, Xiaodong Cun, and Chi-Man Pun: "Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization." ECCV (2022) [arXiv] [code] (high-resolution)
  • Xuqian Ren, Yifan Liu: "Semantic-guided Multi-Mask Image Harmonization." ECCV (2022) [arXiv] [code]
  • Zhanghan Ke, Chunyi Sun, Lei Zhu, Ke Xu, Rynson W.H. Lau: "Harmonizer: Learning to Perform White-Box Image and Video Harmonization." ECCV (2022) [arXiv] [code]
  • Jeya Maria Jose Valanarasu, He Zhang, Jianming Zhang, Yilin Wang, Zhe Lin, Jose Echevarria, Yinglan Ma, Zijun Wei, Kalyan Sunkavalli, Vishal M. Patel: "Interactive Portrait Harmonization." arXiv preprint arXiv:2203.08216 (2022) [arXiv]
  • Wenyan Cong, Xinhao Tao, Li Niu, Jing Liang, Xuesong Gao, Qihao Sun, Liqing Zhang: "High-Resolution Image Harmonization via Collaborative Dual Transformations." CVPR (2022) [arXiv] [dataset] (high-resolution)
  • Zhongyun Bao, Chengjiang Long, Gang Fu, Daquan Liu, Yuanzhen Li, Jiaming Wu, Chunxia Xiao: "Deep Image-based Illumination Harmonization." CVPR (2022) [arXiv] (rendered images)
  • Yucheng Hang, Bin Xia, Wenming Yang, Qingmin Liao: "SCS-Co: Self-Consistent Style Contrastive Learning for Image Harmonization." CVPR (2022) [arXiv] [code]
  • Ziyue Zhu, Zhao Zhang, Zheng Lin, Ruiqi Wu, Chunle Guo: "Image Harmonization by Matching Regional References." arXiv preprint arXiv:2204.04715 (2022) [arXiv]
  • Zhongyun Hu, Ntumba Elie Nsampi, Xue Wang, Qing Wang: "NeSF: Neural Shading Field for Image Harmonization." arXiv preprint arXiv:2112.01314 (2021) [arXiv] (rendered images)
  • Zonghui Guo, Dongsheng Guo, Haiyong Zheng, Zhaorui Gu, Bing Zheng, Junyu Dong: "Image Harmonization with Transformer." ICCV (2021) [pdf] [supp] [code]
  • Yifan Jiang, He Zhang, Jianming Zhang, Yilin Wang, Zhe Lin, Kalyan Sunkavalli, Simon Chen, Sohrab Amirghodsi, Sarah Kong, Zhangyang Wang: "SSH: A Self-Supervised Framework for Image Harmonization." ICCV (2021) [pdf] [supp] [arXiv] [code]
  • Jun Ling, Han Xue, Li Song, Rong Xie, Xiao Gu: "Region-Aware Adaptive Instance Normalization for Image Harmonization." CVPR (2021) [pdf] [supp] [arXiv] [code]
  • Zonghui Guo, Haiyong Zheng, Yufeng Jiang, Zhaorui Gu, Bing Zheng: "Intrinsic Image Harmonization." CVPR (2021) [pdf] [supp] [code]
  • Wenyan Cong, Li Niu, Jianfu Zhang, Jing Liang, Liqing Zhang: "BargainNet: Background-Guided Domain Translation for Image Harmonization." ICME (2021) [arXiv] [code]
  • Konstantin Sofiiuk, Polina Popenova, Anton Konushin: "Foreground-aware Semantic Representations for Image Harmonization." WACV (2021) [pdf] [supp] [arXiv] [code]
  • Guoqing Hao, Satoshi Iizuka, Kazuhiro Fukui: "Image Harmonization with Attention-based Deep Feature Modulation." BMVC (2020) [pdf] [supp] [code]
  • Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, Liqing Zhang: "DoveNet: Deep Image Harmonization via Domain Verification." CVPR (2020) [pdf] [supp] [arXiv] [code].
  • Xiaodong Cun, Chi-Man Pun: "Improving the Harmony of the Composite Image by Spatial-Separated Attention Module." T-IP (2020) [pdf] [arXiv] [code]
  • Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, Ming-Hsuan Yang: "Deep Image Harmonization." CVPR (2017) [pdf] [supp] [arXiv] [code]

Unsupervised deep learning methods

  • Anand Bhattad, David A. Forsyth: "Cut-and-Paste Neural Rendering." arXiv preprint arXiv: 2010.05907 (2020) [arXiv] [supp]
  • Fangneng Zhan, Shijian Lu, Changgong Zhang, Feiying Ma, Xuansong Xie:"Adversarial Image Composition with Auxiliary Illumination." ACCV (2020) [pdf] [arXiv]
  • Bor-Chun Chen, Andrew Kae: "Toward Realistic Image Compositing With Adversarial Learning." CVPR (2019) [pdf]
  • Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, Alexei A. Efros: "Learning a Discriminative Model for the Perception of Realism in Composite Images." ICCV (2015) [pdf] [arXiv] [code]

Traditional methods

  • Shuangbing Song, Fan Zhong, Xueying Qin, Changhe Tu: "Illumination Harmonization with Gray Mean Scale." Advances in Computer Graphics. CGI (2020) [pdf]
  • Su Xue, Aseem Agarwala, Julie Dorsey, Holly E. Rushmeier: "Understanding and improving the realism of image composites." ACM Trans. Graph. 31(4): 84:1-84:10 (2012) [pdf]
  • Kalyan Sunkavalli, Micah K. Johnson, Wojciech Matusik, Hanspeter Pfister: "Multi-scale image harmonization." ACM Trans. Graph. 29, 4 (2010) [pdf]
  • Jean-François Lalonde, Alexei A. Efros: "Using Color Compatibility for Assessing Image Realism." ICCV (2007) [pdf] [code]

Datasets

  • iHarmony4: It contains four subdatasets: HCOCO, HAdobe5k, HFlickr, Hday2night, with a total of 73,146 pairs of unharmonized images and harmonized images. [pdf] [link]
  • GMSDataset: It contains 183 images with image resolution of 1940*1440. It consists of 16 different objects and for each object, one source image and 11 target images in different background scenes and illumination conditions are captured. [pdf] [link] (access code: ekn2)
  • HVIDIT: A dataset built upon VIDIT (Virtual Image Dataset for Illumination Transfer) dataset for image harmonization. It contains 3007 images of 276 scenes for training and 329 images of 24 scenes for testing. [pdf] [link]
  • RdHarmony: A rendered image harmonization dataset, which contains 25,000 ground-truth rendered images with 11 novel categories, and has the potential to generate 225,000 composite rendered images. [pdf] [link]
  • RealHM: A Real-world HarMonization dataset, which contains 216 real composite images with manually harmonized outputs. [pdf] [link]
  • ccHarmony: An image harmonization dataset constructed based on the images captured with color checker (cc). It contains 10 synthetic composite images for each of 426 foregrounds from 350 real images, leading to 4260 pairs of synthetic composite images and ground-truth real images. [pdf] [link]

Related Topics

Painterly/Style image harmonization

  • Junyan Cao, Yan Hong, Li Niu: "Painterly Image Harmonization in Dual Domains." AAAI (2023) [code]
  • Xiao Yan, Yang Lu, Juncheng Shuai, Sanyuan Zhang: "Style Image Harmonization via Global-Local Style Mutual Guided." ACCV (2022) [paper]
  • Hwai-Jin Peng, Chia-Ming Wang, Yu-Chiang Frank Wang: "Element-Embedded Style Transfer Networks for Style Harmonization." BMVC (2019) [paper]
  • Fujun Luan, Sylvain Paris, Eli Shechtman, Kavita Bala: "Deep Painterly Harmonization." Computer graphics forum (2018) [arXiv] [code]

Inharmonious region localization

  • Penghao Wu, Li Niu, Liqing Zhang: "Inharmonious Region Localization with Auxiliary Style Feature." BMVC (2022)
  • Jing Liang, Li Niu, Penghao Wu, Fengjun Guo, Teng Long: "Inharmonious Region Localization by Magnifying Domain Discrepancy." AAAI (2022) [arXiv] [code]
  • Jing Liang, Li Niu, Liqing Zhang: "Inharmonious Region Localization." ICME (2021) [arXiv] [code]

Video harmonization

  • Xinyuan Lu, Shengyuan Huang, Li Niu, Wenyan Cong, Liqing Zhang: "Deep Video Harmonization with Color Mapping Consistency." IJCAI (2022) [arXiv] [dataset&code]
  • Haozhi Huang, Senzhe Xu, Junxiong Cai, Wei Liu, Shimin Hu: "Temporally Coherent Video Harmonization Using Adversarial Networks." IEEE Trans. Image Process. 29: 214-224 (2020) [pdf] [arXiv]

Other Resources