PAL4Inpaint

[Project Page] | Paper | Bibtex

Perceptual Artifacts Localization for Inpainting
European Conference on Computer Vision (ECCV), 2022, Oral Presentation
Lingzhi Zhang, Yuqian Zhou, Connelly Barnes, Sohrab Amirghodsi, Zhe Lin, Eli Shechtman, Jianbo Shi
University of Pennsylvania, Adobe Research/ART

Note: Due to commercial reasons, we release only the inference code and a subset of the datasets.

Prerequisites

Linux
Python 3
NVIDIA GPU + CUDA CuDNN

Setup

Clone this repo:

git clone https://github.com/owenzlz/PA4Inpaint

Install dependencies (Since we use torchscript, we can bypass the mmsegmentation packages.):

pip install torch torchvision

Datasets

You can download our labeled datasets using the following command.

Download our datasets

bash download_datasets.sh

After downloading, the dataset is structured as follows:

- [perceptual artifacts dataset root]
    |- trainset_subset
        |- images
        |- masks
        |- labels
    |- valset
        |- images
        |- masks
        |- labels
    |- testset
        |- images
        |- masks
        |- labels

Note that labels and masks have pixel values 0 or 1. To visualize them, you can multiple the images by 255. We provide a simple script to visualize a few sampled data.

python visualize_labels.py

Checkpoints

Download checkpoints in the 'torchscript' format:

bash download_checkpoints.sh

Perceptual Artifacts Segmentation

Run the following command to predict the perceptual artifacts localization on the inpainted images.

Inference on a single image:

python pal4inpaint.py \
       --img_file ./testimages/images/a_alcove_00004858.png \
       --output_seg_file ./testimages/seg/a_alcove_00004858.png \
       --output_vis_file ./testimages/vis/a_alcove_00004858.png

Inference on a batch of images:

python pal4inpaint.py \
       --img_dir ./demo/images \
       --output_seg_dir ./demo/results/seg \
       --output_vis_dir ./demo/results/vis

Perceptual Artifacts Ratio (PAR) [TODO]

Compute PAR metric on a single image:

python par.py --img_file ./demo/images/xxx.jpg

Compute PAR metric on a batch of images:

python par.py --img_dir ./demo/images

Iterative Fill [TODO]

Localize perceptual artifacts region and run inpainting on it iteratively.

python iterative_fill.py

The blue contour in the first image indicates the original hole, and the pink contours in each images indicate the localized perceptual artifacts region at each iterative fill.

Citation

If you use this code for your research, please cite our paper:

@article{zhang2022perceptual,
  title={Perceptual Artifacts Localization for Inpainting},
  author={Zhang, Lingzhi and Zhou, Yuqian and Barnes, Connelly and Amirghodsi, Sohrab and Lin, Zhe and Shechtman, Eli and Shi, Jianbo},
  journal={arXiv preprint arXiv:2208.03357},
  year={2022}
}

owenzlz/PAL4Inpaint