Style2Paints_V3

Pytorch Reimplementation of Style2Paints V3 (https://github.com/lllyasviel/style2paints/blob/master/papers/sa.pdf)

Results

figure 1_1 您的老婆1	figure 1_2 您的老婆1
figure 2_1 您的老婆2	figure 2_2 您的老婆2
figure 3_1 您的老婆3	figure 3_2 您的老婆3
figure 4_1 您的老婆4	figure 4_2 您的老婆4
figure 5_1 您的老婆5	figure 5_2 您的老婆5
figure 6_1 您的老婆6	figure 6_2 您的老婆6
figure 7_1 您的老婆7	figure 7_2 您的老婆7
figure 8_1 您的老婆8	figure 8_2 您的老婆8
figure 9_1 您的老婆9	figure 9_2 您的老婆9

Step1 : Dataset Simulation

One should modify simulate_step.ipynd or simulate_step.py with your own data path before runing this script.

Simulate_step1 : Random Region Proposal and Pasting

See script : simulate_step1.ipynb

Simulate_step2 : Random transform

See script : simulate_step2.ipynb

Simulate_step3 : Random color spray

I merged this part with the Pytorch data-loader. Refer to ./Pytorch-Style2paints/dataset_multi.py

Effect Picture

Ground truth	Color draft

User hints

I chose [0, 20) user hints from the ground truth randomly, and pasted them on a hint map with 3 channels( RGB while the paper used RGBA )

Ground truth	User hints

Network structure

Train

Just run train.py. Don't forget to modify the easydict in the train.py script!

All the hyper-parameters are chosen as same as the original paper.

args = easydict.EasyDict({
    'epochs' : 100,
    'batch_size' : 16,
    'train_path' : 'train data path'#'./your train data path/train',
    'val_path' : 'val data path'#'./your val data path/val',
    'sketch_path' : 'sketch path'#"./your sketch data path/sketch",
    'draft_path' : 'STL path'#"./your STL data path/STL",
    'save_path' : 'result path'#"./your save path/results" ,
    'img_size' : 270,
    're_size' : 256,
    'learning_rate' : 1e-5,
    'gpus' : '[0,1,2,3]',
    'lr_steps' : [5, 10, 15, 20],
    "lr_decay" : 0.1,
    'lamda_L1' : 0.01,
    'workers' : 16,
    'weight_decay' : 1e-4
})

Output of the data-loader & I/O of the Model

Data-loader:

for i, (input, df, gt) in enumerate(train_loader):

Input : (batch_size, 4, 256, 256) : sketch(1 channel) + hint map(3 channels)

df : (batch_size, 3, 224, 224) : simulated data which is the input of Inception V1

gt: (batch_size, 3, 256, 256) : ground truth

I/O of the Inception V1:

INPUT: df : (batch_size, 3, 224, 224) : simulated data which is the input of Inception V1

OUTPUT: latent code (batch_size, 1024, 1, 1)

I/O of the Style2Paints_V3:

INPUT: Input : (batch_size, 4, 256, 256) : sketch(1 channel) + hint map(3 channels)

OUTPUT: output : (batch_size, 3, 256, 256) : colorized manuscripts

ChenYuxu4n/Style2Paints_V3