Image Colorization

Question

Image Colorization

duxiangcheng opened this issue 9 months ago · 10 comments

duxiangcheng commented 9 months ago

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

pokameng commented 2 months ago

@yangxy

Answer 1 · 2024-04-10T03:53:41.000Z

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

The training procedure is similar to SR. You should change control_type to grayscale and rewrite the checkpoints/stable-diffusion-v1-5/schedule/scheduler_config.json. Please make sure to set rescale_betas_zero_snr=True and prediction_type=v_prediction.

In PASD, we only train the newly added layers due to limited data and gpus. I would recommend to finetune the whole model if possible.

Answer 2 · 2024-04-10T07:58:08.000Z

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

The training procedure is similar to SR. You should change control_type to grayscale and rewrite the checkpoints/stable-diffusion-v1-5/schedule/scheduler_config.json. Please make sure to set rescale_betas_zero_snr=True and prediction_type=v_prediction.

In PASD, we only train the newly added layers due to limited data and gpus. I would recommend to finetune the whole model if possible.

Thanks for your reply. I have another question. When making my own data set, do I still need the txt file corresponding to the image? If necessary, how to generate the corresponding text?

Answer 3 · 2024-04-10T09:02:29.000Z

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

The training procedure is similar to SR. You should change control_type to grayscale and rewrite the checkpoints/stable-diffusion-v1-5/schedule/scheduler_config.json. Please make sure to set rescale_betas_zero_snr=True and prediction_type=v_prediction.

In PASD, we only train the newly added layers due to limited data and gpus. I would recommend to finetune the whole model if possible.

If we need to generate captions, do we use grayscale images to generate captions during training and inference?

Answer 4 · 2024-04-15T07:20:04.000Z

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

The training procedure is similar to SR. You should change control_type to grayscale and rewrite the checkpoints/stable-diffusion-v1-5/schedule/scheduler_config.json. Please make sure to set rescale_betas_zero_snr=True and prediction_type=v_prediction.
In PASD, we only train the newly added layers due to limited data and gpus. I would recommend to finetune the whole model if possible.

If we need to generate captions, do we use grayscale images to generate captions during training and inference?

You can use BLIP2/CoCa to generate captions. You should use RGB(gray) images to get the captions during training(inference). Make sure to replace or discard the words such as black and white b&w etc. in the generated captions during inference.

Answer 5 · 2024-10-20T05:22:45.000Z

@yangxy
How can i train the colorization model based on sd-2.1

Answer 6 · 2024-10-20T09:33:32.000Z

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

The training procedure is similar to SR. You should change control_type to grayscale and rewrite the checkpoints/stable-diffusion-v1-5/schedule/scheduler_config.json. Please make sure to set rescale_betas_zero_snr=True and prediction_type=v_prediction.

In PASD, we only train the newly added layers due to limited data and gpus. I would recommend to finetune the whole model if possible.

Do i need to set rescale_betas_zero_snr=True when trainning? or set rescale_betas_zero_snr=True in both training and test time?

Answer 7 · 2024-10-21T03:23:23.000Z

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

The training procedure is similar to SR. You should change control_type to grayscale and rewrite the checkpoints/stable-diffusion-v1-5/schedule/scheduler_config.json. Please make sure to set rescale_betas_zero_snr=True and prediction_type=v_prediction.
In PASD, we only train the newly added layers due to limited data and gpus. I would recommend to finetune the whole model if possible.

Do i need to set rescale_betas_zero_snr=True when trainning? or set rescale_betas_zero_snr=True in both training and test time?

In both training and test time. sd2.1 works.

Answer 8 · 2024-10-21T03:45:35.000Z

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

The training procedure is similar to SR. You should change control_type to grayscale and rewrite the checkpoints/stable-diffusion-v1-5/schedule/scheduler_config.json. Please make sure to set rescale_betas_zero_snr=True and prediction_type=v_prediction.
In PASD, we only train the newly added layers due to limited data and gpus. I would recommend to finetune the whole model if possible.

Do i need to set rescale_betas_zero_snr=True when trainning? or set rescale_betas_zero_snr=True in both training and test time?

In both training and test time. sd2.1 works.

If I only do the colorization task on controlnet, do I also need to set these two parts during training and testing? In addition, if I use COCO2017 for training, how long does it usually take to train it to work? Or how many steps? I run in 8*RTX4090,batch=10

Answer 9 · 2024-10-21T06:55:16.000Z

Thank you for sharing your work. I want to know how to train a colorization model on my own dataset.

The training procedure is similar to SR. You should change control_type to grayscale and rewrite the checkpoints/stable-diffusion-v1-5/schedule/scheduler_config.json. Please make sure to set rescale_betas_zero_snr=True and prediction_type=v_prediction.
In PASD, we only train the newly added layers due to limited data and gpus. I would recommend to finetune the whole model if possible.

Do i need to set rescale_betas_zero_snr=True when trainning? or set rescale_betas_zero_snr=True in both training and test time?

In both training and test time. sd2.1 works.

If I only do the colorization task on controlnet, do I also need to set these two parts during training and testing? In addition, if I use COCO2017 for training, how long does it usually take to train it to work? Or how many steps? I run in 8*RTX4090,batch=10

Yes, I think so. This is because the color leaking issue during training. Please refer to the Supplementary Materials of our paper. And rescale_betas_zero_snr=True is critical while prediction_type=v_prediction is optional.

I am not sure about that training time based on your settings. Maybe you should conduct some exps.