Cant convert to diffusers

[brurpo]

Hi! Thanks for the awesome work!
Unfortunatelly I am stuck on the last step.

It throws the following error when trying to convert to diffusers:

Transferred model saved at E:\sd\output_gui2/models//control_anything-v3.0_canny/control_anything-v3.0_canny.pth global_step key not found in model In this conversion only the non-EMA weights are extracted. If you want to instead extract the EMA weights (usually better for inference), please make sure to add the --extract_ema flag. Traceback (most recent call last): File "E:\sd\output_gui2\sdiff_GUI.py", line 434, in <module> convertControl(['--checkpoint_path', (dir_path + "/models/" + '/control_' + values["mod"] + "_canny/" + 'control_' + values["mod"] + "_canny.pth"), '--dump_path', (dir_path + "/models/" + 'control_' + values["mod"] + "_canny" ),'--device', 'cpu']) File "E:\sd\output_gui2\convert_controlnet_to_diffusers.py", line 105, in main pipe = load_pipeline_from_control_net_ckpt( File "C:\Users\brurp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 1549, in load_pipeline_from_control_net_ckpt converted_ctrl_checkpoint = convert_controlnet_checkpoint(checkpoint, unet_config) File "C:\Users\brurp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 584, in convert_controlnet_checkpoint new_checkpoint["time_embedding.linear_1.weight"] = unet_state_dict["time_embed.0.weight"] KeyError: 'time_embed.0.weight'

[haofanwang]

Could you provide your full converting command so that I can help you? @brurpo

[brurpo]

Sure! I am doing it within my own gui.
It converts to a safatensor first if it does not exists, then, creates the folder and the control net .pth. It fails on the next step, to convert this pth to diffusers.
basically it fails this command on your readme:
python ./scripts/convert_controlnet_to_diffusers.py --checkpoint_path control_any3_openpose.pth --dump_path control_any3_openpose --device cpu

my code inside my gui is the following:

if event == 'Convert current model to control canny':
            window.Disable()
            window.find_element("message").update(value="Converting model, please wait!", text_color='red')
            window.Refresh()
            window.read(timeout=1)
            pop = popup('Please wait...')
           
            if not os.path.exists(dir_path + "/models/" + values["mod"] + '/' + values["mod"] + ".safetensors" ):
                convertToSafe(['--model_path', (dir_path + "/models/" + values["mod"]), '--checkpoint_path', (dir_path + "/models/" + values["mod"] + '/' + values["mod"] + ".safetensors"), '--half', '--use_safetensors'])

            if not os.path.exists(dir_path + "/models/" + '/control_' + values["mod"] + "_canny/"):
               os.makedirs(dir_path + "/models/" + '/control_' + values["mod"] + "_canny/")

            transferControl(['--path_sd15', (dir_path + "/control/v1-5-pruned.ckpt"), '--path_sd15_with_control', (dir_path + "/control/v1-5-pruned.ckpt"), '--path_input', (dir_path + "/models/" + values["mod"] + '/' + values["mod"] + ".safetensors" ),'--path_output', (dir_path + "/models/" +  '/control_' + values["mod"] + "_canny/" +  'control_' + values["mod"] + "_canny.pth")])

            convertControl(['--checkpoint_path', (dir_path + "/models/" + '/control_' + values["mod"] + "_canny/" + 'control_' + values["mod"] + "_canny.pth"), '--dump_path', (dir_path + "/models/" +  'control_' + values["mod"] + "_canny" ),'--device', 'cuda:0'])

            pop.close()
            window.find_element("message").update(value="Iddle...", text_color='white')
            window.Enable()

Its basically a step by step of your readme. I am doing it with anything v3, same as the tutorial.
values["mod"] is the selected model

edit:
I also tried directly through the command line using:
python convert_controlnet_to_diffusers.py --checkpoint_path "E:\sd\output_gui2\models\control_anything-v3.0_canny\control_anything-v3.0_canny.pth" --dump_path "E:\sd\output_gui2\models\control_anything-v3.0_canny" --device cpu
It also results in the same error.
Weirdly enough, I went to 0.13.1 to test other things, and when I got back to yours, the error now changed to:
KeyError: 'time_embed.2.weight'

[haofanwang]

@brurpo I cannot reproduce the problem. Do you install diffusers as our tutorial?

Also, could you check your checkpoint with following codes?

checkpoint = torch.load("control_any3_openpose.pth")

# extract state_dict for UNet
unet_state_dict = {}
keys = list(checkpoint.keys())

unet_key = "control_model."

for key in keys:
  if key.startswith(unet_key):
    unet_state_dict[key.replace(unet_key, "")] = checkpoint.pop(key)

print("time_embed.0.weight" in unet_state_dict)

print("time_embed.2.weight" in unet_state_dict)

[brurpo]

Hi @haofanwang! Thanks for all your time!
I did! git cloned then changed to 9a37409663a53f775fa380db332d37d7ea75c915
I ran your code, it just threw 2 Falses

False
False

Maybe something went wrong converting it to pytorch?

[brurpo]

I reinstalled controlnet.
Now it returns TRUE for both lines.
Unfortunatelly it now throws a different error on the last step.

RuntimeError: Error(s) in loading state_dict for UNet2DConditionModel:
        Missing key(s) in state_dict: "time_embedding.linear_1.weight", "time_embedding.linear_1.bias".

I edit your script to:

import torch
checkpoint = torch.load("E:\\sd\\output_gui2\\models\\control_anything-v3.0_canny\\control_any3_canny.pth")

# extract state_dict for UNet
unet_state_dict = {}
keys = list(checkpoint.keys())

unet_key = "control_model."

for key in keys:
  if key.startswith(unet_key):
    unet_state_dict[key.replace(unet_key, "")] = checkpoint.pop(key)

print("time_embed.0.weight" in unet_state_dict)
print("time_embed.1.weight" in unet_state_dict)
print("time_embed.2.weight" in unet_state_dict)

It now returns:

True
False
True

[brurpo]

Sorry to bother @haofanwang, but, would you mind, if you have the time, trying to convert my converted pth? So we can know if the problem is in the conversion to pth or the conversion to diffusers?
if so, here is the link:
Thanks again

[brurpo]

hey, @haofanwang, sorry to bother again.
The following lines (584 and 585) on convert_to_ckpt.py were commented

    new_checkpoint["time_embedding.linear_1.weight"] = unet_state_dict["time_embed.0.weight"]
    new_checkpoint["time_embedding.linear_1.bias"] = unet_state_dict["time_embed.0.bias"]

I`ve uncommented them and it now seems to be working. Were they supposed to be commented?