zoom_enhance with controlnet crashes, python stack strace

[jpentland]

Due diligence

• I checked for similar issues and couldn't find any.
• My WebUI and Unprompted are both up-to-date.
• I disabled my other extensions but the problem persists.

Describe the bug

If I use the following prompt:

man standing in a city street, pro wrestler, cyberpunk [after][zoom_enhance replacement="tiger head"][/after]

This seems to work ok. However, I want to increase the denoising strength, and use controlnet to ensure that the replaced image matches the surrounding image. So I would change the prompt to this:

man standing in a city street, pro wrestler, cyberpunk [after][zoom_enhance controlnet_preset=inpainting denoising_strength=0.8 replacement="tiger head"][/after]

With the "inpainting" file created at extensions/unprompted/templates/common/presets/controlnet/inpainting.txt:

[sets 
 cn_0_enabled=1 cn_0_pixel_perfect=1 cn_0_module=inpaint_only cn_0_model=control_v11p_sd15_inpaint cn_0_weight=1.0 cn_0_control_mode=1
 ]

The values inpaint_only and control_v11p_sd15_inpaint were copied from the controlnet extension list of options, which works if selected in the controlnet extension on the img2img tab.

However, when running this I get the following result:

• the base image is generated
• the replacement image is generated, without controlnet
• the replacement image is not pasted onto the base image
• there is the following python stack trace displayed in the terminal:

 2023-09-29 13:26:44,998  (INFO)     [Unprompted.txt2mask] Using cached clipseg model.
 2023-09-29 13:26:45,083  (WARNING)  [Unprompted.file] As of v9.14.0, [file] is a legacy shortcode and will eventually be removed in favor of [call] - the main difference is that [call] also works with functions.
 2023-09-29 13:26:45,084  (WARNING)  [Unprompted.zoom_enhance] Processing mode: Shortcode-Based (may not be compatible with ControlNet)
Total progress: 40it [00:07,  5.62it/s]*** Error running postprocess: /home/user/stable-diffusion-webui/extensions/_unprompted/scripts/unprompted.pyress: 40it [00:07,  6.12it/s]5044]:
     Traceback (most recent call last):
       File "/home/user/stable-diffusion-webui/extensions/_unprompted/lib_unprompted/shortcodes.py", line 115, in render
         return str(self.handler(self.token.keyword, self.pargs, self.kwargs, context))
       File "/home/user/stable-diffusion-webui/extensions/_unprompted/lib_unprompted/shared.py", line 65, in handler
         return (self.shortcode_objects[f"{keyword}"].run_atomic(pargs, kwargs, context))
       File "/home/user/stable-diffusion-webui/extensions/_unprompted/shortcodes/stable_diffusion/zoom_enhance.py", line 460, in run_atomic
         fixed_image = unsharp_mask(fixed_image, sharpen_amount)
       File "/home/user/stable-diffusion-webui/extensions/_unprompted/shortcodes/stable_diffusion/zoom_enhance.py", line 44, in unsharp_mask
         image = numpy.array(image).astype(numpy.uint8)
     ValueError: invalid literal for int() with base 10: ''
     The above exception was the direct cause of the following exception:
     Traceback (most recent call last):
       File "/home/user/stable-diffusion-webui/modules/scripts.py", line 651, in postprocess
         script.postprocess(p, processed, *script_args)
       File "/home/user/stable-diffusion-webui/extensions/_unprompted/scripts/unprompted.py", line 835, in postprocess
         processed = Unprompted.after(p, processed)
       File "/home/user/stable-diffusion-webui/extensions/_unprompted/lib_unprompted/shared.py", line 183, in after
         val = self.shortcode_objects[i].after(p, processed)
       File "/home/user/stable-diffusion-webui/extensions/_unprompted/shortcodes/basic/after.py", line 73, in after
         self.Unprompted.process_string(content, "after")
       File "/home/user/stable-diffusion-webui/extensions/_unprompted/lib_unprompted/shared.py", line 190, in process_string
         string = self.shortcode_parser.parse(self.sanitize_pre(string, self.Config.syntax.sanitize_before), context)
       File "/home/user/stable-diffusion-webui/extensions/_unprompted/lib_unprompted/shortcodes.py", line 251, in parse
         return stack.pop().render(context)
       File "/home/user/stable-diffusion-webui/extensions/_unprompted/lib_unprompted/shortcodes.py", line 55, in render
         return ''.join(child.render(context) for child in self.children)
       File "/home/user/stable-diffusion-webui/extensions/_unprompted/lib_unprompted/shortcodes.py", line 55, in <genexpr>
         return ''.join(child.render(context) for child in self.children)
       File "/home/user/stable-diffusion-webui/extensions/_unprompted/lib_unprompted/shortcodes.py", line 119, in render
         raise ShortcodeRenderingError(msg) from ex
     lib_unprompted.shortcodes.ShortcodeRenderingError: An exception was raised while rendering the 'zoom_enhance' shortcode in line 1.
 ---

Prompt

see above

Log output

see above

Unprompted version

Unprompted v9.16.1

WebUI version

v1.6.0-60-g7ab82787

Other comments

ControlNet v1.1.410

[ComplexRobot]

I browsed though the source code, and to my understanding, unprompted doesn't set the controlnet image value automatically (the error is saying it expects a numpy array, AKA the image, but it's receiving nothing). The expected result would be to do something like adetailer and automatically set the controlnet image to the current output (with the generated mask, if applicable). With unprompted, all you can do is set the controlnet image as a path to a file.
Hopefully that functionality will be added, but as of right now all you can do is write a custom shortcode (python code) to do this, or you're stuck with manually setting the image in the controlnet UI (useless without the mask).