It is only hallucinating
nafrinafri opened this issue · 17 comments
Installed Diffusers from Mac App Store on a MBP-15 (2018, Intel UHD Graphics 630/Radeon Pro 555X with 4GB VRAM). After downloading the SD2 model, I tried generating an image with "Labrador in the style of Vermeer" prompt and default settings. The output image was totally different from what was mentioned in the prompt. Also tried the Small Stable Diffusion Model. Same thing happened. Shall I attach screenshots?
Yes, please! I haven't been able to test it on mixed GPU systems myself. It'd also be helpful if you could select a random seed and post which one it is. Thanks a lot!
Here are two -- one each for SD2 and Small SD -- with default settings:
This one is with a seed value of 509:
Thanks a lot. That certainly doesn't look right. I suppose the app is using the discrete GPU, but it'd be interesting to confirm it. Could you please try to follow these steps to verify? https://support.apple.com/en-us/HT202053
Using the above link, found that MBP was not using the discrete GPU. The reason -- automatic graphics switching was turned on by default -- was given in https://support.apple.com/en-us/HT202043. The remedy was simple: "deselect the automatic graphics switching checkbox". Did that. As per Activity Monitor, Diffusers is still not using the discrete GPU; and still generating hallucinated images. (BTW Photos app is now using the discrete GPU.)
It seems that Diffusers fails to work with discrete GPU, if present (and switched on). Correct?
It also seems that Diffusers fails to work with mixed GPU systems. Correct?
Thanks @nafrinafri! I'll try to force the use of the discrete GPU to see if it solves this problem; unfortunately, I don't have a Mac right now that uses two GPUs and can be upgraded to Ventura, so it'll be a bit challenging to test. I'll keep you updated.
Thank you, @pcuenca!
Sharing another data point as I have the same issue. I'm on an Intel iMac with a 4GB discrete graphics card. I tried the same prompt and seed as @nafrinafri (but not sure if same model, I used stable diffusion 2.1) but got a different hallucination. Activity Monitor confirmed it was using the discrete GPU while creating the image.
Based on our earlier conversation, I assumed that the app was tested on a system with integrated graphics. I just tested Diffusers on a MBP-13 (2017, Intel Iris Plus Graphics 640 1536 MB). The system has 16GB RAM and is running macOS Ventura 13.2; and the app was downloaded from the Mac App Store. After downloading the SD2 model, I tried generating an image using "Labrador in the style of Vermeer" prompt and default settings. Was greeted with "Safety checker triggered, please try a different prompt or seed" message. Changed seed a number of times. Same error message. Also tried with a different prompt ("red apple") with various seed values. Same error. Also repeated all of the above with the Small Stable Diffusion Model. Same thing happened.
Sharing another data point as I have the same issue. I'm on an Intel iMac with a 4GB discrete graphics card. I tried the same prompt and seed as @nafrinafri (but not sure if same model, I used stable diffusion 2.1) but got a different hallucination. Activity Monitor confirmed it was using the discrete GPU while creating the image.
@tommy9 I used Stable Diffusion 2 and Small Stable Diffusion in both instances (i.e. on a mixed GPU as well as on a system with only integrated GPU).
Thanks. So by using the same model, I can reproduce the exact same image with seed 509. So not random hallucinations dependent on the exact hardware configuration, just somehow not aligning the prompt with the image generation.
@tommy9 I just downloaded SD 2.1 and can now verify that I got the same image that you posted earlier. So, yes, it is not a "random" hallucination but their is something terribly mislaligned.
@pcuenca So far, @tommy9 and I have checked it on three hardware (two with mixed GPU systems, and one with integrated GPU). Additionally, I have tried it with automatic graphics switching turned on and off. No joy. Could you please tell me about the hardware it was tested on. I believe that there are many of us who want to run SD on local hardware, and this is a very promising solution.
@nafrinafri @tommy9 We have tested on several M1, M2 and Intel computers, but it's hard to get access to systems with 4 GB of VRAM that can also run Ventura (even in cloud services). If you could download the project from GitHub and run it with Xcode I could prepare a development branch to gather some logs to try to determine whether the problem is solvable.
Hi @pcuenca, I've been able to download and run the project locally, so happy to test any development branch to gather data.
I have an iMac on Ventura with 3 GHz 6-Core Intel Core i5, Radeon Pro 570X 4 GB and I get similar, unrelated results.
Same here, Intel i5 with Radeon Pro 580X 8 GB getting unrelated results