A Presumptuous Suggestion: Adding Localized Redraw Based on Image-to-Image (Like Inpainting)
Opened this issue · 4 comments
The current flux painting approach involves a high degree of randomness. Tools like LoRA, ControlNet, and image-to-image generation are all aimed at providing users with more control to mitigate this randomness. Therefore, I would like to propose the possibility of adding a “localized redraw” feature based on image-to-image generation:
For example, when users upload a reference image, they could also upload an image mask to overlay on the reference. This could potentially enable the following:
Setting the reference image intensity to 0 within the mask area, allowing flux to freely generate content there, while keeping the reference image intensity at 1 outside the mask, preserving the original details. This would effectively achieve “localized redraw.”
In a generation process filled with randomness, users might find some images broadly satisfying while only disliking certain parts. Allowing the specification of these areas for flux to redraw could significantly enhance user control.
I am not familiar with the mlx code structure, and I don’t know how much effort this feature would require to implement. Therefore, this is merely a suggestion for the official team’s reference.
I think you're referring to "in-painting" and there is a reference implementation over at https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/flux/pipeline_flux_inpaint.py, very doable but I personally don't have the time in near term to follow up (after similar work in #77), would love to see someone pick it up though!
Because the masked region needs to be drawn, a GUI is ideally provided for this, otherwise the mask image needs to be drawn in a different app, then saved as a hypothetical file arg --inpaint-mask-image-path, a high friction workflow.
@anthonywu Thank you for your quick response. My English is very poor, so I hope it didn’t make things difficult to understand. I actually always assumed you were a “湾区大佬,” based on your username—maybe Mr. 吴 or Mr. 伍. Please forgive me if I got that wrong! But these are all just topics outside of the main subject, as we still need to use English for communication here.
I feel quite regretful to hear that you don’t have time at the moment, but I appreciate your acknowledgment of this feature's feasibility. All I can do now is wait. Since I can’t resolve this myself, it remains a suggestion purely for the official team’s reference.
You also mentioned that implementing this through the UI might be better. The current UI options are all unofficial, and I noticed your new project also relates to UI functionality. I hope you complete it, and, if possible, make it the official version to enable testing of features that are operable via the terminal but not convenient to use. Best wishes.
@raysers your English is excellent, the issue you filed is clear.
Today I noticed https://github.com/CharafChnioune/MFLUX-WEBUI (another web ui effort by a frequent poster in this project's issues). That UI shows great progress and may be the better first UI to host the in-painting feature whenever that can be done.
@anthonywu Yes, MFLUX-WEBUI is an impressive project. Honestly, my ComfyUI plugin was initially inspired by it, to the point that it still carries some traces of WEBUI. My original intention was to design foundational MFLUX nodes that could be combined with existing plugins in ComfyUI, allowing MFLUX to fully realize its potential. But, unfortunately, I simply don’t have the capability to complete such a deep integration—it’s like “纸上谈兵.” For now, I can only create data transmission nodes that pass data to flux.py to generate images.
I may have gone off-topic. Hearing that you might test the in-painting feature on MFLUX-WEBUI genuinely makes me happy. When you have time to implement it, I’ll be eagerly awaiting. And, of course, if other contributors are interested in following up on this feature, as you mentioned, I’d be happy to assist with testing—that’s the least I can contribute