how to do multi-image to 3d?
biswaroop1547 opened this issue · 10 comments
Just curious if there's any possibility to do multi-image to 3d here?
Good question! I though for a while and actually it is possible with the single image conditioned TRELLIS model. The sampling process of Rectified Flow model contain multiple steps, we can switch to different reference image in each step. Though this is a approximation for the multi-image condition but I think it may work well.
I am going to implement this scheme. If it works, I will add the inference pipeline in the repo!
awesome! do let me know if I can help anywhere
thank you 🙏
I just wanted to say that this is amazing. I think multi-image (if possible -- that would be INSANE) would be really cool. It can help with the visualization of the antique furniture from my shop -- it is already very good with one image, though!!!
Good question! I though for a while and actually it is possible with the single image conditioned TRELLIS model. The sampling process of Rectified Flow model contain multiple steps, we can switch to different reference image in each step. Though this is a approximation for the multi-image condition but I think it may work well.
I am going to implement this scheme. If it works, I will add the inference pipeline in the repo!
wow you guys are superheroes. thatd be so cool
@JeffreyXiang how is the work going on TRELLIS/multi-image-view. Can I support you on coding and thoughts?
Thanks in advance for your reply.
You can also average the conditionings of many input images.
This only works if your input images are somewhat close together though, like 2 characters in the same pose for example.
Definitely wouldn't work for my use case because I want to be able to give the algorithm more clues about how my item looks like.
would this also work for single object but multiple perspectives?
We have update an experimental multi-view image support. Please see the latest update and the new features on our Hugging face demo.
Very solid work. Works well with my furniture. Not perfect but I think multiple images is making a difference. PNGs seem to work better than JPGs. I will try to integrate some of the example vids soon!