Website | Project | Paper | Youtube | Discord
Nov 9th 2022 Update Code and dataset released.
Modelverse is a model sharing and search platform that contains a diverse set of deep generative models such as GANs, diffusion models, and autoregressive models. This platform allows users to find, share, and study deep generative models more easily.
Search by content demo
Search by title/author demo
Through Modelverse, we introduce the first content-based model search engine. Given a user query and a large set of generative models, we aim to the models that best match the query. The query could be an image, a sketch, a text description, another generative model, or a combination of these.
Content-Based Search for Deep Generative Models
Daohan Lu1*, Sheng-Yu Wang1*,
Nupur Kumari1*, Rohan Agarwal1*,
David Bau2,
Jun-Yan Zhu1.
Carnegie Mellon University1, Northeastern University2
arXiv:2210.03116, 2022
- Search models via text queries: https://modelverse.cs.cmu.edu/?q=wild+animals
- Search models via image queries.
- Search models via paper title: https://modelverse.cs.cmu.edu/?q=title:+diffusion+
- Search models via author: https://modelverse.cs.cmu.edu/?q=author:+tero+karras
Our search system consists of a pre-caching stage and an inference stage. Given a collection of models, we first generate 50K samples for each model. We then encode the images into image features and compute the 1st and 2nd order feature statistics for each model. The statistics are cached in our system for efficiency. At inference time, we support queries of different modalities (text, image, or sketch). We encode the query into a CLIP feature vector, and assess the similarity between the query feature and each model’s statistics. The models with the best similarity measures are retrieved.
Qualitative results of model retrieval. Below we show model retrieval results with three different modalities - images, sketches, and text.
Our method also enables multimodal queries (left) and using a model as a query to find similar models (right).
git clone git@github.com:generative-intelligence-lab/modelverse_backend.gitconda env create -f environment.yaml
conda activate modelsearch# Download model features and statistics
python model_features/download_model_features.py
# Download query data
python query_data/download_query_data.pyTo quickly try out our method with your query:
# model search using a text query
python demo.py --query_type text --input anime
# model search using an image query
python demo.py --query_type image --input examples/bedroom.jpg
# model search using a sketch query
python demo.py --query_type sketch --input examples/sketch_human_with_glasses.pngThis command will run evaluation according to the queries in query_data. The output will be saved into result.csv.
python eval.pyTo get features statistics from your own model. First generate 50k samples in a folder (<your_image_samples>), and decide a name for your model (<your_model_name>)
python gather_feats_and_stats.py --model_name <your_model_name> --model_image_folder <your_image_samples>The features will be stored in my_model_features by default.
We are constantly improving our search engine and incorporating additional features to Modelverse. If you encounter issues of our current system or have interesting features in your mind, please reach out to us. You can either fill in the feedback form or join our Discord discussion group. Your feedback and suggestions will be greatly appreciated
If you find this useful for your research, please cite the following work.
@article{lu2022content,
title={Content-Based Search for Deep Generative Models},
author={Lu, Daohan and Wang, Sheng-Yu and Kumari, Nupur and Agarwal, Rohan and Bau, David and Zhu, Jun-Yan},
journal = {arXiv preprint arXiv:2210.03116},
year = {2022}
}