This colab notebook (https://colab.research.google.com/drive/1nCoa55cm8JpIXxBW9RaVSvb8B1WO2ZFz?usp=sharing) is an attempt to automatically optimize stable diffusion prompts.
The rationale is that for a good prompt, you need to have the objects, art movement, art medium, artists, and flavor modifiers.
For all these tags, the closer they are, the most harmonious they are and thus produces higher quality images from stable diffusion. For example, for an impressionism painting (an art movement), it pairs well with Claude Monet, Edouard Manet, Pierre-Auguste Renoir (artists), oil paiting (medium), and unblended color and natural lights (flavors).
To do this process automatically, I did the following:
- Collected lists of art movements, artists, art medium, and flavors tags that work in stable diffusion.
- Turn them into vector embeddings with pre-trained embeddings.
- Compare them with the user's existing tags to find art movements, artists, art medium, and flavors to add to the prompt.
To use the notebook, first read and run the pre-req section, put the tags files from Github (https://github.com/Troyanovsky/stable_diffusion_prompt_optimizer) into your google drive.
The first section creates embeddings as preparation. It only needs to be run once.
The second section is for optimizng the prompt. Once it's run, you can just change and run the "Inference" section.
Before: mountains and rivers, traditional chinese painting
After: mountains and rivers,traditional chinese painting,national geographic,Theophanes the Greek,a renaissance painting,arts and crafts movement
