The paper can be found in the proceedings: IJCAI 2024 Proceedings
Please cite this code and the paper as:
@InProceedings{deckers:2024a,
author = {Niklas Deckers and Julia Peters and Martin Potthast},
booktitle = {Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, {IJCAI-24}},
doi = {10.24963/ijcai.2024/845},
editor = {Kate Larson},
month = aug,
pages = {7636--7644},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
title = {{Manipulating Embeddings of Stable Diffusion Prompts}},
url = {https://doi.org/10.24963/ijcai.2024/845},
year = 2024
}
Additional data (./output
and ./user_study
) can be found in the supplementary material file on Zenodo.
- Docker Configuration: Refer to the Dockerfile at
./deployment/Dockerfile
. - Python Dependencies: All required Python packages are mentioned in
./deployment/requirements.txt
.
Transform the given prompt:
beautiful mountain landscape, lake, snow, oil painting 8 k hd
to
a beautiful and highly detailed matte painting of the epic mountains of avalon, intricate details, epic scale, insanely complex, 8 k, sharp focus, hyperrealism, very realistic, by caspar friedrich, albert bierstadt, james gurney, brian froud,
.
- Seed: 824331
- Execution: Run using
python3 ./interpolation/embedding_interpolation.py
- Output: Images can be found at
./output/beautiful mount_a beautiful an/
.
- Prompt list can be found in
./metric_based_optimization/datasets
. prompts.txt
:- A selection of 150 prompts from the diffusiondb: Available on huggingface
- Selected subsets:
large_random_100k
,large_random_1k
- Used for evaluating metric optimization.
LAION-Aesthetic-V2-prompts.txt
: Prompts utilized for Figure 8.
To begin, download the aesthetic predictor model weights from the repository of its creators: https://github.com/christophschuhmann/improved-aesthetic-predictor/raw/fe88a163f4661b4ddabba0751ff645e2e620746e/sac%2Blogos%2Bava1-l14-linearMSE.pth and place the file under ./aesthetic_predictor/sac+logos+ava1-l14-linearMSE.pth
.
- Source Code: Find the code in:
./metric_based_optimization/utils/aesthetic_metric_generalization.py
(Figure 9)./metric_based_optimization/full_pipeline_descent.py
(Figures 7 & 8)
- Result:
./output/metric_generalization/highly detailed photoreal eldritch biomechani/
- Examples: Refer to Figures 7, 8, and 9.
- User Interface: The interface is showcased in Figure 4. Run with
python3 ./iterative_human_feedback/user_interaction.py
- Results: See Figure 10.
- Prompt:
Single Color Ball
- Seeds: 683395, 417016, 23916, 871288, 383124
- Instructions: To execute, run
python3 ./seed_invariant_embeddings/utils/generate_images_with_varying_seeds.py
- Execution: Use
python3 ./seed_invariant_embeddings/prompt_embedding_space_traversal.py
- Output:
./output/universal_embeddings/embedding_space_traversal.pdf
- Notes: Interpolation values
[alpha]
and[beta]
can be obtained by executingpython3 ./seed_invariant_embeddings/utils/universal_embeddings_slerp.py
. Check the final print statement for specific values.
- Execute the methods
increase_blurriness()
andincrease_sharpness()
found in./metric_based_optimization/full_pipeline_descent.py
. - Output:
- Blurriness:
./output/metric_optimization/Blurriness/
- Sharpness:
./output/metric_optimization/Sharpness/
- Blurriness:
- Execution: Run
increase_aesthetic_score()
in./metric_based_optimization/full_pipeline_descent.py
- Output:
./output/metric_optimization/LAION-Aesthetics V2/
- Execution: Use
python3 ./metric_based_optimization/utils/aesthetic_metric_generalization.py
- Output:
./output/metric_generalization/highly detailed photoreal eldritch biomechani/
- Prompt Engineering UI: Launch the user interface for the prompt engineering reference method with
python3 ./iterative_human_feedback/utils/prompt_engineering.py
(see above for how to run our proposed method) - Seed-Invariance Software: The software to generate the images with the given prompt embedding files can be found in
./iterative_human_feedback/utils/embeddings_to_image.py
(note that the parameter which prompt embedding input to use must be adjusted inside the file) - User Study Questionnaires: View the questionnaires for our user study in
./user_study
.
- Resources: Find images and prompt embeddings for this study in
./user_study
.
- Execution: Run using
python3 ./seed_invariant_embeddings/universal_embeddings.py
- Results: Check
./output/universal_embeddings
.