sw-yx/prompt-eng

notes for prompt engineering

prompt-eng

notes for prompt engineering

Table of Contents

• Motivational Use Cases
• Top Prompt Engineering Reads
• Tooling
• Communities
• Stable Diffusion
  • SD Distros
  • SD Major forks
  • SD Prompt galleries and search engines
  • SD Visual search
  • SD Prompt generators
  • Img2prompt - Reverse Prompt Engineering
  • Explore Artists, styles, and modifiers
  • SD Prompt Tools directories and guides
• How SD Works - Internals and Studies
• SD Results
  • Img2Img
  • SD 1.4 vs 1.5 comparisons
• Hardware requirements
• SD vs DallE vs MJ
• Misc
• Quotes, Reality & Demotivation
• Infrastructure

Motivational Use Cases

• images
  • https://mpost.io/best-100-stable-diffusion-prompts-the-most-beautiful-ai-text-to-image-prompts/
  • 3D MRI synthetic brain images - positive reception from neuroimaging statistician
  • multiplayer stable diffusion
• video
  • img2img of famous movie scenes (lalaland)
    • img2img transforming actor with ebsynth + koe_recast
  • virtual fashion (karenxcheng)
  • seamless tiling images
  • evolution of scenes (xander)
  • outpainting https://twitter.com/orbamsterdam/status/1568200010747068417?s=21&t=rliacnWOIjJMiS37s8qCCw
  • webUI img2img collaboration https://twitter.com/_akhaliq/status/1563582621757898752
  • image to video with rotation https://twitter.com/TomLikesRobots/status/1571096804539912192
  • "prompt paint" https://twitter.com/1littlecoder/status/1572573152974372864
  • music videos
    • video killed the radio star, colab This uses OpenAI's Whisper speech-to-text, allowing you to take a YouTube video & create a Stable Diffusion animation prompted by the lyrics in the YouTube video
    • Stable Diffusion Videos generates videos by interpolating between prompts and audio
  • direct text2video project
    • https://twitter.com/_akhaliq/status/1575546841533497344
    • https://makeavideo.studio/ - explorer https://webvid.datasette.io/webvid/videos
    • https://phenaki.video/
    • https://github.com/THUDM/CogVideo
    • https://imagen.research.google/video/
• text-to-3d https://twitter.com/_akhaliq/status/1575541930905243652
  • https://dreamfusion3d.github.io/
  • open source impl: https://github.com/ashawkey/stable-dreamfusion
  • demo https://twitter.com/_akhaliq/status/1578035919403503616
• text products
• Jasper
• gpt3 email https://github.com/sw-yx/gpt3-email
• gpt3() in google sheet 2020, 2022 - sheet
• https://www.summari.com/ Summari helps busy people read more
• sequoia market map https://twitter.com/sonyatweetybird/status/1584580362339962880?s=12&t=DEyWoA1Azh0Z5btVqoTY4g

Top Prompt Engineering Reads

The more advanced GPT3 reads have been split out to https://github.com/sw-yx/prompt-eng/blob/main/GPT.md

• https://www.gwern.net/GPT-3#prompts-as-programming
• beginner
  • openAI prompt tutorial https://beta.openai.com/docs/quickstart/add-some-examples
  • DALLE2 prompt writing book http://dallery.gallery/wp-content/uploads/2022/07/The-DALL%C2%B7E-2-prompt-book-v1.02.pdf
  • https://medium.com/nerd-for-tech/prompt-engineering-the-career-of-future-2fb93f90f117
  • https://wiki.installgentoo.com/wiki/Stable_Diffusion overview
  • https://www.reddit.com/r/StableDiffusion/comments/x41n87/how_to_get_images_that_dont_suck_a/
  • https://mpost.io/best-100-stable-diffusion-prompts-the-most-beautiful-ai-text-to-image-prompts/
  • https://andymatuschak.org/prompts/
  • for nontechnical
    • https://www.jonstokes.com/p/ai-content-generation-part-1-machine
• Intermediate
  • DALLE2 asset generation + inpainting https://twitter.com/aifunhouse/status/1576202480936886273?s=20&t=5EXa1uYDPVa2SjZM-SxhCQ
  • suhail journey https://twitter.com/Suhail/status/1541276314485018625?s=20&t=X2MVKQKhDR28iz3VZEEO8w
  • composable diffusion - "AND" instead of "and" https://twitter.com/TomLikesRobots/status/1580293860902985728
  • quest for photorealism https://www.reddit.com/r/StableDiffusion/comments/x9zmjd/quest_for_ultimate_photorealism_part_2_colors/
    • https://medium.com/merzazine/prompt-design-for-dall-e-photorealism-emulating-reality-6f478df6f186
  • settings tweaking https://www.reddit.com/r/StableDiffusion/comments/x3k79h/the_feeling_of_discovery_sd_is_like_a_great_proc/
    • seed selection https://www.reddit.com/r/StableDiffusion/comments/x8szj9/tutorial_seed_selection_and_the_impact_on_your/
    • minor parameter parameter difference study (steps, clamp_max, ETA, cutn_batches, etc) https://twitter.com/KyrickYoung/status/1500196286930292742
• Advanced
  • nothing yet
• https://creator.nightcafe.studio/vqgan-clip-keyword-modifier-comparison VQGAN+CLIP Keyword Modifier Comparison We compared 126 keyword modifiers with the same prompt and initial image. These are the results.
  • https://creator.nightcafe.studio/collection/8dMYgKm1eVXG7z9pV23W
• Google released PartiPrompts as a benchmark: https://parti.research.google/ "PartiPrompts (P2) is a rich set of over 1600 prompts in English that we release as part of this work. P2 can be used to measure model capabilities across various categories and challenge aspects."
• Video tutorials
  • Pixel art https://www.youtube.com/watch?v=UvJkQPtr-8s&feature=youtu.be
• Misc
  • StabilityAI CIO perspective https://danieljeffries.substack.com/p/the-turning-point-for-truly-open?sd=pf
  • https://github.com/awesome-stable-diffusion/awesome-stable-diffusion

Tooling

• Prompt Generators:
  • https://huggingface.co/succinctly/text2image-prompt-generator
    • This is a GPT-2 model fine-tuned on the succinctly/midjourney-prompts dataset, which contains 250k text prompts that users issued to the Midjourney text-to-image service over a month period. This prompt generator can be used to auto-complete prompts for any text-to-image model (including the DALL·E family)
  • Prompt Parrot https://colab.research.google.com/drive/1GtyVgVCwnDfRvfsHbeU0AlG-SgQn1p8e?usp=sharing
    • This notebook is designed to train language model on a list of your prompts,generate prompts in your style, and synthesize wonderful surreal images! ✨
    • https://twitter.com/KyrickYoung/status/1563962142633648129
  • https://twitter.com/stuhlmueller/status/1575187860063285248
    • The Interactive Composition Explorer (ICE), a Python library for writing and debugging compositional language model programs https://github.com/oughtinc/ice
  • The Factored Cognition Primer, a tutorial that shows using examples how to write such programs https://primer.ought.org
  • Prompt Explorer
    • https://twitter.com/fabianstelzer/status/1575088140234428416
    • https://docs.google.com/spreadsheets/d/1oi0fwTNuJu5EYM2DIndyk0KeAY8tL6-Qd1BozFb9Zls/edit#gid=1567267935
  • Prompt generator https://www.aiprompt.io/
• Stable Diffusion Interpolation
  • https://colab.research.google.com/drive/1EHZtFjQoRr-bns1It5mTcOVyZzZD9bBc?usp=sharing
  • This notebook generates neat interpolations between two different prompts with Stable Diffusion.
• Easy Diffusion by WASasquatch
  • This super nifty notebook has tons of features, such as image upscaling and processing, interrogation with CLIP, and more! (depth output for 3D Facebook images, or post processing such as Depth of Field.)
  • https://colab.research.google.com/github/WASasquatch/easydiffusion/blob/main/Stability_AI_Easy_Diffusion.ipynb
• Craiyon + Stable Diffusion https://twitter.com/GeeveGeorge/status/1567130529392373761

Misc

• prompt-engine: From Microsoft, NPM utility library for creating and maintaining prompts for LLMs
• Edsynth and DAIN for coherence
• FILM: Frame Interpolation for Large Motion (github)
• Depth Mapping
  • examples: https://twitter.com/TomLikesRobots/status/1566152352117161990
• Art program plugins
  • Krita: https://github.com/nousr/koi
  • GIMP https://80.lv/articles/a-new-stable-diffusion-plug-in-for-gimp-krita/
  • Photoshop: https://old.reddit.com/r/StableDiffusion/comments/wyduk1/show_rstablediffusion_integrating_sd_in_photoshop/
    • download: https://twitter.com/cantrell/status/1574432458501677058
    • https://www.getalpaca.io/
    • demo: https://www.youtube.com/watch?v=t_4Y6SUs1cI and https://twitter.com/cantrell/status/1582086537163919360
    • tutorial https://odysee.com/@MaxChromaColor:2/how-to-install-the-free-stable-diffusion:1
  • Figma: https://twitter.com/RemitNotPaucity/status/1562319004563173376?s=20&t=fPSI5JhLzkuZLFB7fntzoA
  • collage tool https://twitter.com/genekogan/status/1555184488606564353
• Papers
  • 2015: Deep Unsupervised Learning using Nonequilibrium Thermodynamics founding paper of diffusion models
  • Textual Inversion: https://arxiv.org/abs/2208.01618 (impl: https://github.com/rinongal/textual_inversion)
    • Stable Conceptualizer https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/stable_conceptualizer_inference.ipynb
  • 2017: Attention is all you need
  • https://dreambooth.github.io/
    • productized as dreambooth https://twitter.com/psuraj28/status/1575123562435956740
    • https://github.com/JoePenna/Dreambooth-Stable-Diffusion
    • from huggingface diffusers https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/sd_dreambooth_training.ipynb
    • https://twitter.com/rainisto/status/1584881850933456898
      • Commercial offerings
        • https://avatarai.me/
        • https://www.astria.ai/ (formerly https://www.strmr.com/)
        • https://twitter.com/rohanarora_/status/1580413809516511232?s=20&t=XxjfadtkVM8TOvg5EYFCrw
  • very good BLOOM model overview

Communities

• StableDiffusion Discord https://discord.com/invite/stablediffusion
• https://reddit.com/r/stableDiffusion
• Akhaliq Discord: https://discord.gg/nYqfg4gnBt
• Deforum Discord https://discord.gg/upmXXsrwZc
• Lexica Discord https://discord.com/invite/bMHBjJ9wRh
• Midjourney's discord
  • how to use midjourney v4 https://twitter.com/fabianstelzer/status/1588856386540417024?s=20&t=PlgLuGAEEds9HwfegVRrpg
• https://stablehorde.net/

Stable Diffusion

stable diffusion specific notes

Required reading:

• param intuition https://www.reddit.com/r/StableDiffusion/comments/x41n87/how_to_get_images_that_dont_suck_a/
• CLI commands https://www.assemblyai.com/blog/how-to-run-stable-diffusion-locally-to-generate-images/#script-options

SD Distros

• Installer Distros: Programs that bundle Stable Diffusion in an installable program, no separate setup and the least amount of git/technical skill needed, usually bundling one or more UI
  • Diffusion Bee: Diffusion Bee is the easiest way to run Stable Diffusion locally on your M1 Mac. Comes with a one-click installer. No dependencies or technical knowledge needed.
  • https://www.charl-e.com/: Stable Diffusion on your Mac in 1 click. (tweet)
  • https://github.com/cmdr2/stable-diffusion-ui: Easiest 1-click way to install and use Stable Diffusion on your own computer. Provides a browser UI for generating images from text prompts and images. Just enter your text prompt, and see the generated image. (Linux, Windows, no Mac)
  • https://nmkd.itch.io/t2i-gui: A basic (for now) Windows 10/11 64-bit GUI to run Stable Diffusion, a machine learning toolkit to generate images from text, locally on your own hardware. As of right now, this program only works on Nvidia GPUs! AMD GPUs are not supported. In the future this might change.
  • imaginAIry 🤖🧠: Pythonic generation of stable diffusion images with just pip install imaginairy. "just works" on Linux and macOS(M1) (and maybe windows). Memory efficiency improvements, prompt-based editing, face enhancement, upscaling, tiled images, img2img, prompt matrices, prompt variables, BLIP image captions, comes with dockerfile/colab. Has unit tests.
  • Fictiverse/Windows-GUI: A windows interface for stable diffusion
  • https://github.com/razzorblade/stable-diffusion-gui: dormant now.
• Web Distros
  • https://playgroundai.com/create UI for DallE and Stable Diffusion
  • https://www.mage.space/
  • https://dreamlike.art/ has img2img
  • https://inpainter.vercel.app/paint for inpainting
  • https://promptart.labml.ai/feed
  • https://www.strmr.com/ dreambooth tuning for $3
  • https://www.findanything.app browser extension that adds SD predictions alongside Google search
  • https://www.drawanything.app
  • https://creator.nolibox.com/guest open source https://github.com/carefree0910/carefree-creator
• Finetuned Distros
  • Arcane Diffusion a fine-tuned Stable Diffusion model trained on images from the TV Show Arcane.
  • Spider-verse Diffusion rained on movie stills from Sony's Into the Spider-Verse. Use the tokens spiderverse style in your prompts for the effect.
  • Texture inpainting
  • How to finetune your own
    • Naruto version https://lambdalabs.com/blog/how-to-fine-tune-stable-diffusion-naruto-character-edition
    • Pokemon https://lambdalabs.com/blog/how-to-fine-tune-stable-diffusion-how-we-made-the-text-to-pokemon-model-at-lambda
    • https://towardsdatascience.com/how-to-fine-tune-stable-diffusion-using-textual-inversion-b995d7ecc095
• Twitter Bots
  • https://twitter.com/diffusionbot
  • https://twitter.com/m1guelpf/status/1569487042345861121
• Windows "retard guides"
  • https://rentry.org/voldy
  • https://rentry.org/GUItard

SD Major forks

Main Stable Diffusion repo: https://github.com/CompVis/stable-diffusion

• Tensorflow/Keras impl: https://github.com/divamgupta/stable-diffusion-tensorflow
• Diffusers library: https://github.com/huggingface/diffusers

Name/Link	Stars	Description
AUTOMATIC1111	9700	The most well known fork. features: https://github.com/AUTOMATIC1111/stable-diffusion-webui#features launch announcement https://www.reddit.com/r/StableDiffusion/comments/x28a76/stable_diffusion_web_ui/. M1 mac instructions https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Installation-on-Apple-Silicon
Disco Diffusion	5600	A frankensteinian amalgamation of notebooks, models and techniques for the generation of AI Art and Animations.
sd-webui (formerly hlky fork)	5100	A fully-integrated and easy way to work with Stable Diffusion right from a browser window. Long list of UI and SD features (incl textual inversion, alternative samplers, prompt matrix): https://github.com/sd-webui/stable-diffusion-webui#project-features
InvokeAI (formerly lstein fork)	3400	This version of Stable Diffusion features a slick WebGUI, an interactive command-line script that combines text2img and img2img functionality in a "dream bot" style interface, and multiple features and other enhancements. It runs on Windows, Mac and Linux machines, with GPU cards with as little as 4 GB of RAM.
XavierXiao/Dreambooth-Stable-Diffusion	2400	Implementation of Dreambooth (https://arxiv.org/abs/2208.12242) with Stable Diffusion. Dockerized: https://github.com/smy20011/dreambooth-docker
Basujindal: Optimized Stable Diffusion	2100	This repo is a modified version of the Stable Diffusion repo, optimized to use less VRAM than the original by sacrificing inference speed. img2img and txt2img and inpainting under 2.4GB VRAM
stablediffusion-infinity	1900	Outpainting with Stable Diffusion on an infinite canvas. This project mainly works as a proof of concept.
Waifu Diffusion (huggingface, replicate)	1100	stable diffusion finetuned on weeb stuff. "A model trained on danbooru (anime/manga drawing site with also lewds and nsfw on it) over 56k images.Produces FAR BETTER results if you're interested in getting manga and anime stuff out of stable diffusion."
AbdBarho/stable-diffusion-webui-docker	929	Easy Docker setup for Stable Diffusion with both Automatic1111 and hlky UI included. HOWEVER - no mac support yet AbdBarho/stable-diffusion-webui-docker#35
fast-stable-diffusion	753	+25-50% speed increase + memory efficient + DreamBooth
nolibox/carefree-creator	717	An infinite draw board for you to save, review and edit all your creations. Almost EVERY feature about Stable Diffusion (txt2img, img2img, sketch2img, variations, outpainting, circular/tiling textures, sharing, ...). Many useful image editing methods (super resolution, inpainting, ...). Integrations of different Stable Diffusion versions (waifu diffusion, ...). GPU RAM optimizations, which makes it possible to enjoy these features with an NVIDIA GeForce GTX 1080 Ti! It might be fair to consider this as: An AI-powered, open source Figma. A more 'interactable' Hugging Face Space. A place where you can try all the exciting and cutting-edge models, together.
imaginAIry 🤖🧠	639	Pythonic generation of stable diffusion images with just pip install imaginairy. "just works" on Linux and macOS(M1) (and maybe windows). Memory efficiency improvements, prompt-based editing, face enhancement, upscaling, tiled images, img2img, prompt matrices, prompt variables, BLIP image captions, comes with dockerfile/colab. Has unit tests.
neonsecret/stable-diffusion	546	This repo is a modified version of the Stable Diffusion repo, optimized to use less VRAM than the original by sacrificing inference speed. Also I invented the sliced atttention technique, which allows to push the model's abilities even further. It works by automatically determining the slice size from your vram and image size and then allocating it one by one accordingly. You can practically generate any image size, it just depends on the generation speed you are willing to sacrifice.
Deforum Stable Diffusion	347	Animating prompts with stable diffusion. Weighted Prompts, Perspective 2D Flipping, Dynamic Video Masking, Custom MATH expressions, Waifu and Robo Diffusion Models. twitter, changelog. replicate demo: https://replicate.com/deforum/deforum_stable_diffusion
Maple Diffusion	264	Maple Diffusion runs Stable Diffusion models locally on macOS / iOS devices, in Swift, using the MPSGraph framework (not Python). Matt Waller working on CoreML impl
Doggettx/stable-diffusion	137	Allows to use resolutions that require up to 64x more VRAM than possible on the default CompVis build.
Doohickey Diffusion	29	CLIP guidance, perceptual guidance, Perlin initial noise, and other features.

https://github.com/Filarius/stable-diffusion-webui/blob/master/scripts/vid2vid.py with Vid2Vid

SD in Other languages

• Chinese: https://twitter.com/_akhaliq/status/1572580845785083906
• Japanese: https://twitter.com/_akhaliq/status/1571977273489739781
  • https://huggingface.co/blog/japanese-stable-diffusion

Other Lists of Forks

• https://www.reddit.com/r/StableDiffusion/comments/wqaizj/list_of_stable_diffusion_systems/
• https://www.reddit.com/r/StableDiffusion/comments/xcclmf/comment/io6u03s/?utm_source=reddit&utm_medium=web2x&context=3
• https://techgaun.github.io/active-forks/index.html#CompVis/stable-diffusion

Dormant projects, for historical/research interest:

• https://colab.research.google.com/drive/1AfAmwLMd_Vx33O9IwY2TmO9wKZ8ABRRa
• https://colab.research.google.com/drive/1kw3egmSn-KgWsikYvOMjJkVDsPLjEMzl
• bfirsh/stable-diffusion No longer actively maintained byt was the first to work on M1 Macs - blog, tweet, can also look at environment-mac.yaml from https://github.com/fragmede/stable-diffusion/blob/mps_consistent_seed/environment-mac.yaml

Misc SD UI's

UI's that dont come with their own SD distro, just shelling out to one

UI Name/Link	Stars	Self-Description
ahrm/UnstableFusion	815	UnstableFusion is a desktop frontend for Stable Diffusion which combines image generation, inpainting, img2img and other image editing operation into a seamless workflow. https://www.youtube.com/watch?v=XLOhizAnSfQ&t=1s
breadthe/sd-buddy	165	Companion desktop app for the self-hosted M1 Mac version of Stable Diffusion, with Svelte and Tauri
leszekhanusz/diffusion-ui	65	This is a web interface frontend for the generation of images using diffusion models. The goal is to provide an interface to online and offline backends doing image generation and inpainting like Stable Diffusion.
GenerationQ	21	GenerationQ (for "image generation queue") is a cross-platform desktop application (screens below) designed to provide a general purpose GUI for generating images via text2img and img2img models. Its primary target is Stable Diffusion but since there is such a variety of forked programs with their own particularities, the UI for configuring image generation tasks is designed to be generic enough to accommodate just about any script (even non-SD models).

SD Prompt galleries and search engines

• 🌟 Lexica: Content-based search powered by OpenAI's CLIP model. Seed, CFG, Dimensions.
• PromptFlow: Search engine that allows for on-demand generation of new results. Search 10M+ of AI art and prompts generated by DALL·E 2, Midjourney, Stable Diffusion
• https://synesthetic.ai/ SD focused
• https://visualise.ai/ Create and share image prompts. DALL-E, Midjourney, Stable Diffusion
• https://nyx.gallery/
• OpenArt: Content-based search powered by OpenAI's CLIP model. Favorites.
• PromptHero: Random wall. Seed, CFG, Dimensions, Steps. Favorites.
• Libraire: Seed, CFG, Dimensions, Steps.
• Krea: modifiers focused UI. Favorites. Gives prompt suggestions and allows to create prompts over Stable diffusion, Waifu Diffusion and Disco diffusion. Really quick and useful
• Avyn: Search engine and generator.
• Pinegraph: discover, create and edit with Stable/Disco/Waifu diffusion models.
• Phraser: text and image search.
• https://arthub.ai/
• https://pagebrain.ai/promptsearch/
• https://avyn.com/
• https://dallery.gallery/
• The Ai Art: gallery for modifiers.
• urania.ai: Top 500 Artists gallery, sorted by image count. With modifiers/styles.
• Generrated: DALL•E 2 table gallery sorted by visual arts media.
• Artist Studies by @remi_durant: gallery and Search.
• CLIP Ranked Artists: gallery sorted by weight/strength.
• https://promptbase.com/ Selling prompts that produce desirable results
• https://publicprompts.art/ very basic/limited but some good prompts. promptbase competitor

SD Visual search

• Lexica: enter an image URL in the search bar. Or next to q=. Example
• Phraser: image icon at the right.
• same.energy
• Yandex, Bing, Google, Tineye, iqdb: reverse and similar image search engines.
• Pinterest
• dessant/search-by-image: Open-source browser extension for reverse image search.

SD Prompt generators

• promptoMANIA: Visual modifiers. Great selection. With weight setting.
• Phase.art: Visual modifiers. SD Generator and share.
• Phraser: Visual modifiers.
• AI Text Prompt Generator
• Dynamic Prompt generator
• succinctly/text2image: GPT-2 Midjourney trained text completion.
• Prompt Parrot colab: Train and generate prompts.
• cmdr2: 1-click SD installation with image modifiers selection.

Img2prompt - Reverse Prompt Engineering

• img2prompt Replicate by methexis-inc: Optimized for SD (clip ViT-L/14).
• CLIP Interrogator by @pharmapsychotic: select ViTL14 CLIP model.
  • https://huggingface.co/spaces/pharma/sd-prism Sends an image in to CLIP Interrogator to generate a text prompt which is then run through Stable Diffusion to generate new forms of the original!
• CLIP Artist Evaluator colab
• BLIP

Explore Artists, styles, and modifiers

See https://github.com/sw-yx/prompt-eng/blob/main/PROMPTS.md for more details and notes

• Artist Style Studies & Modifier Studies by parrot zone: Gallery, Style, Spreadsheet
• Clip retrieval: search laion-5b dataset.
• Datasette: image search; image-count sort by artist, celebrities, characters, domain
• Visual arts: media list, related; Artists list by genre, medium; Portal

SD Prompt Tools directories and guides

• Useful Prompt Engineering tools and resources https://np.reddit.com/r/StableDiffusion/comments/xcrm4d/useful_prompt_engineering_tools_and_resources/
• Tools and Resources for AI Art by pharmapsychotic
• Akashic Records
• Awesome Stable-Diffusion

SD Tooling

• AI Dreamer iOS/macOS app https://apps.apple.com/us/app/ai-dreamer/id1608856807
• SD's DreamStudio https://beta.dreamstudio.ai/dream
• Stable Worlds: colab for 3d stitched worlds via StableDiffusion https://twitter.com/NaxAlpha/status/1578685845099290624
• Midjourney + SD: https://twitter.com/EMostaque/status/1561917541743841280
• Nightcafe Studio
• misc
  • (super super raw dont try yet) https://github.com/breadthe/sd-buddy

How SD Works - Internals and Studies

• How SD works
  • https://huggingface.co/blog/stable_diffusion
  • https://github.com/ekagra-ranjan/huggingface-blog/blob/main/stable_diffusion.md
    • tinygrad impl https://github.com/geohot/tinygrad/blob/master/examples/stable_diffusion.py
  • https://colab.research.google.com/drive/1dlgggNa5Mz8sEAGU0wFCHhGLFooW_pf1?usp=sharing
  • https://twitter.com/johnowhitaker/status/1565710033463156739
  • https://twitter.com/ai__pub/status/1561362542487695360
  • https://twitter.com/JayAlammar/status/1572297768693006337
    • https://jalammar.github.io/illustrated-stable-diffusion/
  • https://colab.research.google.com/drive/1dlgggNa5Mz8sEAGU0wFCHhGLFooW_pf1?usp=sharing
  • inside https://keras.io/guides/keras_cv/generate_images_with_stable_diffusion/#wait-how-does-this-even-work
• Samplers studies
  • https://twitter.com/iScienceLuvr/status/1564847717066559488
• Disco Diffusion Illustrated Settings
• Understanding MidJourney (and SD) through teapots.
• A Traveler’s Guide to the Latent Space
• Exploring 12 Million of the 2.3 Billion Images Used to Train Stable Diffusion’s Image Generator
  • explore: https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images
  • search: https://haveibeentrained.com/ (tweet)

SD Results

Img2Img

• A black and white photo of a young woman, studio lighting, realistic, Ilford HP5 400
  • https://twitter.com/TomLikesRobots/status/1566027217892671488

SD 1.4 vs 1.5 comparisons

• https://twitter.com/TomLikesRobots/status/1583836870445670401

Hardware requirements

• https://news.ycombinator.com/item?id=32642255#32646761
  • For something like this, you ideally would want a powerful GPU with 12-24gb VRAM.
  • A $500 RTX 3070 with 8GB of VRAM can generate 512x512 images with 50 steps in 7 seconds.
• https://huggingface.co/blog/stable_diffusion_jax uper fast inference on Google TPUs, such as those available in Colab, Kaggle or Google Cloud Platform - 8 images in 8 seconds
• Intel CPUs: https://github.com/bes-dev/stable_diffusion.openvino

SD vs DallE vs MJ

DallE banned so SD https://twitter.com/almost_digital/status/1556216820788609025?s=20&t=GCU5prherJvKebRrv9urdw

Misc

• Imagen
  • https://www.youtube.com/watch?v=R_f-v6prMqI
• Whisper
  • https://huggingface.co/spaces/sensahin/YouWhisper YouWhisper converts Youtube videos to text using openai/whisper.
  • https://twitter.com/jeffistyping/status/1573145140205846528 youtube whipserer
  • multilingual subtitles https://twitter.com/1littlecoder/status/1573030143848722433
  • video subtitles https://twitter.com/m1guelpf/status/1574929980207034375
  • you can join whisper to stable diffusion for reasons https://twitter.com/fffiloni/status/1573733520765247488/photo/1
  • known problems https://twitter.com/lunixbochs/status/1574848899897884672 (edge case with catastrophic failures)
• textually guided audio https://twitter.com/FelixKreuk/status/1575846953333579776
• Codegen
  • CodegeeX https://twitter.com/thukeg/status/1572218413694726144
  • https://github.com/salesforce/CodeGen https://joel.tools/codegen/
• pdf to structured data https://www.impira.com/blog/hey-machine-whats-my-invoice-total
• text to Human Motion diffusion https://twitter.com/GuyTvt/status/1577947409551851520
  • abs: https://arxiv.org/abs/2209.14916
  • project page: https://guytevet.github.io/mdm-page/

Quotes, Reality & Demotivation

• Narrow, tedium domain usecases https://twitter.com/WillManidis/status/1584900092615528448?s=20&t=aV0Np-2Sx-zq-TQNn2y5AQ
• antihype https://twitter.com/alexandr_wang/status/1573302977418387457
• New Google
  • https://twitter.com/alexandr_wang/status/1585022891594510336
• New Powerpoint
• via emad
• Appending prompts by default in UI
• DALLE: https://twitter.com/levelsio/status/1588588688115912705?s=20&t=0ojpGmH9k6MiEDyVG2I6gg

Infrastructure

• bananadev cold boot problem https://twitter.com/erikdunteman/status/1584992679330426880?s=20&t=eUFvLqU_v10NTu65H8QMbg