/aitools_client

Seth's AI Tools: A Unity based Stable Diffusion front-end for AUTOMATIC1111's WebUI focused on gamedev

Primary LanguageC#OtherNOASSERTION

Seth's AI Tools: A Unity based front-end for Stable Diffusion WebUI (and other AI stuff)

License: BSD style attribution, see LICENSE.md

Download the latest AI Tools Client (Windows, 56 MB)

To use this, you'll need at least one Stable Diffusion WebUI server running somewhere (it can run on Windows or linux, the same machine as the client is ok) This client supports either of the following servers:

Seth's AI Tools Server (Same as below but with a few extra features, including background removal)

or

AUTOMATIC1111's Stable Diffusion WebUI (must run with the --api parm)

or

For the AI Guide feature, you can use your OpenAI api key to use GPT4 and Dalle 3 and not need a local server at all

Features

  • It's not a web app, it's a native locally run Windows .exe
  • Live update integration (image and masking) with Photoshop or other image editors
  • text to image, img2img, image interrogation, face fixing, upscaling, tiled texture generation with preview, automasking of subject, background removal
  • Drag and drop images in as well as paste images from the windows clipboard
  • Built-in image processing like cropping and resizing
  • Pan/zoom with thousands of images on the screen
  • Mask painting with controllable brush size
  • Option to automatically save all generations, with accompanying .txt file that includes all settings
  • Supports pix2pix and ControlNet (For ControlNet, the server requires Mikubill's sd-webui-controlnet extension) and its models to be installed, otherwise the option is grayed out
  • Can utilize multiple servers (three video cards on one machine? Run three servers!) allowing seamless use of all GPUs for ultra fast generation and a single click to change the active model
  • Neat workflow that allows evolving images with loopback while live-selecting the best alternatives to shape the image in real-time
  • Open source, uses the Unity game engine and C# to do stuff with AI art
  • Privacy respected - does not phone home or collect any statistics, purely local usage. (it does check a single file on github.com to check for newer versions, but that's it)
  • Includes "experiments", little built-in games and apps designed to test using AI/SD for specific things: CrazyCam is a realtime webcam filter with 30+ presets, Shooting Gallery tests realtime craetion of sprites during a game, etc
  • AI Guide feature harnesses the power of GPT-4 or open source LLMs to create motivational posters, illustrated stories or whatever, with presets and web-slideshower viewer. Comes with presets like Pixel Art Gaming Lies and Random Story That Teaches Japanese
  • Can optionally use Dalle 3 for rendering with AI Guide

Current version: V0.78 (released Dec 9th 2023)

Recent changes:

  • Added way to set the ChatGPT model via config.txt: example: set_openai_gpt4_model|gpt-4
  • Added "Render with Dalle 3" option to pic menu, works with the same openai key entered for gpt. Once rendered, you can do normal operations like select the foreground, upscale, etc
  • AI Guide feature now has a renderer dropdown which can set Dalle 3 for rendering instead of SD WebUI
  • Added an optional Refiner setting
  • Experimental: Added a "Turbo mode" checkbox, if on, it does non-stop realtime rendering to the latest pic, this is so you can do "type and see it in realtime" with faster models. it also works with batch things like img2img, but the "turbo" models I tested need 6+ steps to get this to work and don't look great
  • Errors files are written out in release mode too, no reason to have those be debug only
  • BUGFIX: AI Guide button is aligned correctly at all screen sizes
  • AI Guide's "Text generation web UI" support fixed to work with its latest version
  • Added new AI Guide preset called "Random basic story for lzlv_70b (TexGen WebUI)", amazingly it works nearly as well as chatgpt, go open source!
  • Added a "text to prepend" option to AI Guide, helped for the above setting as I feed it the "text:" part and it's missing from the generation
  • Note, AI Guide options are a bit outdated, they should all probably use non-json styles and then prompts would probably work with both TextGen WebUI and GPT4 but I'm too lazy to organize it better for now

NOTE: For pix2pix stuff, you need to add the 7 gb model to your models/Stable-diffusion folder

For ControlNet options, you need to install Mikubill's sd-webui-controlnet extension and at least one of its models. You should make sure it's working from the webUI interface on the server first. You must run the server with the parm "--xformers", this is required by the extension. Also, if you see "Protocol errors" anywhere, remove the Dreambooth extension, lately it's been causing API issues. (as of March 2nd/2023 at least)

You only need to download the zip and run the .exe to use this, However, the source might be useful to generate a build for other platforms, fork or steal pieces to use for yourself. Go ahead!

Media (outdated videos of the app)

Setup

If using AUTOMATIC1111's Stable Diffusion WebUI, make sure it has been started with the --api parm. (additionally, with the --listen parm if it isn't on the local machine)

On Windows, an easy way to do that is to edit webui-user.bat and add them after the "set COMMANDLINE_ARGS=" part. Start the server by double clicking webui-user.bat.

Next run aitools_client.exe. Click on the "Configuration" button and a text editor will open with the default settings:

#add as many add_server commands as you want, just replace the localhost:7860 part with the
#server name/ip and port.  You can control any number of servers at the same time.

#You need at least one server running to work. It can be either an automatic1111 Stable Diffusion WebUI server or
#a Seth's AI Tools server which supports a few more features.  It will autodetect which kind it is.

add_server|http://localhost:7860

#Set the below path and .exe to an image editor to use the Edit option. Changed files will auto
#update in here.

set_image_editor|C:\Program Files\Adobe\Adobe Photoshop 2023\Photoshop.exe

#set_default_sampler|DDIM
#set_default_steps|50

#To generate text with the AI Guide features, you need to set your OpenAI GPT4 key and/or a Text Generation web IO API url (presumably your own local server).

set_openai_gpt4_key|<key goes here>|
set_openai_gpt4_endpoint|https://api.openai.com/v1/chat/completions|
set_openai_gpt4_model|gpt-4

set_texgen_webui_address|localhost:5000|

If your Stable Diffusion WebUI server isn't running locally or at port 7860, change the http://localhost:7860 part to where it is. Add multiple add_server commands for multiple servers.

NOTE: Using automatic1111, on the server side, you will see a scary error saying "RuntimeError: File at path D:\pro\stable-diffusion-webui\aitools\get_info.json does not exist.", this is ok, the app checks for the file to see what kind of server it is once at the start. It doesn't break anything.

Building from source

  • Requires Unity 2022.3.15+
  • Open the scene "Main" and click play to run

Credits and links