This project aims to help with the creation and management of LoRa training datasets. Scroll down to the bottom of the page for a feature overview.
This is still in beta - please report any bugs you find Pull requests are welcome. Currently everything is just cobbled together
Roadmap:
- Refractor tag management
- Add single-image tag overrides [mostly done]
- Add superresolution/upscaling for small images
Known issues:
- No files/folders are ever deleted, leading to clutter/orphaned images
- Deleting an enite folder can break the step
- When an image is open and you save the dataset, it will throw an error and half of the files are left where they are.
(optional) create a venv first:
python -m venv venv
venv\Scripts\activate
install the requirements:
pip install -r requirements.txt
start either by running start.bat
or manually using:
python webserver.py
(see python webserver.py --help
for launch arguments)
Access the webui on the following URL: http://127.0.0.1:8080/
If tagging/cropping/etc is too slow, try run pip install onnxruntime-gpu
, but keep in mind that this will use some of your VRAM.
Running this script is recommended to get all features of the webui.
using start.bat already downloads all dependencies by default
It will gives you the option to download the following files:
danbooru-tags.json
andgelbooru-tags.json
from github gist or catbox.moe.- You also have the option to scrape the tags from the site directly.
cropper.js
andcropper.css
from Cloudflare/cdnjs.
Clear your browser cache between updates. It tends to leave the old scripts/css loaded
The folders created are meant to be used as follows:
0 - raw
- raw images from the internet / screenshots1 - cropped
- cropped images (1:1 aspect ratio)2 - sorted
- images grouped by quality / topic / etc3 - tagged
-.txt
or.json
files containing autotagger output4 - fixed
- pruned tags in.txt
format.5 - out
- scaled down images and pruned tags - point your training script heredatasets
- all your datasets are saved here
Some of these images/videos might be outdated. There's also in UI tooltips. If something breaks just open an issue here on Github.
- Save / load datasets you're working on
- Avoid having to change training folder, just point your training script at the
5 - out
folder and load the right dataset - Write notes for yourself
- Crop images in your browser
- Edit already cropped images
- Duplicate image - crop two separate parts
- Quickly set the cropped area, copy it from the previous image
- Keyboard shortcuts
ui_crop.mp4
- This is a good "baseline" to work from
- You can go back and quickly fix what it misses
- It can detect multiple subjects.
autocrop.mp4
- add categories
- quickly sort multiple images, captcha style
- hit detection can be janky
ui_sort.mp4
- Don't want to do it manually? Set the tags and sort automatically.
ui_sorter.mp4
- A bit slow but does not take any vram, doesn't influence training.
- The output isn't realtime, I think it can do about 1 image/sec on my 11gen i5
ui_tagger.mp4
- Prune useless tags from the autotagger
- Normalize tags
- Quickly blackist/whitelist tags
- Replace tags on all images
- Edit rules, test the effects
- Scale images to required training resolution