Not too long ago, Turquoit#3987 from the Stable Diffusion Discord server shared a text file containing links to every image generated by the bot from August 6th (its inception) to August 20 (when they closed the bot off). This repository contains some code to convert that file to a SQLite database for easy querying. A prebuilt ImageLinks.db file is provided here (3.4gb)
Structure of the database is two tables:
prompts, with a primary keyidand a stringpromptdatawhich has apromptIdand aurl, among extra metadata columns
The converter is written using Bun and it's bun:sqlite built in. I also wrote it so it can use nodejs and better-sqlite. They both take about took 45 seconds to convert on my desktop.
To run, you need an extracted copy of ImageLinksFinal.zip (2.5gb). Then you can run bun convert.mjs or node convert.mjs.
Maybe in the future, a web app / public web api will be made to query the database without storing a local copy. I could do this but I wouldn't be able to fund a server to hold the database (ideally use postgres instead of sqlite).
- Update db download with
ImageLinksFinal.txt, which has "+622,709 prompts / +380,355 unique prompts compared to the last scrape - Program performance imrpoved by ~3x
- Add indexes to the SQLite database, decreasing query times of "all images for a prompt id" from 8000ms to 3ms on my machine
- Initial Publication