This project is a Visual Studio Code extension that converts screenshots of websites into React code. The extension uses the Open AI API for image recognition & code generation.
It also uses Color Thief to extract the color palette from the screenshot and apply it to the generated code using a simple color matching algorithm.
Twitter card:
Becomes this:
Github comment:
Becomes this:
This is a really basic implementation of the idea. In the future, I'd like to extend it to segment the UI into different regions for more accurate color picking.
I'd also like to use more specific models for identifying fonts. I think borders radius's could use a more procedural technique.
The color replacement is also just using a simple regex, where as if I could step through the AST I'd have a lot more flexibility for how the code should be updated.
A question may come up around why this is a vscode extension & not just a stand alone website. I think a tool like this could lean into your existing settings for stylistic preferences such as language, framework, tabs / spaces, semicolons, etc. Being a part of the core tool as well would reduce the friction in code generation and could allow for generation of code in place in the editor within an existing project.
I'm also thinking a bit about the UX patterns for how things like color palettes, fonts, etc are chosen. Right now, it's automatic, but it would be nice to show on the image where the colors are being picked from, with the ability to drag them around and adjust it (as well as add / remove colors that may be reducing the quality of the results).
I think there's room to explore how these styles could be populated from existing styles in the codebase as well (e.g. lets get the structure from the tweet but apply my own fonts & colors, etc).