Frontend Diffusion: Exploring Intent-Based User Interfaces through Sketching to Coding Task Transition

Paper • Website • Demo Video • Twitter/X

Frontend Diffusion is an end-to-end LLM-powered tool that generates high-quality websites from user sketches. Users can input a rough sketch on the canvas, which the system then transforms into a high-quality website page. You could use it to generate websites from sketch.

The generated websites, as illustrated below, exhibit generally satisfactory visual appearances. These include contextually appropriate textual content, imagery, color schemes, layouts, and functionalities.

Screenshots of generated websites spanning commercial and academic domains:

This tool is used by other open source projects such as GPU Bartender.

How to Use

Draw your sketch in the canvas panel using the toolbar on the left.
Provide a brief website description in the input area at the top right corner.
Click "Generate" to process your input.
View the generated UI in the preview pane on the right.
Access and copy the HTML code by toggling to the "Code Panel" and using the "Copy" button.

Requirements

Install Node.js (through Homebrew or download from http://nodejs.org/). If you want to install through Homebrew, you need to download Homebrew first.
Install npm (through https://www.npmjs.com/get-npm).

Open the terminal on your computer or use the terminal provided by your integrated development environment (e.g., Visual Studio Code). Ensure that the terminal is opened in the project folder directory.

API Key Requirements

Local Deployment Instructions

Navigate to the .env file in your project directory.
Insert your API key information using the following format (you could get ANTHROPIC API Key from : anthropic.com):

ANTHROPIC_API_KEY=YOUR_ANTHROPIC_API_KEY

Save the .env file after adding your API keys.

Note: Keep your API keys confidential and never share them publicly.

Module Requirements

Installation

Before running the program, follow these steps:

Install dependencies:

npm install

If you encounter errors like "Cannot find module '@module_name'", install the specific module:

For example :

npm install @anthropic-ai/sdk
npm install dotenv

(Add any other specific modules as needed)

Running the Application

Navigate to the /backend directory (cd backend), then use the command:

`node server.js`

Navigate to the /frontend directory (cd frontend), then use the command:

`npx live-server`

You should now see the application in your browser.

Customization

To edit the page, modify the frontend/index.html file.

Citation

If you use this project in your research, please cite our paper:

@article{zhang2024frontend,
  title={Frontend Diffusion: Exploring Intent-Based User Interfaces through Abstract-to-Detailed Task Transitions},
  author={Zhang, Qinshi and Hendra, Latisha Besariani and Chi, Mohan and Ding, Zijian},
  journal={arXiv preprint arXiv:2408.00778},
  year={2024}
}

Carolzhangzz/frontendiffusion