/proper-pixel-art

Converts pixel-art-style images such as those from generative models or low-quality sprites to true resolution usable assets

Primary LanguagePythonMIT LicenseMIT

Proper Pixel Art

Summary

  • Converts noisy, high resolution pixel-art-style images such as those produced by generative models to true pixel resolution assets.

  • Clean screenshots or low-quality web uploads of sprites.

Installation

Clone the Repository:

git clone git@github.com:KennethJAllen/proper-pixel-art.git
cd proper-pixel-art

Create Virtual Environment

  • Install uv if not already installed.
  • Sync environments
    • uv sync

Usage

First, obtain a source pixel-art-style image (e.g. a pixel-art-style image generated by GPT-4o or a screenshot of pixel-art).

CLI

uv run ppa -i <input_path> -o <output_path> -c <num_colors> -p <pixel_size> [-t]
# or directly using uvx
uvx --from https://github.com/KennethJAllen/proper-pixel-art.git ppa -i <input_path> -o <output_path> -c <num_colors> -p <pixel_size> [-t]

Flags

Flag Description
-i, --input <path> Source image file in pixel-art-style
-o, --output <path> Output directory or file path for result
-c, --colors <int> Number of colors for output. May need to try a few different values (default 16)
-p, --pixel-size <int> Size of each “pixel” in the output (default: 1)
-t, --transparent Output with transparent background (default: off)

Example

uv run ppa -i assets/blob/blob.png -o . -c 16 -p 20 -t

Python

Simple Example

from PIL import Image
from proper_pixel_art.pixelate import pixelate

image = Image.open('input.png')
result = pixelate(image, num_colors=16)
result.save('output.png')

Parameters

  • image : PIL.Image.Image

    • A PIL image to pixelate.

  • num_colors : int

    • The number of colors in result.
    • May need to try a few values if the colors don't look right.
    • 8, 16, 32, or 64 typically works.

  • pixel_size : int

    • Upscale result after algorithm is complete if not None.

  • upsample_factor : int

    • Upscale initial image. This may help detect lines.

  • transparent_background : bool

    • If True, floos fills each corner of the result with transparent alpha.

  • intermediate_dir : Path | None

    • Directory to save images visualizing intermediate steps of algorithm. Useful for development.

Returns

A PIL image with

Examples

The algorithm is robust. It performs well for images that are already approximately alligned to a grid.

Here are a few examples. A mesh is computed, where each cell corresponds to one pixel.

Bat

  • Generated by GPT-4o.

Noisy, High Resolution
Mesh
True Pixel Resolution

Ash

  • Screenshot from Google images of Pokemon asset.

Noisy, High Resolution
Mesh
True Pixel Resolution

Demon

  • Original image generated by GPT-4o.

Noisy, High Resolution
Mesh
True Pixel Resolution

Pumpkin

  • Screenshot from Google Images of Stardew Valley asset. This is an adversarial exmaple as the source image is both low quality and the object is round.

Noisy, High Resolution
Mesh
True Pixel Resolution

Real Images To Pixel Art

  • This tool can also be used to convert real images to pixel art by first requesting a pixelated version of the original image from GPT-4o, then using the tool to get the true pixel-resolution image.

  • Consider this image of a mountain

Original mountain

  • Here are the results of first requesting a pixalated version of the mountain, then using the tool to get a true resolution pixel art version.

Noisy, High Resolution
Mesh
True Pixel Resolution

Challenges

The result of pixel-art style images from LLMs are noisy, high resolution images with a non-uniform grid and random artifacts. Due to these issues, standard downsampling techniques do not work. How can we recover the pixel art with "true" resolution and colors?

The current approach to turning pixel art into useable assets for games are either

  1. Use naive downsampling which does not give a result that is faithful to the original image.
  2. Manually re-create the image in the approperiate resolution pixel by pixel.

Algorithm

  • The main algorithm solves these challenges. Here is a high level overview. We will apply it step by step on this example image of blob pixel art that was generated from GPT-4o.

blob

  • Note that this image is high resolution and noisy.

The blob is noisy.

  1. Trim the edges of the image and zero out pixels with more than 50% alpha.

    • This is to work around some issues with models such as GPT-4o not giving a perfectly transparent background.

  2. Upscale by a factor of 2 using nearest neighbor.

    • This can help identify the correct pixel mesh.
    • It is possible that a similar result could be achived by tuning the parameters in the following steps. Further investigation is required.

  3. Find edges of the pixel art using Canny edge detection.

blob edges

  1. Close small gaps in edges with a morphological closing.

blob closed edges

  1. Take the probabalistic Hough transform to get the coordinates of lines in the detected edges. Only keep lines that are close to vertical or horizontal giving some grid coordinates. Cluster lines that are closeby together.

blob lines

  1. Find the grid spacing by filtering outliers and taking the median of the spacings, then complete the mesh.

blob mesh

  1. Quantize the original image to a small number of colors.

    • Note: The result is sensitive to the number of colors chosen.
    • The parameter is not difficult to tune, but the script may need to be re-run if the colors don't look right.
    • 8, 16, 32, or 64 typically works.

  2. In each cell specified by the mesh, choose the most common color in the cell as the color for the pixel. Recreate the original image with one pixel per cell.

    • Result upscaled by a factor of $20 \times$ using nearest neighbor.

blob upscaled