sherry0429/magic-copy

Magic Copy is a Chrome extension that uses Meta's Segment Anything Model to extract a foreground object from an image and copy it to the clipboard.

Magic Copy

Magic Copy is a Chrome extension that uses Meta's Segment Anything Model to extract a foreground object from an image and copy it to the clipboard.

Untitled.video.-.Made.with.Clipchamp.1.mp4

Installation

(This might not be available yet, as the extension is still in review. If you would like to be notified when they do, subscribe to the Chrome or Firefox issues.)

Alternatively, the extension can be installed manually:

1. Download the latest magic-copy.zip from releases.
2. Extract the ZIP file.
3. In Chrome, go to chrome://extensions/, enable "Developer mode", and click "Load unpacked".
4. Select the folder where the extension was extracted.

Implementation

This extension uses the same procedure as the Segment Anything Model demo to extract a foreground object from an image. The only difference is that the extracted object is copied to the clipboard instead of being displayed on the page.

Building

Manually

Build the extension with npm and then run the included ./buildcrx.sh script to generate the crx file:

npm ci
npm run build
./buildcrx.sh -d dist

Docker

A Dockerfile is provided to cleanly build the crx file. To build the extension, run:

docker build --output out .

Self-hosting

The Meta Segment Anything Model requires running the vision transformer on a server to generate the image's embeddings. Magic Copy uses the same service that their demo uses, however some people may not want to send their images to a third party.

The server-example directory contains a simple example of how to self-host the vision transformer service. It is not meant to be used in production, but rather as a proof of concept to document the input/output format of the service.

In particular, Magic Copy (and the SAM demo) expect a POST endpoint that accepts an image file and returns a JSON array of length 1 with the embedding of shape (1, 256, 64, 64) as a base64 encoded string. See the code for specific details on how to perform this encoding to be compatible with the demo.

If you are looking to quickly get the service running, you can use the provided Dockerfile to build a container and run it. The container will expose port 8000 and will serve the service at the / endpoint.

docker build -t segment-anything .
docker run --gpus all -p 8000:8000 segment-anything

In the Magic Copy chrome extension, you can then change the endpoint to http://localhost:8000/.