Embedding image and sentence into fixed-length vectors via CLIP
CLIP-as-service is a low-latency high-scalability embedding service for images and texts. It can be easily integrated as a microservice into neural search solutions.
⚡ Fast: Serve CLIP models with ONNX runtime and PyTorch JIT with 800QPS[*]. Non-blocking duplex streaming on requests and responses, designed for large data and long-running tasks.
🫐 Elastic: Horizontally scale up and down multiple CLIP models on single GPU, with automatic load balancing.
🐥 Easy-to-use: No learning curve, minimalist design on client and server. Intuitive and consistent API for image and sentence embedding.
👒 Modern: Async client support. Easily switch between gRPC, HTTP, Websocket protocols with TLS and compressions.
🍱 Integration: Smoothly integrated with neural search ecosystem including Jina and DocArray. Build cross-modal and multi-modal solution in no time.
[*] with default config (single replica, PyTorch no JIT) on GeForce RTX 3090.
CLIP-as-service consists of two Python packages clip-server and clip-client that can be installed independently. Both require Python 3.7+.
pip install clip-serverTo run CLIP model via ONNX (default is via PyTorch):
pip install "clip-server[onnx]"pip install clip-clientYou can run a simple connectivity check after install.
| C/S | Command | Expect output |
|---|---|---|
| Server |
python -m clip_server |
|
| Client |
from clip_client import Client
c = Client('grpc://0.0.0.0:23456')
c.profile() |
|
You can change 0.0.0.0 to the intranet or public IP address to test the connectivity over private and public network. If you encounter some errors, please find the solution here.
- Start the server:
python -m clip_server. Remember its address and port. - Create a client:
from clip_client import Client c = Client('grpc://87.191.159.105:51000')
- To get sentence embedding:
r = c.encode(['First do it', 'then do it right', 'then do it better']) print(r.shape) # [3, 512]
- To get image embedding:
r = c.encode(['apple.png', # local image 'https://docarray.jina.ai/_static/favicon.png', # remote image 'data:image/gif;base64,R0lGODlhEAAQAMQAAORHHOVSKudfOulrSOp3WOyDZu6QdvCchPGolfO0o/XBs/fNwfjZ0frl3/zy7////wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACH5BAkAABAALAAAAAAQABAAAAVVICSOZGlCQAosJ6mu7fiyZeKqNKToQGDsM8hBADgUXoGAiqhSvp5QAnQKGIgUhwFUYLCVDFCrKUE1lBavAViFIDlTImbKC5Gm2hB0SlBCBMQiB0UjIQA7']) # in image URI print(r.shape) # [3, 512]
More comprehensive server & client configs can be found in the docs.
Let's build a text-to-image search using CLIP-as-service. Namely, user input a sentence and the program returns the matched images. We will use Totally Looks Like dataset and DocArray package. Note that DocArray is included within clip-client as an upstream dependency, so you don't need to install it separately.
First we load images. You can simply pull it from Jina Cloud:
from docarray import DocumentArray
da = DocumentArray.pull('ttl-original', show_progress=True, local_cache=True)or download TTL dataset, unzip, load manually
Alternatively, you can go to Totally Looks Like official website, unzip and load images as follows:
from docarray import DocumentArray
da = DocumentArray.from_files(['left/*.jpg', 'right/*.jpg'])The dataset contains 12,032 images, hence it may take half minute to pull. Once done, you can visualize it and get the first taste of those images.
da.plot_image_sprites()
Start the server with python -m clip_server. Say it is at 87.191.159.105:51000 with GRPC protocol (you will get this information after running the server).
Create a Python client script:
from clip_client import Client
c = Client(server='grpc://87.191.159.105:51000')
da = c.encode(da, show_progress=True)Depending on your GPU and client-server network, it could take a while to embed 12K images. In my case, it takes ~2 minute.
Download the pre-encoded dataset
For people who are impatient or lack of GPU, waiting can be a hell. In this case, you can simply pull our pre-encoded image dataset.
from docarray import DocumentArray
da = DocumentArray.pull('ttl-embedding', show_progress=True, local_cache=True)Let's build a simple prompt to allow user to type sentence:
while True:
vec = c.encode([input('sentence> ')])
r = da.find(query=vec, limit=9)
r.plot_image_sprites()Now you can input arbitrary English sentences and view the top-9 matched images. Search is fast and instinct. Let's have some fun:
| "a happy potato" | "a super evil AI" | "a guy enjoying his burger" |
|---|---|---|
|
|
|
|
| "professor cat is very serious" | "an ego engineer lives with parent" | "there will be no tomorrow so lets eat unhealthy" |
|---|---|---|
|
|
|
|
Let's save the embedding result for our next example:
da.save_binary('ttl-image')We can also switch the input and output of the last program to achieve image-to-text search. Precisely, given a query image find the sentence that best describes the image.
Let's use all sentences from the book "Pride and Prejudice".
from docarray import Document, DocumentArray
d = Document(uri='https://www.gutenberg.org/files/1342/1342-0.txt').load_uri_to_text()
da = DocumentArray(
Document(text=s.strip()) for s in d.text.replace('\r\n', '').split('.') if s.strip()
)Let's look at what we got:
da.summary() Documents Summary
Length 6403
Homogenous Documents True
Common Attributes ('id', 'text')
Attributes Summary
Attribute Data type #Unique values Has empty value
──────────────────────────────────────────────────────────
id ('str',) 6403 False
text ('str',) 6030 False
Now encode these 6403 sentences, it may take 10s or less depending on your GPU and network:
from clip_client import Client
c = Client('grpc://87.191.159.105:51000')
r = c.encode(da, show_progress=True)Download the pre-encoded dataset
Again, for people who are impatient or lack of GPU, we have prepared a pre-encoded text dataset.
from docarray import DocumentArray
da = DocumentArray.pull('ttl-textual', show_progress=True, local_cache=True)Let's load our previously stored image embedding; randomly sample image Document from it, then find top-1 nearest neighbour of each.
from docarray import DocumentArray
img_da = DocumentArray.load_binary('ttl-image')
for d in img_da.sample(10):
print(da.find(d.embedding, limit=1)[0].text)Fun time! Note, unlike the previous example, here the input is an image, the sentence is the output. All sentences come from the book "Pride and Prejudice".
|
|
|
|
|
|
| Besides, there was truth in his looks | Gardiner smiled | what’s his name | By tea time, however, the dose had been enough, and Mr | You do not look well |
|
|
|
|
|
|
| “A gamester!” she cried | If you mention my name at the Bell, you will be attended to | Never mind Miss Lizzy’s hair | Elizabeth will soon be the wife of Mr | I saw them the night before last |
Intrigued? That's only scratching the surface of what CLIP-as-service is capable of. Read our docs to learn more.
- Use Discussions to talk about your use cases, questions, and support queries.
- Join our Slack community and chat with other community members about ideas.
- Join our Engineering All Hands meet-up to discuss your use case and learn Jina's new features.
- When? The second Tuesday of every month
- Where? Zoom (see our public events calendar/.ical) and live stream on YouTube
- Subscribe to the latest video tutorials on our YouTube channel
CLIP-as-service is backed by Jina AI and licensed under Apache-2.0. We are actively hiring AI engineers, solution engineers to build the next neural search ecosystem in open-source.