josephrocca/lit-encoder-js

LiT (Zero-Shot Transfer with Locked-image text Tuning) image and text encoder models, working in the browser

LiT Image and Text Encoders For Browser & Deno

This repo demonstrates LiT converted to ONNX and running in the browser. Check out the demos below. Note that the models are ~400mb each so they may take a while to initalize depending on your internet speed.

Demos:

• Image embedding: https://josephrocca.github.io/lit-encoder-js/embed_images_onnx.html
• Text embedding: https://josephrocca.github.io/lit-encoder-js/embed_text_onnx.html
• Sort/search a local directory of images: https://josephrocca.github.io/clip-image-sorter/ (use drop-down to select LiT model)

JAX to ONNX conversion notebook: https://colab.research.google.com/github/josephrocca/lit-encoder-js/blob/main/LiT_conversion_jax2tf_to_SavedModel_to_ONNX_text_%26_image_models.ipynb