chroma-core/chromadb-default-embed

Compatibility with Bun

Fruup opened this issue · 2 comments

Fruup commented

System Info

I'm using Bun 1.1.4 inside an ubuntu:22.04 Docker container.

Environment/Platform

  • Website/web-app
  • Browser extension
  • Server-side (e.g., Node.js, Deno, Bun)
  • Desktop app (e.g., Electron)
  • Other (e.g., VSCode extension)

Description

Currently, the package sharp@0.32.0 is used, which depends on libvips@8.14.5. This version of libvips is only distributed via Brotli-compressed files.
Since as of now, Bun (1.1.4) does not implement Brotli compression, installing this package fails ("not implemented"):

bun install v1.1.4 (fbe2fe0c)
  ⚙️  sharp [2/2] sharp: Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.14.5/libvips-8.14.5-linux-arm64v8.tar.br


104 |   };
105 | 
106 |   stream.pipeline(
107 |     fs.createReadStream(tarPath),
108 |     verifyIntegrity(platformAndArch),
109 |     new zlib.BrotliDecompress(),
          ^
error: zlib.BrotliDecompress is not implemented
      at new  (node:zlib:2269:65)
      at extractTarball (/work/node_modules/chromadb-default-embed/node_modules/sharp/install/libvips.js:109:5)
      at /work/node_modules/chromadb-default-embed/node_modules/sharp/install/libvips.js:214:15
      at node:stream:3748:89
      at node:fs:61:45

A solution - which works locally for me - is to bump to sharp@0.33.3, which depends on libvips@8.15.2. Starting at this minor version, the package is distributed via .tar.gz files (as can be seen here: https://github.com/lovell/sharp-libvips/releases).

Reproduction

Try to install this package using Bun: bun add chromadb-default-embed.
The following error should occur:

bun install v1.1.4 (fbe2fe0c)
  ⚙️  sharp [2/2] sharp: Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.14.5/libvips-8.14.5-linux-arm64v8.tar.br


104 |   };
105 | 
106 |   stream.pipeline(
107 |     fs.createReadStream(tarPath),
108 |     verifyIntegrity(platformAndArch),
109 |     new zlib.BrotliDecompress(),
          ^
error: zlib.BrotliDecompress is not implemented
      at new  (node:zlib:2269:65)
      at extractTarball (/work/node_modules/chromadb-default-embed/node_modules/sharp/install/libvips.js:109:5)
      at /work/node_modules/chromadb-default-embed/node_modules/sharp/install/libvips.js:214:15
      at node:stream:3748:89
      at node:fs:61:45
jeffchuber commented

@Fruup thanks for opening the issue.

chroma-default-embed is a thin wrapper around https://www.npmjs.com/package/@xenova/transformers - can you try that repo directly? You can see the JS embedding_function for that here: https://github.com/chroma-core/chroma/blob/main/clients/js/src/embeddings/TransformersEmbeddingFunction.ts

Fruup commented

@jeffchuber Thanks for your reply :) I managed to get it to work with the v3 branch of the original package (xenova/transformers.js#545). Looking forward to its release.