HollowDB Vector

Implementation of Hierarchical Navigable Small Worlds (HNSW) index over HollowDB.

Installation

Install the package:

yarn add hollowdb-vector
pnpm add hollowdb-vector
npm install hollowdb-vector

Usage

You can create the VectorDB as follows:

import HollowDBVector from "hollowdb-vector";
import { WarpFactory, defaultCacheOptions } from "warp-contracts";
import { SetSDK } from "hollowdb";
import { Redis } from "ioredis";
import { RedisCache } from "warp-contracts-redis";

// connect to Redis
const redis = new Redis();

// create Warp instance with Redis cache
const warp = WarpFactory.forMainnet().useKVStorageFactory(
  (contractTxId: string) =>
    new RedisCache({ ...defaultCacheOptions, dbLocation: `${contractTxId}` }, { client: redis }),
);

// create HollowDB SDK
const wallet = JSON.parse(readFileSync("./path/to/wallet.json", "utf-8"));
const contractTxId = "your-contract-tx-id";
const hollowdb = new SetSDK<string>(wallet, contractTxId, warp);

// create HollowDB Vector
const vectordb = new HollowDBVector(hollowdb);

Inserting a Vector

With this, you can insert a new point:

const point = [-0.28571999073028564 /* and many more... */, 0.13964000344276428];

// any object
const metadata = {
  name: "My favorite vector!",
};

// insert a point
await vectordb.insert(point, metadata);

Metadata is optional, and you can leave it out during insert. If you would like to set it a later time, you can always do:

vectordb.db.set_metadata(index, metadata);

Note

The complexity of inserting a point may increase with more points in the DB.

Fetching a Vector

You can get a vector by its index, which returns its point value and metadata:

const { point, metadata } = await vectordb.get_vector(index);

Querying a Vector

You can make a query and return top K relevant results:

// a query point
const query = [-0.28571999073028564 /* and many more... */, 0.13964000344276428];

// number of top results to return
const K = 10;

// make a KNN search
const results = await vectordb.knn_search(query, K);

// each result contains the vector id, its distance to query, and metadata
const { id, distance, metadata } = results[0];

Deploying your own Contract

HollowDB Vector exports a static function that allows you two deploy a new contract that you own. Assuming that you have a wallet and a warp instance as described above, you can create a new contract with:

const { contractTxId } = await HollowDBVector.deploy(wallet, warp);
console.log("Deployed at:", contractTxId);

Setup

For local setup of this repo, first clone it.

git clone https://github.com/firstbatchxyz/hollowdb-vector

Then, install packages:

pnpm install

Peer dependencies should be installed automatically.

Protobuffers

We include the pre-compiled protobuf within the repo, but if you were to change the protobuf later, you can generate the compiled code as follows:

# HNSW protobufs
pnpm proto:code:hnsw # generate js code
pnpm proto:type:hnsw # generate types

# Request protobufs
pnpm proto:code:req # generate js code
pnpm proto:type:req # generate types

Testing

Tests are ran over a few cases for a fixed set of $N, K$ parameters that are prepared in Python, and are compared in Typescript. Run the tests via:

pnpm test

Warning

You need a live Redis server for the HollowDB test to work. Furthermore, the HollowDB tests may take some time.

Styling

Check the formatting with:

pnpm format

Lint everything with:

pnpm lint

You can also check types with:

pnpm check

Legacy

HollowDB Vector replaces DANNY, for the legacy code please refer to this branch.

License

HollowDB Vector is licensed under Apache 2.0.

karbalan/danny