vearch: A Go repository from TopHK

Overview

Vearch is a scalable distributed system for efficient similarity search of deep learning vectors.

Architecture

Data Model

space, documents, vectors, scalars
Components

Master, Router and PartitionServer
Master

Responsible for schema mananagement, cluster-level metadata, and resource coordination.
Router

Provides RESTful API: create , delete search and update ; request routing, and result merging.
PartitionServer (PS)

Hosts document partitions with raft-based replication.

Gamma is the core vector search engine implemented based on faiss. It provides the ability of storing, indexing and retrieving the vectors and scalars.

Quick start

One-click binary installation, please see docs/BinaryInstallation.md(中文版).
Quickly compile the source codes to build a distributed vector search system with RESTful API, please see docs/SourceCompileDeployment.md.
Vearch can be leveraged to build a complete visual search system to index billions of images. The image retrieval plugin for object detection and feature extraction is also required. For more information, please refer to docs/Quickstart.md.
Vearch Python SDK enables vearch to use locally. Vearch python sdk can be installed easily by pip install vearch. For more information, please refer to docs/APIPythonSDK.md.

APIs and Use Cases

Document

Benchmarks

benchmarks

Publication

Jie Li, Haifeng Liu, Chuanghua Gui, Jianyu chen, Zhenyun Ni, Ning Wang, Yuan Chen, Qiang Liu. The Design and Implementation of a Real Time Visual Search System on JD E-commerce Platform. In the 19th International ACM Middleware Conference, December 10–14, 2018, Rennes, France. https://arxiv.org/abs/1908.07389