jina-ai/GSoC

Expand ANNLite capabilities with BM25 to build Hybrid Search

Nick17t opened this issue · 5 comments

Nick17t commented

Project idea 4: Expand ANNLite capabilities with BM25 to build Hybrid Search

info details
Skills needed Python, C++, Lucene, ANN, Inverted Index
Project size 350 hours
Difficulty level Hard
Mentors @Felix Wang @Joan Martínez @Girish Chandrashekar

Project Description

  • In relation to Research about deploying LLM with Jina project, another interesting approach would be to incorporate BM25 and Hybrid Search into ANNLite, which would enable Jina to build scalable Hybrid Search solutions in the cloud with a powerful default solution.
  • ANNlite is a Vector search library developed by Jina which is using HNSW as the algorithm to perform a search. On top of this, it allows the filtering of Documents.
  • However, it can be important for the performance of search systems to be able to combine Vector Search algorithms with traditional text-search ones to get the best of both worlds.
  • This project is about evaluating and trying to apply Hybrid Search approaches on top of ANNLite.

Resources:

Expected outcomes

  • ANNLite is ready to be used as a default library to solve Hybrid Search applications.
JoanFM commented

ANNlite is a Vector search library developed by Jina which is using HNSW as the algorithm to perform search. On top of this it allows to do filtering on Documents.

However, it can be important for the performance of search systems to be able to combine Vector Search algorithms with traditional text-search ones to get the best of both worlds.

This project is about evaluating and trying to apply Hybrid Search approaches on top of ANNLite.

Resources:

  • ANNLite github [https://github.com/jina-ai/annlite]
  • BM25 [https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables]
  • HNSW [https://github.com/naver/splade]
  • Splade [https://github.com/naver/splade]
  • 👍1
matchyc commented

Hi, Michael here. I'm familiar with ANNS, including various graph-based indexes (NSG, HNSW, Vamana, etc.), and a contributor to the Milvus community (advanced vector database project). I'm trying to understand the details in ANNLite. I will deliver my proposal draft as soon as possible.

Just a few concerns: Does this project need to integrate models to generate sparse vectors? It means that we only need to focus on hybrid search (maybe hybrid index construction) not how the input vectors (dense or sparse) are produced, am I correct?

JoanFM commented

Hi, Michael here. I'm familiar with ANNS, including various graph-based indexes (NSG, HNSW, Vamana, etc.), and a contributor to the Milvus community (advanced vector database project). I'm trying to understand the details in ANNLite. I will deliver my proposal draft as soon as possible.

Just a few concerns: Does this project need to integrate models to generate sparse vectors? It means that we only need to focus on hybrid search (maybe hybrid index construction) not how the input vectors (dense or sparse) are produced, am I correct?

It is correct, it should not care about how to create them at the beginning at least

matchyc commented

I'm going to submit the proposal with a detailed framework design, but should I talk to mentors before submitting it?

Nasafato commented

I think you can submit first. I just submitted for this as well.

  • 👍1