Superlinked is a compute framework for your information retrieval and feature engineering systems, focused on turning complex data into vector embeddings within your RAG, Search, RecSys and Analytics stack.
Our current release allows you to explore our computational model in simple scripts and python notebooks, our next major release will focus on helping you run Superlinked in production, with built-in data infra and vector database integrations.
Visit Superlinked for more information about the company behind this product and our other initiatives.
If you like what we do, give us a star! ⭐
- RAG: HR Knowledgebase
- Semantic Search: Movies, Business News
- Recommendation Systems: E-commerce
- Analytics: User Acquisition
You can check a full list of examples here.
- Describe your data using Python classes with the @schema decorator.
- Describe your vector embeddings from building blocks with Spaces.
- Combine your embeddings into a queryable Index.
- Define your search with dynamic parameters and weights as a Query.
- Load your data using a Source.
- Define your transformations with a Parser (e.g.: from
pd.DataFrame). - Run your configuration with an Executor.
You can check a list of our features or head to our documentation.
Example on how to use Superlinked to experiment with the semantic search use-case.
Install the superlinked library:
%pip install superlinked
Ensure your python version is 3.10.x.
$> python -V
Python 3.10.9
If your python version is not 3.10.x you might use pyenv to install it.
Upgrade pip and install the superlinked library
$> python -m pip install --upgrade pip
$> python -m pip install superlinked
First run will take slightly longer as it has to download the embedding model.
from superlinked.framework.common.schema.schema import schema
from superlinked.framework.common.schema.schema_object import String
from superlinked.framework.common.schema.id_schema_object import IdField
from superlinked.framework.dsl.space.text_similarity_space import TextSimilaritySpace
from superlinked.framework.dsl.index.index import Index
from superlinked.framework.dsl.query.param import Param
from superlinked.framework.dsl.query.query import Query
from superlinked.framework.dsl.source.in_memory_source import InMemorySource
from superlinked.framework.dsl.executor.in_memory.in_memory_executor import InMemoryExecutor
@schema # Desribe your schemas.
class Document:
id: IdField # Each schema should have exactly one `IdField`.
body: String # Use `String` for text fields.
document = Document()
relevance_space = TextSimilaritySpace(text=document.body, model="sentence-transformers/all-mpnet-base-v2") # Select your semantic embedding model.
document_index = Index([relevance_space]) # Combine your spaces to a queryable index.
query = Query(document_index).find(document).similar(relevance_space.text, Param("query_text")) # Define your query with dynamic parameters.
source: InMemorySource = InMemorySource(document) # Connect a data source to your schema.
executor = InMemoryExecutor(sources=[source], indices=[document_index]) # Tie it all together to run your configuration.
app = executor.run()
source.put([{"id": "happy_dog", "body": "That is a happy dog"}])
source.put([{"id": "happy_person", "body": "That is a very happy person"}])
source.put([{"id": "sunny_day", "body": "Today is a sunny day"}])
print(app.query(query, query_text="Who is a positive friend?")) # Run your query.Ready to go to production? We are launching our first Vector DB connectors soon! Tell us which Vector DB we should support!
- Vector DB Comparison: Open-source collaboritve comparison of vector databases by Superlinked.
- Vector Hub: VectorHub is a free and open-sourced learning hub for people interested in adding vector retrieval to their ML stack
If you encounter any challanges during your experiments, feel free to create an issue, request a feature or to start a discussion. Make sure to group your feedback in separate issues and discussions by topic. Thank you for your feedback!