pip install -r requirements.txt
python accelerate
pip install accelerate
sentence Transformers
pip install -U sentence-transformers
vicuna embeddings
minilm v6 sentence transformer resp
vector store
pip install chromadb
pip install streamlit
parquet python
pip install parquet
llm pipeline hugging face
inference langchain
openai embedding pinecone
langchain embeddings vector store chains by langchain llm / embeddings
vectore db vs sql db inbuilt algorithms semantic search levenstein jacquard
lower dimensional space inbuilt default algorithms cosine similarity
db docs lamini-t4-738m pycache .venv license requirements.txt .gitignore
code .
conda activate deeplearning
langchain streamlit UI for citation transformers requests torch einops accelerate large models bitsandbytes pdfminer.six pi pdf bs4 sentence_transformers chromadb
constants.py app.py ingest.py
what's going on in ingest.py
import things
define directory
main function
recursive text splitter
create embeddings store in Chroma
python ingest.py
streamlit run app.py
validate response
client requirements
summarization and retrieval