A bot that accepts PDF docs and lets you ask questions on it.
The LLMs are downloaded and served via Ollama.
make startmake start-gpuWhen the server is up and running, access the app at: http://localhost:8501
Note:
- It takes a while to start up since it downloads the specified model for the first time.
- If your hardware does not have a GPU and you choose to run only on CPU, expect high response time from the bot.
- Only Nvidia is supported as mentioned in Ollama's documentation. Others such as AMD isn't supported yet. Read how to use GPU on Ollama container and docker-compose.
- Make sure to have Nvidia drivers setup on your execution environment for the best results.
Image on DockerHub: https://hub.docker.com/r/amithkoujalgi/pdf-bot
PDF.Bot.Demo.mp4
- The above provided PDFs were used for benchmarking.
- Models used:
Llama2 - Model download time
Llama2-~6-8 minutes
- PC: 3.6 GHz Intel i9 (9th gen), Nvidia RTX 2080, 32 GB memory
- Laptop: 2.6 GHz 6-Core Intel Core i7 MacBook Pro (2019)
| Model | Device | Operation | Time Taken |
|---|---|---|---|
| Llama2 | PC | Load embedding model | <1 minute |
| Llama2 | PC | Create embeddings and vector store | ~3-4 minutes |
| Llama2 | PC | Answer the questions on the uploaded PDFs | ~5-10 seconds |
| Llama2 | Laptop | Load embedding model | ~120 seconds |
| Llama2 | Laptop | Create embeddings and vector store | ~80 seconds |
| Llama2 | Laptop | Answer the questions on the uploaded PDFs | ~150 seconds |
- Expose model params such as
temperature,top_k,top_pas configurable env vars
Contributions are most welcome! Whether it's reporting a bug, proposing an enhancement, or helping with code - any sort of contribution is much appreciated.
docker run -it -v ~/ollama:/root/.ollama -p 11434:11434 ollama/ollamapip install -r requirements.txtstreamlit run pdf_bot/app.pyThanks to the incredible Ollama, Langchain and Streamlit projects.