Owl is now a public repo.
Can't say I recommend anyone try to install or run it yet. Both installation and run docs sorely lacking. It also needs far too many python pkgs. Sorry.
I'll add some installation docs soon.
Meanwhile, here is an initial project description:
Owl is my LLM research assistant. It has several capabilities, many stand-alone, most integrated into a chatbot-style UI.
- react-style chat loop including a recency/similarity 'react-thought-trace' based memory-stream for context.
- Ability to accept uri's of research papers for insertion into local research library.
- RAG-style retrieval over library
- "Search" - interactive search and ingest of papers from SemanticScholar
- "Research" - a thoughtful, reasoned, library search, with citations.
- "Report" - a full walkthrough of the process of creating a research report draft, from topic description through outlining, online and/or local background research, and multi-section report creation.
Under the covers it supports a working memory and a scripting language (and pieces of a planner). Works in progress.
The newest adventure is LLMOntology construction from the ingested papers. Hypothesis: There is far more, and more nuanced, knowledge in the research literature than in any current formal ontology or ontologic database. This is a work in progress, though, and not available through the UI. I'll post a link to the paper draft eventually.
Owl supports
- OpenAI, Claude, and Mistral APIs
- Owl has three local LLM servers: a wrapper for exllamav2 (my fav), llama.cpp, and hf (transformers).
- Owl supports templating. Automatic, usually, unless the llm isn't coded into the server.
#Install This is one of the biggest weak spots. I suspect in its current state you just can't do it. For now, you can try to clone the repo then:
cd owl/src/owl
python3 Owl.py
and painfully work through the hundreds of missing python packages. I'll get a requirements.txt 'real soon now', I promise.
Owl maintains several subdirectories, files, and faiss indices. It mostly will autocreate ones it doesn't find. Testing a clean install is on my 'todo' list, so I suspect you will eventually run into an issue with a missing directory that it doesn't autocreate, or some other nonsense.
I'm going to be testing and checking out all that in the next few days, sorry.
# start exllamav2 local llm server - only if running a local llm using exllamav2, unneeded if using OpenAI, Anthropic, or Mistral
# put this in background after telling it what model you want to load. It assumes models are in ./models/ dir
uvicorn exl_server:app --host 0.0.0.0 --port 5004 #exllamav2 local llm server on default Owl port - bg after telling it what model to load
# google search and pdf retrieval server. allows you to work in parallel w google searches
uvicorn chat_search_service:app --host 127.0.0.1 --port 5005 &
# paper indexing server - needs to know what llm to use, I should make that a command line arg....
# but for now just put it in background after answering.
uvicorn index_service:app --host 127.0.0.1 --port 5006 # paper indexing server - bg after telling it what llm to use.
# Grobid pdf parser. I usually run this on a separate machine, takes 23GB VRam
sudo docker run -d --rm --gpus '"device=1"' --init --ulimit core=0 -p 8070:8070 -p 8081:8071 grobid/grobid:0.8.1-SNAPSHOT
# finally, we can start Owl UI
python3 Owl.py
First time it will have to create a bunch of sub-directories
<All keys matched successfully>
My city and state is: Berkeley, CA
...various other startup msgs...
gpt-4
...
gpt-3.5-turbo
gpt-4-turbo
claude-3-opus
claude-3-sonnet
claude-3-haiku
mistral-tiny
mistral-small
mistral-medium
My city and state is: Berkeley, CA
My city and state is: Berkeley, California
port(5004):
On startup, Owl will show (on command line) a list of models, followed by "port (5004):". If you are using anything other than a local model on a non-standard port as llm backend, just press 'enter'
Next you will see 'template'. Enter any of the model names listed earlier, or just 'enter' to use the local model with its default template. (If Owl doesn't already know about the local model, and it probably doesn't, you can enter the template manually, e.g. vicuna, alpaca, llama-3, chatml, and a couple of dozen others). If you run into trouble running a local model, post an issue for now.)
The UI has tooltips!
Disclaimer: my emphasis at the moment is on quality, not speed. Owl can spend time 'thinking' sometimes, esp if you use larger llms like Llama-3-70B, my current favorite. Llama-3-8B is pretty fast. I like the Dolphin and Smaug fine-tunes too!.
With that disclaimer, enough of Owl has moved from development-mode to daily-use mode that I will be expanding documentation in the near future.
Notes:
- the local-llm templating is a hardcoded mess adapted (stolen) from an early version of fastapi. I'll be changing that to using the template jinja in config.json when available when I get a chance. Anthropic and Mistral formatting are done in the drivers to provider spec
- instruction-following can be challenging. I try not to spend my life writing prompts. There are a LOT of them in Owl. They work pretty reliably with OpenAI, Anthropic, and MistralAI, and with most of the larger OS models I've tried. YMMV.
- yes, you can easily change Owl's image. You can even put several in images/, and it will rotate through them on click.