Tools being compared:
- Paper QA https://github.com/whitead/paper-qa
- mayooear https://github.com/mayooear/gpt4-pdf-chatbot-langchain
- LlamaIndex https://github.com/jerryjliu/llama_index
- Dagster https://github.com/petehunt/langchain-github-bot
- Onboard https://www.getonboard.dev/chat/zulip/zulip
My personal rank:
- Onboard
- Mendable
- Mayo
- Paper QA
- Dagster
- Unoptimized LlamaIndex
Paper QA | mayooear | LlamaIndex default | Dagster | |
---|---|---|---|---|
embedding model | ? | ? | ? | ? |
vector DB | FAISS | Pinecone | SimpleVectorStore | FAISS |
similarity | FAISS (L2 distance) | cosine | cosine | FAISS (L2 distance) |
response mode | tree_summarize | ? | compact | langchain stuff |
All of them use GPT-3.5.
Paper QA
PAPERQA_PROMPT = (
"Write an answer "
"for the question below based on the provided context. "
"If the context provides insufficient information, "
'reply "I cannot answer". '
"For each part of your answer, indicate which sources most support it "
"via valid citation markers at the end of sentences, like (Example2012). "
"Answer in an unbiased, comprehensive, and scholarly tone. "
"If the question is subjective, provide an opinionated answer in the concluding 1-2 sentences. \n\n"
"{context_str}\n"
"Question: {query_str}\n"
"Answer: "
)
mayooear
const QA_PROMPT = `You are a helpful AI assistant. Use the following pieces of context to answer the question at the end.
If you don't know the answer, just say you don't know. DO NOT try to make up an answer.
If the question is not related to the context, politely respond that you are tuned to only answer questions that are related to the context.
{context}
Question: {question}
Helpful answer in markdown:`;
LlamaIndex
DEFAULT_TEXT_QA_PROMPT_TMPL = (
"Context information is below.\n"
"---------------------\n"
"{context_str}\n"
"---------------------\n"
"Given the context information and not prior knowledge, "
"answer the question: {query_str}\n"
)
Dagster
Questions 6-8 are rephrases to figure out if the bot would have answered differently based on the phrasing.
Questions asked:
- How do I configure LDAP authentication in Zulip?
- How do I import Slack workspace export into a Zulip organization?
- How do I upgrade my Zulip instance to the latest Git version?
- How do I create a self-signed SSL certificate for my Zulip server?
- How do I delete a Zulip organization?
- Can you explain the rough working of Zulip?
- Can you describe Zulip architecture, as in, its inner working?
- Can you describe an overview of Zulip architecture?
- How do I get an account's username via the Zulip Python API?
- How I set up a bot for Zulip?
- What framework does Zulip use for its frontend?