DocQA-Bench

DocQA-Bench is a library for benchmarking document question-answering systems. It provides a flexible framework for evaluating various components of a QA pipeline, including document chunking, embedding generation, vector storage, question generation, and answer generation.

Features

Modular design allowing easy swapping of components
Support for custom document preprocessing
Integration with OpenAI's GPT models for question and answer generation
Customizable chunking strategies
Vector store integration for efficient similarity search
F1 score evaluation for answer quality
Asynchronous processing for improved performance

Installation

You can install DocQA-Bench using pip:

pip install docqa-bench

Quick Start

Check out the example.py file in the root directory for a demonstration of how to use docqa_bench. This example shows how to:

Set up the necessary components Create a benchmark Run the benchmark on preprocessed content Handle and display the results

To run the example: python example.py

Usage

Here's a basic example of how to use DocQA-Bench:

import asyncio
from docqa_bench import (
    Benchmark, PreprocessedDocument, SimpleChunker, OpenAIEmbedder,
    ChromaStore, OpenAIQuestionGenerator, OpenAIAnswerGenerator, F1Evaluator
)

async def run_benchmark(content: str):
    document = PreprocessedDocument(content)
    chunker = SimpleChunker(chunk_size=1000, chunk_overlap=200)
    embedder = OpenAIEmbedder("text-embedding-ada-002")
    vector_store = ChromaStore("benchmark_collection")
    question_generator = OpenAIQuestionGenerator("gpt-4o-mini")
    answer_generator = OpenAIAnswerGenerator("gpt-4o-mini")
    evaluator = F1Evaluator()

    benchmark = Benchmark(
        document, chunker, embedder, vector_store,
        question_generator, answer_generator, evaluator
    )
    
    results = await benchmark.run()
    print(results)

asyncio.run(run_benchmark("Your document content here"))

Components

DocQA-Bench consists of several modular components:

Document: Represents the input document. Use PreprocessedDocument for already processed text.
Chunker: Splits the document into manageable pieces. SimpleChunker is provided out of the box.
Embedder: Generates vector representations of text. OpenAIEmbedder uses OpenAI's embedding model.
VectorStore: Stores and retrieves vector representations. ChromaStore is provided as a default implementation.
QuestionGenerator: Generates questions based on the document content. OpenAIQuestionGenerator uses OpenAI's GPT model.
AnswerGenerator: Generates answers to questions. OpenAIAnswerGenerator uses OpenAI's GPT model.
Evaluator: Evaluates the quality of generated answers. F1Evaluator is provided as a default implementation.

Customization

You can create custom implementations of any component by subclassing the respective base classes in the docqa_bench.core module.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License.

Support

If you encounter any problems or have any questions, please open an issue on the GitHub repository.

JoshuaOliphant/docqa_bench