ConvFinQAInsights is a repository aimed at providing insights into financial question answering (FinQA) models using conversational AI techniques. The repository contains code for data preprocessing, model evaluation, and performance analysis. Additionally, it includes a research document detailing the findings and implications of the analysis, along with instructions on replicating the experiments.
- Data Preprocessing: Efficient mechanisms for preprocessing financial data, including parsing, cleaning, and formatting.
- Model Evaluation: Evaluation scripts for assessing the performance of FinQA models against annotated datasets.
- Performance Analysis: Tools for analyzing model performance metrics and generating insightful visualizations.
- Research Document: A comprehensive research document summarizing key findings, implications, and recommendations based on the analysis.
Before running the code, ensure you have the following dependencies installed:
- Python 3.x
- Required Python packages (install via
pip
):anthropic
,openai
,requests
,numpy
,glob
,hashlib
,json
,math
,re
-
Clone the repository:
git clone https://github.com/your-username/ConvFinQAInsights.git cd ConvFinQAInsights
-
Install the required packages:
pip install -r requirements.txt
-
Data Parsing: The first step is to parse the FinQA dataset using the script
01_parse_finqa_data.py
. This script combines thetrain.json
anddev.json
data files from the original dataset. This merging process ensures correct handling for subsequent analysis. Run the following command to execute the parsing script:python 01_parse_finqa_data.py
-
Data Enrichment: After parsing the dataset, enrich the FinQA data using Anthropic and OpenAI models. Two scripts handle this enrichment process:
02_enrich_finqa_data_anthropic.py
: Enriches the dataset using Anthropic models.03_enrich_finqa_data_openai.py
: Enriches the dataset using OpenAI's GPT-4 model. These scripts can be run in parallel to expedite the enrichment process.
python 02_enrich_finqa_data_anthropic.py python 03_enrich_finqa_data_openai.py
-
Results Analysis: Once the data enrichment is complete, analyze the results for accuracy using the script
04_plot_finqa_results.py
. This script plots the FinQA model performance metrics and generates insightful visualizations for analysis.python 04_plot_finqa_results.py
By following the numerical order (01, 02, 03, 04), you ensure a systematic and streamlined execution of the entire FinQA analysis pipeline, from data parsing to results analysis.
- Research Document: Read the research document Tomoro AI - Insights from the ConvFinQA Dataset.pdf for detailed insights, findings, and recommendations.
This project is licensed under the MIT License - see the LICENSE file for details.