This repo is an accompaniment to this Roundup article and extends the work done in A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases, a preprint on arxiv.org from the team at data.world.
Using OpenAI's completions API (gpt-4), we provide a few-shot prompt to introduce the LLM to proper dbt SL syntax (which is otherwise not available due to the knowledge cutoff of April 2023) and ask it to generate a SL query to answer a selection of questions from the benchmark.
- This repo includes an export of our Hex notebook, as well as the files necessary to generate the same results.
- Make a copy of this Google Sheet, which is a csv output of the notebook's final dataframe.
- Run the benchmark yourself by importing a copy of the notebook, then providing your own dbt Cloud service token and OpenAI API key.
Created by Jason Ganz, Joel Labes and Jordan Stein