ChatSQC: Grounding LLM Answers to Statistical Quality Control Literature

Hosted ChatSQC

ChatSQC is a Python-based web app that uses the OpenAI GPT-4 Turbo language model. It is hosted at https://chatsqc.osc.edu/. ChatSQC has two modes:

Basic: This mode is grounded in the NIST/SEMATECH Handbook of Engineering Statistics, allowing users to ask questions about foundational industrial statistics, quality control, experimental design, and reliability concepts. In our basic mode, the Chatbot will only present answers based on concepts explained in the vetted, highly referenced NIST/SEMATECH Handbook of Engineering Statistics. While there are many excellent text and reference books, we chose the NIST/SEMATECH Handbook of Engineering statistics as it is in the public domain per the NIST Copyright Notice.
Research: This mode is grounded in the entire collection of open-access (CC BY and CC BY-NC) papers that were available from Technometrics, Quality Engineering, and Quality and Reliability Engineering International by Jan 31, 2024. Combined, these journals have published 52 articles that are open-access, with licenses that allow for derivative work or derivative work for non-commercial purposes. This mode allows users to ask, potentially more advanced, questions about our entire collection of open-access papers. Note that at the time of our data collection, we were unable to find any BY CC or BY CC-NC open-access papers from the Journal of Quality Technology and Quality Technology & Quantitative Management.

Introduction

This is the official implementation of AI and the Future of Work in Statistical Quality Control: Insights from a First Attempt to Augmenting ChatGPT with an SQC Knowledge Base (ChatSQC), where we attempt to address the imprecise answers obtained from generally trained large language models (LLMs) when explaining industrial statistics and quality control concepts, see Megahed et al. 2023 for a detailed discussion. Our work attempts to address this issue and reduce hallucinations by grounding the answers to vetted and high-quality statistical quality control (SQC) reference materials. As opposed to traditional LLMs such as ChatGPT (GPT 4.0 with no plugins), our ChatSQC bot provides the following advantages:

grounded in SQC literature: Our chatbot will only present answers based on concepts explained in NIST/SEMATECH Engineering Statistics Handbook or the 52 CC BY or CC BY-NC open access papers. The grounding of LLMs with reference material reduces hallucinations and improves domain-based response quality.
increased likelihood for ''I do not know'' answers: The grounding also prevents the LLM from generating answers which are not in the reference materials, which allowed us to program the chatbot to state: 'As an SQC chatbot grounded only in ..., I do not know the answer to this question as it is not in my referenced/grounding material. I am sorry for not being able to help.'
highlighting of relevant text chunks: Our chatbot utilizes up to 5 most relevant text chunks to generate its response. In our app, we present these most relevant chunks in HTML disclosure widgets along with their L2 distance to the prompt; in the widget's summary view, we provide: (a) a statement saying "Click for relevant text chunk" to indicate to the user that the text chunk can be viewed in detail by clicking on the text, and (b) we provide the L2-dist to the prompt in the summary view in parentheses after our "click here" statement. This allows users to understand how the generated response was created, providing insights into the response quality and accuracy.
web links of relevant sources: for each text chunk, we provide the title of the webpage (in basic mode, this contains the subsection number and title; in research mode, this contains the APA citation for the paper and its license) along with a hyperlink to its URL to allow the reader to read the full context in which our relevant text chunks were presented in the referenced material. If more than one text chunk belong to a given webpage, we nest them together.
free to use: Our app is free to use and does not require any registration or login. Through the use of the API, we are also not restricted by the rate limits that are imposed on the ChatGPT Web Interface.

Researchers can implement and host their own versions of ChatSQC by setting up a virtual environment with python=3.10.13 and the package versions presented in the requirements.txt file. This allows SQC researchers to have a testbed/playground to examine the impact of the different LLMs, their parameters, and prompting strategies on response quality.

ChatSQC's Design

Let us include our methods figure here and provide a short description of how the chatbot is designed.

Local Installation and Running of the App

If you would like to run the chatbot on your local Python environment, please follow these steps:

Instructions for First-Time Usage:

Clone the repository to your local machine.
Download/Install Anaconda (if needed) and Create a standalone virtual environment (note Python users can also venv to create their virtual environment; we do not assume Python expertise so our instructions are somewhat detailed):
- Download, install, and open Anaconda
- Create a virtual environment by running this at the command line (opened from Anaconda): conda create -n chatsqc python=3.10. It will ask you to click, y; please type: y and click Enter.
- conda activate chatsqc
- Use the command window to cd (change drive) to the location of your requirements.txt file
- Run the command pip install -r requirements.txt to install the required dependencies (libraries with specific versions)
Apply for an OpenAI account. Obtain an API key from OpenAI and add it to a file titled .env, which must be placed in the project directory. The .env file must contain a variable called OPENAI_API_KEY, where you can assign your actual API Key as: OPENAI_API_KEY=sk-mo9KXYZfk7pvRnIcdZzPFU8WlzuJB1EFLmihGYop4YZnTjk. Note that:
- This is a dummy API key, where we have maintained the format of the real key, but it should not have any real functionality or access permissions.
- For exploring and testing the API, all new OpenAI users get free $5 in free tokens. These tokens expire after 3 months. Once you exceed your quota, you can subscribe to one of their paid plans and continue using the API.
Run the following commands in Anaconda's cmd in this order:
- streamlit run ChatSQC.py, which will run the app

Instructions for Running the App every time:

Open the Anaconda virtual environment from Anaconda by clicking on the dropdown menu with the default value of base (root) and you will find chatsqc below it.
Install (if needed) and launch the command window (CMD) after you click on chatsqc.
- Alternatively, you can type: conda activate chatsqc in the CMD from your base environment.
Make sure that your terminal points out to the directory where you have our Python files.
Run the app using: streamlit run ChatSQC.py

Roadmap

Our current version of the app uses gpt-4-0125-preview as the LLM and NIST/SEMATECH Engineering Statistics Handbook. In future versions, we plan on using Llama 2 7B since it is open-sourced and can be locally installed (i.e., there will be no API costs for the embeddings and chat APIs).

Credits

Our ChatSQC's design was inspired by and utilized many features of @alejandro-ao's Ask Multiple PDFs app for online grounding of ChatGPT using multiple PDFs. Due to the large size of our referenced materials, we have modified the app to:

Preprocess the reference materials offline.
Preprocess HTML files instead of PDF files.
Use the recommended RecursiveCharacterTextSplitter() instead of the CharacterTextSplitter() based on the LangChain documentation of the CharacterTextSplitter().
Provide links/titles for the verified sources, relevant text chunks and the L2 distance of these chunks to the users' prompts.

Furthermore, interfacing with LLM and Embedding APIs was simplified by the LangChain Python library. Similarly, the use of the streamlit Python library has streamlined the development of our GUI. We have also capitalized on Chris Klose's suggestions to change the streamlit footer default.

Contributing to ChatSQC

We warmly welcome contributions from our vibrant community and are glad that you are interested in enhancing the ChatSQC app. Your ideas, suggestions, and improvements can help make this application even more user-friendly and efficient. If you have any suggestions or find a bug, please don't hesitate to submit an issue through GitHub's issue tracker. It's a great way to discuss new ideas or get help with problems. Please be as descriptive as possible when submitting an issue to help us understand your concern or proposal better.

If you'd like to contribute code, please fork the repository and create a pull request when you're ready. A pull request (PR) is a way to suggest changes to the project. We use the PR mechanism for both bug fixes and new features. Please include a detailed explanation of your changes, and be sure to reference any issues that will be resolved by your PR. When you make a PR, it will be reviewed by our team. If it meets the guidelines and aligns with our roadmap, we will merge your code into the main branch. Please make sure your code is clean, well-commented, and follows our coding standards.

We greatly appreciate all of our contributors. Each contribution helps us to create a better app for everyone, so thank you for taking the time and making the effort to improve ChatSQC! Happy contributing!

Citing our work

If you make use of our work, please cite our paper:

@article{megahed2023chat,
    author    = {Megahed, Fadel M and Chen, Ying-Ju, and Zwetsloot, Inez and Knoth, Sven and Montgomery, Douglas C and Jones-Farmer, L Allison},
    title     = {{AI} and the Future of Work in Statistical Quality Control: Insights from a First Attempt to Augmenting {ChatGPT} with an {SQC} Knowledge Base {(ChatSQC)}},
    year      = {2023},
    journal={},
    doi = {},
    url = {}
}

fmegahed/chatsqc