This repo contains the notebooks I used to teach a 2024 Text Analysis Pedagogy Institute course on text classification with LLMs.
In this course, you will learn the basics of using a large language model (specifically, ChatGPT) for text classification. Using the ChatGPT application programming interface (API), we will explore how LLMs can assist humans (and humanists) with various text classification tasks (e.g., binary, labeling, applying confidence intervals to judgments, etc.). We will get to know the API, create validation data, engineer prompts, and automate API calls for large data sets.
Each numbered notebook corresponds with one 90-minute class session.
Sessions presume that participants are already familiar with Python, Jupyter Notebooks, and pandas
.
- What is text classification?
- Why is text classification useful?
- LLMs: The good, the bad, and the ugly
- Overview of LLMs in general
- Distinction between ChatGPT on the website and the API
- Overview of APIs generally and ChatGPT's API specifically
- Overview of JSON and
response_format={ "type": "json_object" }
- How to calculate the cost of a job
- How to get an API key
- Pricing by model, input, and output
- Batch API reduces costs by 50%
- Why classify texts?
- LLMs: The good, the bad, and the ugly
- Advantages of the API: automation, hidden options, structured output
- Sample data: 500 random Jeopardy! questions somebody scraped from J!-Archive and posted on GitHub
- binary, multi-class, multi-label, hierarchical, ordinal
- How well can the LLM approximate human classification?
- Gold-standard data
- Inter-rater reliability
- Measuring Human-LLM agreement
- Precision, recall, and F-score
- Outputting confidence intervals via JSON
- Using
logprobs
to output classification token probabilities
- Systematically testing prompts to find the those that perform best
- How to measure performance
- Beware: garbage in, garbage out (GIGO)
- Prompt engineering techniques
- Generate sample data
- Iterate through questions
- Get classifications in JSON
- Check low confidence classification results
- Test multiple prompts systematically
- Study the classified texts
- Use the classification results as evidence to describe the larger body of texts of which they are a part
- Use that subset of data to extract additional data
- Perform additional classification or labeling steps (e.g., sub-classifications)
- Extract data (e.g., authors and texts from questions about literature)