/speak-memory

Code and data to support "Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4"

Primary LanguagePython

gpt4-books

Code and data to support the following paper:

Kent K. Chang, Mackenzie Cramer, Sandeep Soni and David Bamman (2023), "Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4", ArXiv.

This work probes what books GPT-4 and ChatGPT have memorized by assessing the degree to which models are able to predict the masked token (a proper name) in name cloze passages like the following:

Though of a very ingenious mechanical turn, [MASK] could never get this table to suit him. He put chips under it, blocks of various sorts, bits of pasteboard, and at last went so far as to attempt an exquisite adjustment by final pieces of folded blotting paper.

Detailed results on all books can be found here: gpt4-books.

Generating name cloze passages

To generate name cloze passages for your own texts, see the generate_name_cloze folder.

100 most memorized books

The following books in our set have the highest GPT-4 name cloze accuracy (where accuracy here is the fraction of times a model predicts the masked name exactly right in 100 passages from a book). Full name cloze accuracy results (GPT-4, ChatGPT and BERT) for all books can be found here: gpt4-books.

GPT-4 Accuracy ChatGPT Accuracy BERT Accuracy Year Author Title
0.98 0.82 0.00 1865 Lewis Carroll Alice's Adventures in Wonderland
0.76 0.43 0.00 1997 J.K. Rowling Harry Potter and the Sorcerer's Stone
0.74 0.29 0.00 1850 Nathaniel Hawthorne The Scarlet Letter
0.72 0.11 0.00 1892 Arthur Conan Doyle The Adventures of Sherlock Holmes
0.70 0.10 0.00 1815 Jane Austen Emma
0.65 0.19 0.00 1823 Mary Wollstonecraft Shelley Frankenstein; Or, The Modern Prometheus
0.62 0.13 0.00 1813 Jane Austen Pride and Prejudice
0.61 0.18 0.00 1838 Charles Dickens Oliver Twist
0.61 0.30 0.00 1853 Herman Melville Bartleby, the Scrivener: A Story of Wall-Street
0.61 0.35 0.00 1884 Mark Twain Adventures of Huckleberry Finn
0.61 0.08 0.00 1897 Bram Stoker Dracula
0.59 0.22 0.00 1851 Herman Melville Moby Dick; Or, The Whale
0.59 0.13 0.00 1902 Arthur Conan Doyle The Hound of the Baskervilles
0.58 0.35 0.00 1876 Mark Twain The Adventures of Tom Sawyer
0.57 0.30 0.00 1949 George Orwell 1984
0.54 0.10 0.00 1908 L. M. Montgomery Anne of Green Gables
0.51 0.20 0.01 1954 J.R.R. Tolkien The Fellowship of the Ring
0.49 0.16 0.13 2012 E.L. James Fifty Shades of Grey
0.49 0.16 0.00 1847 Charlotte Brontë Jane Eyre: An Autobiography
0.49 0.12 0.00 1883 Robert Louis Stevenson Treasure Island
0.49 0.22 0.00 1903 Jack London The Call of the Wild
0.49 0.24 0.01 1911 Frances Hodgson Burnett The Secret Garden
0.48 0.14 0.00 2008 Suzanne Collins The Hunger Games
0.47 0.20 0.01 1916 James Joyce A Portrait of the Artist as a Young Man
0.45 0.09 0.00 1847 Emily Brontë Wuthering Heights
0.45 0.10 0.01 1868 Louisa May Alcott Little Women
0.43 0.27 0.00 1954 William Golding Lord of the Flies
0.43 0.17 0.00 1979 Douglas Adams The Hitchhiker's Guide to the Galaxy
0.42 0.08 0.00 1890 Oscar Wilde The Picture of Dorian Gray
0.42 0.03 0.00 1898 Henry James The Turn of the Screw
0.41 0.20 0.00 1912 Edgar Rice Burroughs Tarzan of the Apes
0.40 0.06 0.00 1861 Charles Dickens Great Expectations
0.37 0.06 0.00 1897 H. G. Wells The Invisible Man: A Grotesque Romance
0.36 0.02 0.02 1914 James Joyce Dubliners
0.34 0.05 0.00 1849 Charles Dickens David Copperfield
0.33 0.02 0.01 1794 Ann Ward Radcliffe The Mysteries of Udolpho
0.30 0.16 0.00 1959 Chinua Achebe Things Fall Apart
0.28 0.12 0.00 1977 J. R. R. Tolkien and Christopher Tolkien The Silmarillion
0.28 0.09 0.00 1922 James Joyce Ulysses
0.27 0.13 0.00 1953 Ray Bradbury Fahrenheit 451
0.27 0.13 0.00 1996 George R.R. Martin A Game of Thrones
0.26 0.05 0.01 2003 Dan Brown The Da Vinci Code
0.26 0.08 0.00 1965 Frank Herbert Dune
0.25 0.20 0.01 1937 Zora Neale Hurston Their Eyes Were Watching God
0.25 0.04 0.00 1851 Nathaniel Hawthorne The House of the Seven Gables
0.25 0.03 0.00 1861 George Eliot Silas Marner
0.25 0.01 0.00 1920 Edith Wharton The Age of Innocence
0.25 0.14 0.00 1961 Harper Lee To Kill a Mockingbird
0.24 0.03 0.03 1953 Ian Fleming Casino Royale
0.24 0.00 0.00 1915 W. Somerset (William Somerset) Maugham Of Human Bondage
0.23 0.09 0.00 1891 Thomas Hardy Tess of the d'Urbervilles: A Pure Woman
0.23 0.05 0.00 1903 Henry James The Ambassadors
0.22 0.13 0.00 1984 William Gibson Neuromancer
0.22 0.00 0.00 1885 H. Rider (Henry Rider) Haggard King Solomon's Mines
0.21 0.02 0.00 1900 Theodore Dreiser Sister Carrie: A Novel
0.20 0.10 0.00 1985 Orson Scott Card Ender's Game
0.20 0.03 0.00 1852 Charles Dickens Bleak House
0.20 0.04 0.00 1877 Anna Sewell Black Beauty
0.19 0.12 0.00 1932 Aldous Huxley Brave New World
0.19 0.07 0.00 1826 James Fenimore Cooper The Last of the Mohicans; A narrative of 1757
0.19 0.06 0.00 1868 Wilkie Collins The Moonstone
0.19 0.00 0.00 1922 F. Scott (Francis Scott) Fitzgerald The Beautiful and Damned
0.18 0.05 0.00 1855 Elizabeth Cleghorn Gaskell North and South
0.18 0.04 0.00 1905 Emmuska Orczy, Baroness Orczy The Scarlet Pimpernel
0.18 0.07 0.00 1937 Margaret Mitchell Gone with the Wind
0.17 0.05 0.00 1968 Philip K. Dick Do Androids Dream of Electric Sheep?
0.17 0.03 0.00 1848 Anne Brontë The Tenant of Wildfell Hall
0.17 0.08 0.00 1908 E. M. (Edward Morgan) Forster A Room with a View
0.16 0.06 0.05 2009 Dan Brown The Lost Symbol
0.16 0.02 0.01 1848 William Makepeace Thackeray Vanity Fair
0.16 0.03 0.00 1871 George Eliot Middlemarch
0.16 0.05 0.01 1879 Henry James Daisy Miller: A Study
0.15 0.04 0.04 2013 Dan Brown Inferno
0.15 0.08 0.01 2014 Veronica Roth Divergent
0.15 0.07 0.00 1899 Kate Chopin The Awakening, and Selected Short Stories
0.15 0.03 0.00 1903 Samuel Butler The Way of All Flesh
0.15 0.04 0.00 1940 John Steinbeck The Grapes of Wrath
0.14 0.00 0.00 1913 D. H. (David Herbert) Lawrence Sons and Lovers
0.13 0.05 0.00 1983 James Kahn Return of the Jedi
0.13 0.02 0.01 1928 D. H. Lawrence Lady Chatterley's Lover
0.13 0.01 0.00 1749 Henry Fielding History of Tom Jones, a Foundling
0.13 0.04 0.00 1818 Jane Austen Persuasion
0.13 0.03 0.00 1977 Alex Haley Roots
0.12 0.02 0.00 1874 Thomas Hardy Far from the Madding Crowd
0.11 0.11 0.00 1961 Irving Stone The Agony and the Ecstasy
0.11 0.01 0.03 1957 Ian Fleming From Russia with Love
0.11 0.04 0.00 1962 Madeleine L'Engle A Wrinkle in Time
0.11 0.04 0.01 1939 Marjorie Kinnan Rawlings The Yearling
0.10 0.05 0.00 1975 E. L. Doctorow Ragtime
0.10 0.05 0.00 1929 Dashiell Hammett The Maltese Falcon
0.10 0.08 0.07 1991 Diana Gabaldon Outlander
0.10 0.02 0.00 1989 Kazuo Ishiguro The Remains of the Day
0.10 0.05 0.00 1898 Elizabeth Von Arnim Elizabeth and Her German Garden
0.10 0.00 0.00 1915 Charlotte Perkins Gilman Herland
0.10 0.01 0.00 1983 Alice Walker The Color Purple
0.09 0.02 0.00 1934 Dorothy L. Sayers The Nine Tailors
0.09 0.03 0.00 1985 Margaret Atwood The Handmaid's Tale
0.09 0.01 0.00 1778 Fanny Burney Evelina, Or, the History of a Young Lady's Entrance into the World
0.09 0.02 0.00 1896 Sarah Orne Jewett The Country of the Pointed Firs
0.09 0.04 0.00 1988 Toni Morrison Beloved