Resources
dariusk opened this issue · 48 comments
This is an open issue where you can comment and add resources that might come in handy for NaNoGenMo.
There are already a ton of resources on the old resources threads for the 2013 edition and the 2014 edition.
Maybe someone finds my collection of libraries and APIs useful: https://github.com/javierarce/toolbox
After last year's NaNoGenMo I said I would try to extract useful things from the NaNoGenLab and package them in a more useful way, and, so, yes, this is what I extracted:
- T-Rext cleans up spacing and punctuation in a text file.
- Guten-gutter attempts to strip Project Gutenberg boilerplate from a text file (and succeeds more often than gutenizer does)
- seedbank makes any Python script record and be able to replay the random seeds it chooses, with a one-line change. Useful for improving reproducibility.
They all require Python 2.x (tested on Python 2.7.6 but will probably still work on some earlier versions) but only seedbank requires that you write your script in Python - the other two can be used as stand-alone tools.
They're all in the public domain, as is all the stuff in the NaNoGenLab. So, please steal, fold, spindle, and mutilate as you see fit.
Conceptually, you might want to stumble around The Living Handbook of Narratology - for some overviews of narrative theory, and some alternate takes on what might constitute narrative.
Tracey was recently mentioned in Generative Text - the mailing list/google-group that sprouted from the end of NNGM2014. Lately, there's been a lot of partisan ranting about templates. I may resemble some of that remark.
I'd like to point out that the BotAlly slack group has a NaNoGenMo room,
and that the GenArt slack group that sprouted from the Generative Text
mailing list is mostly full of NaNoGenMo2014 people.
On Mon, Oct 26, 2015 at 11:40 AM Michael Paulukonis <
notifications@github.com> wrote:
Tracey was recently mentioned in Generative Text
https://groups.google.com/forum/#!forum/generativetext - the mailing
list/google-group that sprouted from the end of NNGM2014. Lately, there's
been a lot of partisan ranting about templates. I may resemble some of that
remark.—
Reply to this email directly or view it on GitHub
#1 (comment)
.
Since we're speaking of visual novels, I should probably mention the Yarn dialog editor. I don't know that'll be all that useful directly for generative stuff, but in case someone is trying to procedurally generate a dialog tree, it might be handy. Or you could always generate a Twine file.
The genart group is at https://generativeart.slack.com ; I think it might
be invite-only. Mike Paulukonis controls the group & can probably send
invites.
On Mon, Oct 26, 2015 at 12:15 PM Isaac Karth notifications@github.com
wrote:
@enkiv2 https://github.com/enkiv2 Where's the signup for the GenArt
group? I need to check that out.—
Reply to this email directly or view it on GitHub
#1 (comment)
.
As for botally, the public signup is here: http://t.co/OiOB7197BH ; people
periodically check for registrations and send invites out.
On Mon, Oct 26, 2015 at 12:18 PM John Ohno john.ohno@gmail.com wrote:
The genart group is at https://generativeart.slack.com ; I think it
might be invite-only. Mike Paulukonis controls the group & can probably
send invites.On Mon, Oct 26, 2015 at 12:15 PM Isaac Karth notifications@github.com
wrote:@enkiv2 https://github.com/enkiv2 Where's the signup for the GenArt
group? I need to check that out.—
Reply to this email directly or view it on GitHub
#1 (comment)
.
Yeah, there's not a lot going on in https://generativeart.slack.com - and if there's a way to allow people to request an invite, I'm all ears. @dariusk - what's the source for the #botALLY registration doc+script?
UPDATE: he's the GenerativeArt invite page: http://tinyurl.com/genartslackrequest
@MichaelPaulukonis I've seen people use a Google Form to manage signups for Slack groups.
So far as I can tell, the signup just is a web form that sends an email to
the slack maintainer. Maybe google forms has this functionality by itself?
On Mon, Oct 26, 2015 at 12:29 PM Isaac Karth notifications@github.com
wrote:
@MichaelPaulukonis https://github.com/MichaelPaulukonis I've seen
people use a Google Form to manage signups for Slack groups.—
Reply to this email directly or view it on GitHub
#1 (comment)
.
There's been a lot of work done with Neural Networks this year, including text-based output (such as the generated Magic: the Gathering cards). That might be an interesting avenue to pursue.
Abulafia uses MediaWiki for its generators. Not directly useful, but I keep forgetting to mention it here.
Here's a method of procedurally generating a wilderness in a text adventure format. People who are doing a Swallows type simulation approach may find its ideas useful.
Multiverse JSON is a light JSON syntax/spec and Python compiler script with which to store editorial projects in small logical units so they can be quickly reconfigured according to build parameters you define.
Since last year's NaNoGenMo I've made a few libraries that might be of interest:
- pronouncingpy, a simple and easy-to-use interface to the CMU pronouncing dictionary
- pronouncingjs, a Javascript port of the above
- pycorpora, a Python interface for Corpora Project
I also made Context-Free GenGen, which is sort of like a mini-Tracery except driven by Google Sheets (a la the original GenGen). Probably not great for making a whole novel, but maybe cool for prototyping ideas with context-free grammar generation.
- some NYPL-related resources
- if you want to add some savory meals to your novel, you could try the NYPL menus dataset
- a sanitized version of the menus dataset by @trevormunoz and Katie Rawson
I have a few tools I've made in the past year too:
- GGC https://github.com/enkiv2/ggc, a language for generative
grammars that compiles to python (very similar to tracery)
Some novelty filters:
- dirtyreader https://github.com/enkiv2/dirtyreader, a tool to
increase the profanity of text in a configurable way - aphasia https://github.com/enkiv2/aphasia and synonym-warp
https://github.com/enkiv2/synonym-warp, tools for replacing words with
related words - tone-poem https://github.com/enkiv2/tonepoem, blackout-gen
https://github.com/enkiv2/blackout-gen, sestina-gen
https://github.com/enkiv2/sestina-gen, and haiku-generator
https://github.com/enkiv2/haiku-generator: tools for producing
different varieties of poems from source text
And, some novelty generators:
- anxiety-generator https://github.com/enkiv2/anxiety-generator, a
python port of Paul Ford's tool for procedurally generated
passive-aggressive verbal abuse - Tinder For Guinea Pigs https://github.com/enkiv2/tfgp, which
generates nonsense startup pitches - tftf-gen https://github.com/enkiv2/tftf-gen, which describes
randomly generated science fiction set pieces (based on a description of
Thing From The Future)
On Mon, Oct 26, 2015 at 2:21 PM Allison Parrish notifications@github.com
wrote:
Since last year's NaNoGenMo I've made a few libraries that might be of
interest:
- pronouncingpy https://github.com/aparrish/pronouncingpy, a simple
and easy-to-use interface to the CMU pronouncing dictionary- pronouncingjs https://github.com/aparrish/pronouncingjs, a
Javascript port of the above- pycorpora https://github.com/aparrish/pycorpora, a Python
interface for Corpora Project https://github.com/dariusk/corporaI also made Context-Free GenGen http://cfgg.decontextualize.com/, which
is sort of like a mini-Tracery except driven by Google Sheets (a la the
original GenGen http://tinysubversions.com/gengen/). Probably not great
for making a whole novel, but maybe cool for prototyping ideas with
context-free grammar generation.—
Reply to this email directly or view it on GitHub
#1 (comment)
.
If you're looking for a large corpus of internet comments, the mostly complete Reddit corpus is available.
i have been pretty heavily tooling up for nanogenmo this year by way of publishing a ton of poetic node modules to npm.
- word-vomit replaces words with other words of the same part of speech
- queneau buckets for letters
- queneau buckets for words
- replace words with words that rhyme
- n-plus, replace words with the word N words ahead of it in a dicitonary
- replace character sequences with other character sequences, good for l-system poetry potentially
- detect lipogrammatic text (omits certain characters)
- detect text that snowballs (i.e, every word is 1 char longer than the previous)
- add some random diacritic marks to yr text
Another resource i have created is the weirdly-specific-corpora project, which is a fork of dariusk/corpora for any lists that are too weirdly specific to be included in that project. I expect I will end up making a lot of weird lists while generating novels.
Someone (I believe @JKirchartz) posted this link to an actual spam blog comment generator template that was accidentally posted as a comment on a guy's blog.
There is a generator for hilariously poorly written sex scenes
http://www.fiftyshadesgenerator.com/. The source is embedded in the page
there.
On Tue, Oct 27, 2015 at 4:14 PM Darius Kazemi notifications@github.com
wrote:
Someone (I believe @JKirchartz https://github.com/jkirchartz) posted
this link to an actual spam blog comment generator template
http://alexking.org/blog/2013/12/22/spam-comment-generator-script that
was accidentally posted as a comment on a guy's blog.—
Reply to this email directly or view it on GitHub
#1 (comment)
.
See sample code in various languages via the Language Survey, issue #17
@enkiv2 suggested that a short introduction to some historical techniques might be useful in the resources thread to point out possible approaches for people who might not be familiar with some of the precedents of text generation. So I thought I'd write about a few of them.
Dada incorporated several techniques that involved randomly assembling prior texts into new poetry. The cut-up technique takes an existing text, cuts it to pieces, and then reassembles the bits in a new order. This list of Surrealist techniques might also be inspirational.
Oulipo is a group of writers who use constraints to define their writing. Relevant examples include Queneau's Cent Mille Milliards de Poèmes (A Hundred Thousand Billion Poems), which combines ten-line sonnets line-by-line; Perec's Life a User's Manual, which is structured like a knight's tour of a chessboard; Queneau's Excercises in Style, 99 retellings of the same story; Calvino's The Castle of Crossed Destinies, stories interperted via Tarot cards; and Queneau's "A Story as You Like It".
Even where they aren't directly generative, the approaches used by Oulipo writers often point to alternative forms a novel can take. For example, Calvino's Invisible Cities and If on a winter's night a traveler.
Oulipo is also significant because many of its members wrote about systems for generating stories and about incorporating computers into writing. Italio Calvino's essay "Prose and Anticombinatorics," about using the computer to to find the constraints for a murder mystery; Paul Fournel's "Computer and Writer: The Centre Pompidou Experiment"; and Claude Berge's "For a Potential Analysis of Combinatory Literature" are particularly relevant.
Lastly, Jorges Luis Borges tends to crop up a lot, particularly for his short stories: "The Library of Babel", "The Book of Sand", "An Examination of the Work of Herbert Quain", "The Garden of Forking Paths", "Pierre Menard, Author of the Quixote", and "Tlön, Uqbar, Orbis Tertius".
After @rbechtel mentioned the source code for Micro-Talespin, I went poking around to see what other early AI storytelling systems had source code availible.
Micro-Talespin in Common Lisp, by Warren Sack
another source
[About Talespin](http://lispm.de/mts https://grandtextauto.soe.ucsc.edu/2006/09/13/the-story-of-meehans-tale-spin/)
For Tale-Spin itself: In 2008 Meehan found a copy of the original MLisp source code for Tale-Spin (sans the data files). There's an extensive discussion of the source code in Wardrip-Fruin's Expressive Processing, though I'm not aware of any copies online.
Eliza/Doctor:
In Java
Also in Java
BASIC
JavaScript
Python
The original Lisp source code- github repo
Skald: a Scala reimplementation of Minstrel Remixed, which is itself a Scala reimplementation of Scott Turner's Minstrel (originally in Lisp). Scott Turner on MINSTREL.
As an example of exercises in minimalism and apophenia, Nick Monfort has a
collection of one kilobyte story generators in python.
https://grandtextauto.soe.ucsc.edu/2008/11/30/three-1k-story-generators/
On Sat, Oct 31, 2015 at 11:20 AM Isaac Karth notifications@github.com
wrote:
After @rbechtel mentioned the source code for Micro-Talespin
#59, I went poking
around to see what other early AI storytelling systems had source code
availible.Micro-Talespin in Common Lisp, by Warren Sack
http://lispm.de/source/misc/micro-talespin.lisp
another source http://eliterature.org/images/microtalespin.txt
About Talespin
http://lispm.de/mts%20https://grandtextauto.soe.ucsc.edu/2006/09/13/the-story-of-meehans-tale-spin/For Tale-Spin itself
https://grandtextauto.soe.ucsc.edu/2006/09/13/the-story-of-meehans-tale-spin/:
In 2008 Meehan found a copy of the original MLisp source code for Tale-Spin
(sans the data files). There's an extensive discussion of the source code
in Wardrip-Fruin's Expressive Processing, though I'm not aware of any
copies online.Eliza/Doctor: https://en.wikipedia.org/wiki/ELIZA
In Java
https://code.google.com/p/simple-semantic-desktop/source/browse/trunk/Progs2/Eliza/eliza.java?r=4
Also in Java http://www.chayden.net/eliza/Eliza.html
BASIC https://www.jesperjuul.net/eliza/ELIZA.BAS
JavaScript https://www.jesperjuul.net/eliza/
Python http://www.jezuk.co.uk/cgi-bin/view/software/eliza
The original Lisp source code http://elizagen.org/\- github repo
https://github.com/jeffshrager/elizagenSkald https://sites.google.com/a/soe.ucsc.edu/eis-skald/: a Scala
reimplementation of Minstrel Remixed, which is itself a Scala
reimplementation of Scott Turner's Minstrel (originally in Lisp). Scott
Turner on MINSTREL
https://grandtextauto.soe.ucsc.edu/2007/10/30/scott-turner-on-minstrel/.—
Reply to this email directly or view it on GitHub
#1 (comment)
.
@ikarth et alia - this year I have been looking at Tale-Spin a bit, even though I don't think I will be using it as a model for what I will be doing. However, I do have some more resources in my NMGM repo and will be adding more there as the month progresses.
Public service announcement: if you try to download too many files from Project Gutenberg too quickly, they'll ban you for 24 hours.
Gutenberg has a bot policy & instructions on how to avoid rate limits:
https://www.gutenberg.org/wiki/Gutenberg:Information_About_Robot_Access_to_our_Pages
(Not sure if this applies to official mirrors. Unofficial mirrors are also
floating around.)
On Sun, Nov 1, 2015 at 9:44 AM Darius Kazemi notifications@github.com
wrote:
If you're hitting Gutenberg rate limits there are also tons of mirror
sites: http://www.gutenberg.org/MIRRORS.ALL—
Reply to this email directly or view it on GitHub
#1 (comment)
.
Project Gutenberg also has CDs and DVDs of many books for offline use, I've used one of these a lot.
https://www.gutenberg.org/wiki/Gutenberg:The_CD_and_DVD_Project
https://github.com/TheCommieDuck/nell-extractor
I've just written a quick thing that extracts useful data from CMU's NELL ontology.
Output is a list such as:
lemon_balm_tea beverage agriculturalproduct Lemon Balm Tea
which is an entity called lemon_balm tea, which is a type of beverage and a type of agriculturalproduct, and is called Lemon Balm Tea in English.
Plenty of false-positive entries - beverage general gives about 8,700 entries, mostly different kinds of tea and wine - but it's nice enough if someone wants to try the ontological approach?
I'll probably be refining it throughout the month.
For those interested in Project Gutenberg texts, there is also GITenberg, mentioned in #89 by NNGM participant @sethwoodworth
If you need food, NYPL has an API around (and CSV dumps of) their historical collection of restaurant menus at What's On The Menu?
Lots of the dishes and prices have been transcribed and are also available via the API.
I made a Python interface: https://github.com/hugovk/whatsonthemenu
(for https://twitter.com/menubot and http://menubot.tumblr.com)
For those of us using NLTK: Text Analysis with NLTK Cheatsheet: https://blogs.princeton.edu/etc/files/2014/03/Text-Analysis-with-NLTK-Cheatsheet.pdf
neural-storyteller is a recurrent neural network for generating little stories about images. Code is at https://github.com/ryankiros/neural-storyteller
and samples https://medium.com/@samim/generating-stories-about-images-d163ba41e4ed
Linked Data has links to a large number of datasets, many of which are also Open Data.
Datahub lists a large number of datasetes, and can be filtered by license, if you're only interested in public domain or Creative Commons datasets.
Pleiades is a gazetter of ancient geographic places.
Baba - a Javascript-based designer for text generators. A generator that outputs text generators should be a useful resource for non-coders, but it looks like the live web demo is down so it'll take a little bit of work to get it to run if you download it.
I've created a thread for beginner's questions and tutorials, as a supplement to the main Resources thread: #152
Working in Python? Having unicode issues? Even if you properly take care of the proper encoding and decoding on your end, your source data may be incorrectly encoded. Or, worse, Windows might have mucked it up. If you've got “ or — all over your text, you might want to consider using ftfy. Fixes bad unicode and mojibake; doesn't convert non-unicode to unicode.
If you want to avoid certain words in your output text, here's a number of things. These are mainly aimed at bots, but should still be generally useful.
Here's a JavaScript, Python, Ruby and PHP word filter:
https://github.com/dariusk/wordfilter
Here's a headline filter:
https://github.com/molly/CyberPrefixer/blob/master/offensive.py
Tips on transphobic joke detection:
http://tinysubversions.com/notes/transphobic-joke-detection/
Some lists of bad words:
https://github.com/shutterstock/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words
https://gist.github.com/ryanlewis/a37739d710ccdb4b406d
http://www.bannedwordlist.com/lists/swearWords.txt
[Inactive] muted Twitter topics:
https://github.com/sjml/bot-innocence
Some general etiquette things:
http://tinysubversions.com/2013/03/basic-twitter-bot-etiquette/
http://www.crummy.com/2013/11/27/0
If you need to plug in some Markov Chains in Perl, you could do a lot worse than the module I built for it this year.
https://github.com/greg-kennedy/MarkovChain
- List of Longest Novels (not necessarily available as free etexts)
- Top 100 Gutenberg texts - changes weekly
- Project Gutenberg tips-n-tricks
- Vocabulary analysis of Project Gutenberg
For anyone working with a twitter-sourced corpus, myna is a "JS lib that compiles a tweet to machine speakable text". Or non-twitter-literate text.
For example, the tweet:
"OH: Myna is #SuperAwsome for #TTS and #TwitterBeginners! It is written by @kn bit.ly/ykbz6b"
compiles into plain English:
"Katsuya Noguchi overheard: Myna is Super Awesome for TTS and Twitter Beginners! It is written by Katsuya Noguchi"