/wtf_wikipedia

a pretty-committed wikipedia markup parser

Primary LanguageJavaScriptMIT LicenseMIT

wtf_wikipedia
parse data from wikipedia
npm install wtf_wikipedia
by Spencer Kelly and many contributors

it is very, very hard.
we're not joking!

const wtf = require('wtf_wikipedia')

fetch and parse an article:

wtf.fetch('Toronto Raptors').then(doc => {
  let coach = doc.infobox().get('coach')
  coach.text() //'Nick Nurse'

  doc.sentences(0).text() //'The Toronto Raptors are a Canadian professional basketball team based in Toronto.'
})

get plain-text:

let str = `[[Greater_Boston|Boston]]'s [[Fenway_Park|baseball field]] has a {{convert|37|ft}} wall.`
wtf(str).text()
//"Boston's baseball field has a 37ft wall."

get json:

let doc = await wtf.fetch('Whistling')

let json = doc.json()
json.categories
//['Oral communication', 'Vocal music', 'Vocal skills']

let sec = doc.sections('see also')
sec.links().map(l => l.json())
//[{ page: 'Slide whistle' }, { page: 'Hand flute' }, { page: 'Bird vocalization' }...]

doc.images(0).json()
// {url: https://upload.wikimedia.org..../300px-Duveneck_Whistling_Boy.jpg', file: 'Image:Duveneck Whistling Boy.jpg' }

run on the client-side:

<script src="https://unpkg.com/wtf_wikipedia"></script>
<script>
  wtf.fetch('On a Friday', function(err, doc) {
    // get links from an infobox prop
    let members = doc.infobox().get('current members')
    members.links().map(l => l.page())
    //['Thom Yorke', 'Jonny Greenwood', 'Colin Greenwood'...]
  })
</script>

Ok first, 🛀

Wikitext is no small thing.

Consider:

this library supports many recursive shenanigans, depreciated and obscure template variants, and illicit wiki-shorthands.

What it does:

  • Detects and parses redirects and disambiguation pages
  • Parse infoboxes into a formatted key-value object
  • Handles recursive templates and links- like [[.. [[...]] ]]
  • Per-sentence plaintext and link resolution
  • Parse and format internal links
  • creates image thumbnail urls from File:XYZ.png filenames
  • Properly resolve dynamic templates like {{CURRENTMONTH}} and {{CONVERT ..}}
  • Parse images, headings, and categories
  • converts 'DMS-formatted' (59°12'7.7"N) geo-coordinates to lat/lng
  • parse and combine citation and reference metadata
  • Eliminate xml, latex, css, and table-sorting cruft

What doesn't do:

  • external 'transcluded' page data [1]
  • AST output
  • smart (or 'pretty') formatting of html in infoboxes or galleries [1]
  • maintain perfect page order [1]
  • per-sentence references (by 'section' element instead)
  • maintain template or infobox css styling

It is built to be as flexible as possible. In all cases, tries to fail in considerate ways.

What about html scraping..?

Wikimedia's official parser turns wikitext ➔ HTML.

if you prefer this screen-scraping workflow, you can pluck parts of a page like that.

that's cool, too!

getting structured data this way is still a complex, weird process. Manually spelunking the html is usually just as tricky and error-prone as scanning the wikitext itself.

The contributors to this library have come to that conclusion, as many others have.

This library has (lovingly) borrowed a lot of code and data from the parsoid project, and is gracious to those contributors.

ok, enough chat.

flip your wikitext into a Doc object

import wtf from 'wtf_wikipedia'

let txt = `
==Wood in Popular Culture==
* harry potter's wand
* the simpsons fence
`
wtf(txt)
// Document {text(), json(), lists()...}

doc.links()

let str = `Whistling is featured in a number of television shows, such as [[Lassie (1954 TV series)|''Lassie'']], and the title theme for ''[[The X-Files]]''.`
let doc = wtf(str)
doc.links().map(l => l.page())
// [ 'Lassie (1954 TV series)',  'The X-Files' ]

doc.text()

returns nice plain-text of the article

var wiki =
  "[[Greater_Boston|Boston]]'s [[Fenway_Park|baseball field]] has a {{convert|37|ft}} wall.<ref>{{cite web|blah}}</ref>"
var text = wtf(wiki).text()
//"Boston's baseball field has a 37ft wall."

doc.sections():

(a section is a heading, '==Like This==')

wtf(page)
  .sections(1)
  .children() //traverse nested sections
wtf(page)
  .sections('see also')
  .remove() //delete one

doc.sentences()

s = wtf(page).sentences(4)
s.links()
s.bolds()
s.italics()
s.dates() //structured date templates

doc.categories()

let doc = await wtf.fetch('Whistling')
doc.categories()
//['Oral communication', 'Vocal music', 'Vocal skills']

doc.images()

img = wtf(page).images(0)
img.url() // the full-size wikimedia-hosted url
img.thumbnail() // 300px, by default
img.format() // jpg, png, ..

Fetch

This library can grab, and automatically-parse articles from any wikimedia api. This includes any language, any wiki-project, and most 3rd-party wikis.

// 3rd-party wiki
let doc = await wtf.fetch('https://muppet.fandom.com/wiki/Miss_Piggy')

// wikipedia français
doc = await wtf.fetch('Tony Hawk', 'fr')
doc.sentences(0).text() // 'Tony Hawk est un skateboarder professionnel et un acteur ...'

// accept an array, or wikimedia pageIDs
let docs = wtf.fetch(['Whistling', 2983], { follow_redirects: false })

// article from german wikivoyage
wtf.fetch('Toronto', { lang: 'de', wiki: 'wikivoyage' }).then(doc => {
  console.log(doc.sentences(0).text()) // 'Toronto ist die Hauptstadt der Provinz Ontario'
})

you may also pass the wikipedia page id as parameter instead of the page title:

let doc = await wtf.fetch(64646, 'de')

the fetch method follows redirects.

fetch categories:

wtf.category(title, [lang], [options | callback])

retrieves all pages and sub-categories belonging to a given category:

let result = await wtf.category('Category:Politicians_from_Paris')
//{
//  pages: [{title: 'Paul Bacon', pageid: 1266127 }, ...],
//  categories: [ {title: 'Category:Mayors of Paris' } ]
//}

fetch random article:

wtf.random(title, [lang], [options | callback])

retrieves all pages and sub-categories belonging to a given category:

let result = await wtf.category('Category:Politicians_from_Paris')
//{
//  pages: [{title: 'Paul Bacon', pageid: 1266127 }, ...],
//  categories: [ {title: 'Category:Mayors of Paris' } ]
//}

Good practice:

The wikipedia api is pretty welcoming though recommends three things, if you're going to hit it heavily -

  • pass a Api-User-Agent as something so they can use to easily throttle bad scripts
  • bundle multiple pages into one request as an array (say, groups of 5?)
  • run it serially, or at least, slowly.
wtf
  .fetch(['Royal Cinema', 'Aldous Huxley'], 'en', {
    'Api-User-Agent': 'spencermountain@gmail.com'
  })
  .then(docList => {
    let links = docList.map(doc => doc.links())
    console.log(links)
  })

Plugins

Tutorials

API

  • .title() - get/set the title of the page from the first-sentence
  • .pageID() - get/set the wikimedia id of the page, if we have it.
  • .namespace() - get/set the wikimedia namespace of the page, if we have it
  • .isRedirect() - if the page is just a redirect to another page
  • .redirectTo() - the page this redirects to
  • .isDisambiguation() - is this a placeholder page to direct you to one-of-many possible pages
  • .categories() -
  • .sections() - return a list, or given-index of the Document's sections
  • .paragraphs() - return a list, or given-index of Paragraphs, in all sections
  • .sentences() - return a list, or given-index of all sentences in the document
  • .images() -
  • .links() - return a list, or given-index of all links, in all parts of the document
  • .lists() - sections in a page where each line begins with a bullet point
  • .tables() - return a list, or given-index of all structured tables in the document
  • .templates() - any type of structured-data elements, typically wrapped in like {{this}}
  • .infoboxes() - specific type of template, that appear on the top-right of the page
  • .references() - return a list, or given-index of 'citations' in the document
  • .coordinates() - geo-locations that appear on the page
  • .text() - plaintext, human-readable output for the page
  • .json() - a 'stringifyable' output of the page's main data

Section

  • .title() - the name of the section, between ==these tags==
  • .index() - which number section is this, in the whole document.
  • .indentation() - how many steps deep into the table of contents it is
  • .sentences() - return a list, or given-index, of sentences in this section
  • .paragraphs() - return a list, or given-index, of paragraphs in this section
  • .links() -
  • .tables() -
  • .templates() -
  • .infoboxes() -
  • .coordinates() -
  • .lists() -
  • .interwiki() - any links to other language wikis
  • .images() - return a list, or given index, of any images in this section
  • .references() - return a list, or given index, of 'citations' in this section
  • .remove() - remove the current section from the document
  • .nextSibling() - a section following this one, under the current parent: eg. 1920s → 1930s
  • .lastSibling() - a section before this one, under the current parent: eg. 1930s → 1920s
  • .children() - any sections more specific than this one: eg. History → [PreHistory, 1920s, 1930s]
  • .parent() - the section, broader than this one: eg. 1920s → History
  • .text() -
  • .json() -

Paragraph

  • .sentences() -
  • .references() -
  • .lists() -
  • .images() -
  • .links() -
  • .interwiki() -
  • .text() - generate readable plaintext for this paragraph
  • .json() - generate some generic data for this paragraph in JSON format

Sentence

  • .links() -
  • .bolds() -
  • .italics() -
  • .dates() -
  • .json() -

Image

  • .links() -
  • .thumbnail() -
  • .format() -
  • .json() - return some generic metadata for this image
  • .text() - does nothing

Infobox

  • .links() -
  • .keyValue() - generate simple key:value strings from this infobox
  • .image() - grab the main image from this infobox
  • .get() - lookup properties from their key
  • .template() - which infobox, eg 'Infobox Person'
  • .text() - generate readable plaintext for this infobox
  • .json() - generate some generic 'stringifyable' data for this infobox

List

  • .lines() - get an array of each member of the list
  • .links() - get all links mentioned in this list
  • .text() - generate readable plaintext for this list
  • .json() - generate some generic easily-parsable data for this list

Reference

  • .title() - generate human-facing text for this reference
  • .links() - get any links mentioned in this reference
  • .text() - returns nothing
  • .json() - generate some generic metadata data for this reference

Table

  • .links() - get any links mentioned in this table
  • .keyValue() - generate a simple list of key:value objects for this table
  • .text() - returns nothing
  • .json() - generate some useful metadata data for this table

Configuration

Adding new methods:

you can add new methods to any class of the library, with wtf.extend()

wtf.extend(models => {
  // throw this method in there...
  models.Doc.prototype.isPerson = function() {
    return this.categories().find(cat => cat.match(/people/))
  }
})

await wtf.fetch('Stephen Harper').isPerson() //hmm?

Adding new templates:

does your wiki use a {{foo}} template? Add a custom parser for it:

wtf.extend((models, templates) => {
  // create a custom parser function
  templates.foo = (text, data) => {
    data.templates.push({ name: 'foo', cool: true })
    return 'new-text'
  }

  // array-syntax allows easy-labeling of parameters
  templates.foo = ['a', 'b', 'c']

  // number-syntax for returning by param # '{{name|zero|one|two}}'
  templates.baz = 0

  // replace the template with a string '{{asterisk}}' -> '*'
  templates.asterisk = '*'
})

Notes:

3rd-party wikis

by default, a public API is provided by a installed mediawiki application. This means that most wikis have an open api, even if they don't realize it. Some wikis may turn this feature off.

It can usually be found by visiting http://mywiki.com/api.php

to fetch pages from a 3rd-party wiki:

wtf.fetch('Kermit', { domain: 'muppet.fandom.com' }).then(doc => {
  console.log(doc.text())
})

some wikis will change the path of their API, from ./api.php to elsewhere. If your api has a different path, you can set it like so:

wtf.fetch('2016-06-04_-_J.Fernandes_@_FIL,_Lisbon', { domain: 'www.mixesdb.com', path: 'db/api.php' }).then(doc => {
  console.log(doc.templates('player'))
})

i18n and multi-language:

wikitext is (amazingly) used across all languages, wikis, and even in right-to-left languages. This parser actually does an okay job at it too.

Wikipedia I18n langauge information for Redirects, Infoboxes, Categories, and Images are included in the library, with pretty-decent coverage.

Please make a PR if you see something missing for your language.

Builds:

this library ships seperate client-side and server-side builds, to preserve filesize.

the browser version uses fetch() and the server version uses require('https').

Performance:

It is not the fastest parser, and is very unlikely to beat a single-pass parser in C or Java.

Using dumpster-dive, this library can parse a full english wikipedia in around 4 hours on a macbook.

That's about 100 pages/second, per thread.

See also:

and many more!

MIT