/wmfdata

Tools for working with Wikimedia Foundation–internal data

Primary LanguagePythonBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

wmfdata is an Python package for analyzing Wikimedia data on Wikimedia's non-public analytics clients. It is maintained by the Wikimedia Foundation Product Analytics team.

Features

wmfdata's most popular feature is SQL data access. The hive.run, spark.run, and mariadb.run functions allow you to run commands against these different data sources and receive the results as a Pandas dataframe in one line of code.

Other features include:

  • Easy generation of Spark sessions using spark.get_session
  • Loading CSV or TSV files into Hive using hive.load_csv
  • Turning cryptic Kerberos-related errors into clear reminders to renew your Kerberos credentials

Installation

You should be able to install the latest version of package using the following terminal command:

pip install git+https://github.com/wikimedia/wmfdata-python.git@release

Upgrading

If you have an older version of wmfdata and want to upgrade, simply add the --upgrade flag to the command, resulting in the following:

pip install --upgrade git+https://github.com/wikimedia/wmfdata-python.git@release

Troubleshooting

Importing wmfdata fails with AttributeError: module 'matplotlib.ticker' has no attribute 'PercentFormatter'

This happens because wmfdata requires matplotlib 2.1 or greater, but the preinstalled version on SWAP is older. wmfdata automatically upgrades matplotlib during its installation, but for some reason the new version doesn't take effect immediately. Restarting your Jupyter server should fix it (don't worry, this won't affect anyone else). This is different that restarting an individual notebook's kernel; to restart your server, follow these steps:

  1. Navigate to /hub/home on your Jupyter server (for example, if your server is available at localhost:8000, go to http://localhost:8000/hub/home).
  2. Click the big red button that says "Stop My Server".
  3. Wait a while, reloading the page if necessary, until the red button has disappeared and you see a big green button that says "Start My Server".
  4. Log back in, and everything should work correctly.

Support and bug reports

Tasks related to wmfdata are tracked in Wikimedia Phabricator in the wmfdata-python project.

You can also email the Product Analytics team at product-analytics AT wikimedia.org with questions or feedback.

If you're a hero who would like to contribute code, we welcome pull requests here on GitHub.