/phlogiston

A script to pull Phabricator task history out of MySQL and denormalize it for reporting

Primary LanguagePython

Phlogiston

Purpose

Generate burnup, cycle time, and other charts from Phabricator. Intended for experimental prototyping and proof of concept for similar functionality to built into Phabricator. Also a platform to do complex scripted data handling for Phab projects prior to reporting. works from data at at http://dumps.wikimedia.org/other/misc/phabricator_public.dump, generated by https://gerrit.wikimedia.org/r/#/c/214398/2/wmfphablib/phabdb.py

Function

  1. --load
  • reads a JSON file produced by phabricator dump.
  • loads a postgresql database containing the data
  1. --reconstruct
  • Reconstructs the historical state of all tasks in the specified projects project day by day
  1. --report
  • Generates CSV files for all reports
  • Uses R to graph the data as PNG files

TODOs

Typical usage:

wget http://dumps.wikimedia.org/other/misc/phabricator_public.dump
cd phlogiston
python3 phlogiston.py --load --reconstruct --report --scope ve_source.py

Environment:

~phlogiston/                       <- dump goes here
~phlogiston/phlogiston/            <- program goes here
~phlogiston/html                   <- html index and reports go here
~/tmp/                             <- PNG output goes here
Postgresql database named "phab"   <- data goes here

Installation Notes:

  1. Procure an Linux server. Instructions are tested on a WMF labs virtual Ubuntu 14.04 host, and should work verbatim for any standard Ubuntu 14.04 host; they may need modificaton otherwise.
  2. Install prerequisites on the system. As root:
  3. Create a Phlogiston user. adduser phlogiston
  4. Download Phlogiston:
  • su - phlogiston
  • git clone https://github.com/wikimedia/phlogiston
  • exit
  1. (Probably obsolete: As of 2016-02-24, Labs 14.04 image seems to include Postgresql 9.5, which is better. If using stock 14.04, follow instructions to add Postgresql backport to get 9.4. 9.3 and earlier are missing features that Phlogiston uses): http://www.postgresql.org/download/linux/ubuntu/)
  2. Get access to newer R
    • echo deb http://cran.es.r-project.org/bin/linux/ubuntu trusty/ > /etc/apt/sources.list.d/r.list
    • gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9
    • gpg -a --export E084DAB9 | sudo apt-key add -
    • apt-get update
  3. Install ubuntu packages
    • apt-get install nginx postgresql python3-pip python3-psycopg2 python3-dev postgresql-contrib r-base r-base-dev python3-dateutil
  4. Install R packages.
    • R
    • install.packages(c("ggplot2", "ggthemes", "argparse", "reshape"))
    • update.packages()
    • quit()
    • Note: there seems to be a fatal bug in ggthemes 3.0.1; get 2.2.1 or 3.0.2.
  5. Set up Nginx website
    • cp ~phlogiston/phlogiston/site-phlogiston /etc/nginx/sites-available
    • rm /etc/nginx/sites-enabled/default
    • ln -s /etc/nginx/sites-available/site-phlogiston /etc/nginx/sites-enabled
    • service nginx restart
  6. Change permissions so that the postgres user can share files with phlogiston
    • usermod -a -G phlogiston postgres
    • restart postgres so that this takes effect
  7. Add Python packages
    • pip3 install pyscopg2 pytz jinja2
  8. Set up database. As user postgres,
    • createuser -s phlogiston
    • createdb -O phlogiston phab
  9. Set up the website and cron job. As phlogiston,
    • mkdir ~/html
    • cp ~/phlogiston/html/*.html ~/html
    • cp ~/phlogiston/html/*css ~/html
    • Set up the script to run via cron
      • crontab -e
      • put 15 4 * * * bash ~/phlogiston/batch_phlog.bash -m incremental -s phl -s tpg -s ve -s and -s ios -s col -s cot -s discir -s dismap -s dis -s diswik -s fr -s ja -s red -s rel -s ana >>~/phlog.log 2>&1 at the end of the crontab and save and exit
  10. Do the initial run of Phlogiston. As phlogiston:
  • cd ~/phlogiston
  • ~/phlogiston/phlogiston.py --initialize
  • bash ~/phlogiston/batch_phlog.bash -m incremental -s phl -s tpg -s ve -s and -s ios -s col -s cot -s discir -s dismap -s dis -s diswik -s fr -s ja -s red -s rel -s ana >>~/phlog.log 2>&1