Generate burnup, cycle time, and other charts from Phabricator. Intended for experimental prototyping and proof of concept for similar functionality to built into Phabricator. Also a platform to do complex scripted data handling for Phab projects prior to reporting. works from data at at http://dumps.wikimedia.org/other/misc/phabricator_public.dump, generated by https://gerrit.wikimedia.org/r/#/c/214398/2/wmfphablib/phabdb.py
- --load
- reads a JSON file produced by phabricator dump.
- loads a postgresql database containing the data
- --reconstruct
- Reconstructs the historical state of all tasks in the specified projects project day by day
- --report
- Generates CSV files for all reports
- Uses R to graph the data as PNG files
wget http://dumps.wikimedia.org/other/misc/phabricator_public.dump
cd phlogiston
python3 phlogiston.py --load --reconstruct --report --scope ve_source.py
~phlogiston/ <- dump goes here
~phlogiston/phlogiston/ <- program goes here
~phlogiston/html <- html index and reports go here
~/tmp/ <- PNG output goes here
Postgresql database named "phab" <- data goes here
- Procure an Linux server. Instructions are tested on a WMF labs virtual Ubuntu 14.04 host, and should work verbatim for any standard Ubuntu 14.04 host; they may need modificaton otherwise.
- Install prerequisites on the system. As root:
- Create a Phlogiston user.
adduser phlogiston
- Download Phlogiston:
su - phlogiston
git clone https://github.com/wikimedia/phlogiston
exit
- (Probably obsolete: As of 2016-02-24, Labs 14.04 image seems to include Postgresql 9.5, which is better. If using stock 14.04, follow instructions to add Postgresql backport to get 9.4. 9.3 and earlier are missing features that Phlogiston uses): http://www.postgresql.org/download/linux/ubuntu/)
- Get access to newer R
echo deb http://cran.es.r-project.org/bin/linux/ubuntu trusty/ > /etc/apt/sources.list.d/r.list
gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9
gpg -a --export E084DAB9 | sudo apt-key add -
apt-get update
- Install ubuntu packages
apt-get install nginx postgresql python3-pip python3-psycopg2 python3-dev postgresql-contrib r-base r-base-dev python3-dateutil
- Install R packages.
R
install.packages(c("ggplot2", "ggthemes", "argparse", "reshape"))
update.packages()
quit()
- Note: there seems to be a fatal bug in ggthemes 3.0.1; get 2.2.1 or 3.0.2.
- Set up Nginx website
cp ~phlogiston/phlogiston/site-phlogiston /etc/nginx/sites-available
rm /etc/nginx/sites-enabled/default
ln -s /etc/nginx/sites-available/site-phlogiston /etc/nginx/sites-enabled
service nginx restart
- Change permissions so that the postgres user can share files with phlogiston
usermod -a -G phlogiston postgres
- restart postgres so that this takes effect
- Add Python packages
pip3 install pyscopg2 pytz jinja2
- Set up database. As user postgres,
createuser -s phlogiston
createdb -O phlogiston phab
- Set up the website and cron job. As phlogiston,
mkdir ~/html
cp ~/phlogiston/html/*.html ~/html
cp ~/phlogiston/html/*css ~/html
- Set up the script to run via cron
crontab -e
- put
15 4 * * * bash ~/phlogiston/batch_phlog.bash -m incremental -s phl -s tpg -s ve -s and -s ios -s col -s cot -s discir -s dismap -s dis -s diswik -s fr -s ja -s red -s rel -s ana >>~/phlog.log 2>&1
at the end of the crontab and save and exit
- Do the initial run of Phlogiston. As phlogiston:
cd ~/phlogiston
~/phlogiston/phlogiston.py --initialize
bash ~/phlogiston/batch_phlog.bash -m incremental -s phl -s tpg -s ve -s and -s ios -s col -s cot -s discir -s dismap -s dis -s diswik -s fr -s ja -s red -s rel -s ana >>~/phlog.log 2>&1