/metk

Model Evaluation Toolkit

Primary LanguagePythonMIT LicenseMIT

metk

Model Evaluation Toolkit

In metk, I've collected a set of routines for evaluating predictive models.
I put a lot of this code together when I was doing the evaluation for the
TDT and D3R
projects, as well as
a book chapter I wrote in 2013.
I'm releasing this project as a way for the community to collaborate
and (hopefully) agree on best practices for model evaluation. Most of the
initial release is oriented toward the evaluation of free energy calculations.
This is just a start and I plan to add a lot more. Currently, there are
routines to calculate
  • Root mean squared (RMS) error
  • Mean absolute error (MAE)
  • Pearson correlation coefficient (with confidence limits)
  • Spearman rank correlation (rho) (still need to add confidence limits)
  • Kendall tau (still need to add confience limits)
  • Maximum possible correlation given a specific experimental error. This is based on on a 2009 paper by Brown, Muchmore and Hajduk
Most of the statistics is done with routines from scikitlearn
and scipy.
The toolkit also includes code to generate a few diagnositc plots that I
find helpful when looking at model performance. Examples of these plots can be found
here
  • A scatter plot of experimental vs predicted ΔG. Lines are drawn at 1 and 2 kcal error
  • A histogram of the error distribution.
  • The two plots above with ΔG converted to a binding affinity (in uM or nM). On the scatter plot, lines are drawn at 5-fold and 10-fold error. I find that I mentally relate to a fold error in binding affinity better than I do to error expressed in kcal/mol. However, if you like looking at error in kcal/mol, use that plot.
Ultimately, the plan is to implement a number of other methods for model
evaluation including those described in papers by Anthony Nicholls.

Usage

This relase of metk contains a rudimentary command-line interface. More options
will be added in time.
Usage: metk.py --in INFILE_NAME --prefix OUTFILE_PREFIX [--units UNIT_NAME] [--example]

--in INFILE_NAME         input file name
--prefix OUTFILE_PREFIX  prefix for output file names
--units UNIT_NAME        units to display (uM (default) or nM)
--example                show example command lines

Installation

The toolkit works under both Python 2.7 and Python 3.6. Installation is relatively painless.

  1. Install the dependencies, you can do this with pip

    pip install numpy pandas matplotlib scipy docopt
  2. Get the code from github. You can either download and unpack the zip file
    or just clone the repository.
    git clone https://github.com/PatWalters/metk.git
  3. There's one more trick to make the plots work with matplotlib. When pip installed
    matplotlib, it created a directory under your home directory called .matplotlib. Create
    a file in this directory called matplotlibrc and put this line in that file.
backend: TkAgg
  1. At this point you should be all set. The main script is metk.py. The other Python
    files need to either be in the same directory or in your PYTHONPATH. You can
    then run the script with this command.
    python metk.py
    If you're running under Linux or OS-X and you hate typing "python" all the time
    (I know I do) you can do
    chmod +x metk.py
./metk.py

A Few Notes

I use tabs

Please don't hesitate to let me know if you run into problems or have additions or improvements.

Pat Walters - July 2017