Jon Sedar Personal Project
Experimental models to demonstrate pymc3 capabilities.
Where suitable, I will submit these Notebooks for inclusion in the pymc3 documentation
Note:
- Each Notebook is designed to be standalone with a suggested MVP condaenv install, but you can also just install the env for this project.
- This project is hosted publicly on GitHub.
- This README is MacOS oriented
This repo uses Python 3.6.*
Download Anaconda distro at https://www.continuum.io/downloads
Theano devs recommend using clang (aka clang++) by default which you can install using Xcode Command Line Tools
$> xcode-select --installIt's also possible to use g++ (gcc), but in my experience that's about 1/2 the speed.
e.g. install via Homebrew
$> brew update && brew upgrade `brew outdated`
$> brew install gcc...optional cleanup of old versions afterwards
$> brew cleanupNote:
- By default
gccis symlinked / points toclangon MacOS - It is possible to install a particular version of
gccvia Homebrew, but this gets installed to a versioned dir e.g./usr/local/Cellar/gcc/9.3.0_1/bin/gcc-9and by default is not symlinked fromgcc, rather it is available asgcc-9 - Regardless, it is best to use
clangbecause theano is/was developed to this and is optimized (much faster) - Note you then don't need to configure the cxx in
.theanorc, it usesclangby default
[user]
name = $YOUR_USERNAME
email = $YOUR_EMAIL
[filter "lfs"]
clean = git-lfs clean -- %f
smudge = git-lfs smudge -- %f
process = git-lfs filter-process
required = true[global]
device=cpu$> jupyter notebook --generate-config
$> jupyter qtconsole --generate-config
$> jupyter nbconvert --generate-config$> git clone https://github.com/jonsedar/pymc3_examples.git
$> cd pymc3_examples/IMPORTANT NOTE: We're not using the latest python here because we want
stability for the pymc3 and theano installs.
Pay extra attention to the .theanorc config
$> conda env create --file condaenv_pymc3_examples.yml
$> activate pymc3_examplesCheat sheet of conda commands available online here.
$> /.pip_install.shRequires NodeJS
$> jupyter labextension install @telamonian/theme-darculaA quick and cheap way to confirm your binaries are good and the installation works!
View the BLAS / MKL install
$> python -c "import numpy as np; np.__config__.show()"Example output...
mkl_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/include']
blas_mkl_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/include']
blas_opt_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/include']
lapack_mkl_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/include']
lapack_opt_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/include']$> python -c "import numpy as np; np.test()"Example output...
7285 passed, 80 skipped, 167 deselected, 12 xfailed, 3 xpassed, 2 warnings in 355.51 seconds$> python -c "import scipy as sp; sp.test()"
... many tests will run and output a results summary e.g. ...Example output...
14522 passed, 1274 skipped, 1225 deselected, 79 xfailed, 10 xpassed, 25 warnings in 777.53 seconds Method 1:
$> python -m pytest -xv --cov=pymc3 --cov-report=html pymc3/Method 2:
Takes ~90mins on my Macbook 2017 :/
$> python -c "import pymc3 as pm; pm.test()"Example output...
TBDTakes forever, see installation docs
Quicker:
$> python -c "import theano; theano.test()"Alternative (takes ~3 hours)
$> theano-nose -s
... many tests will run and output a results summary e.g. ...As per https://stackoverflow.com/questions/40993626/list-memory-usage-in-ipython-and-jupyter
import sys
# These are the usual ipython objects, including this one you are creating
ipython_vars = ['In', 'Out', 'exit', 'quit', 'get_ipython', 'ipython_vars']
# Get a sorted list of the objects and their sizes
sorted([(x, sys.getsizeof(globals().get(x))) for x in dir() if not x.startswith('_') and x not in sys.modules and x not in ipython_vars], key=lambda x: x[1], reverse=True)Remove unused packages and tarballs etc as explained https://conda.io/docs/commands/conda-clean.html
$> conda clean -v -aNOTE @2019-01-19 there's an open bug in conda on Windows that the ~/AppData/Local/Continuum/Anaconda3/pkgs/.trash directory needs manual emptying afterwards
Ought to provide a speed-boost for MKL operations (many numpy, scipy operations under the hood). See docs
$> python -c "import mkl; mkl.set_num_threads(4); print(mkl.get_max_threads())
4Interactive widgets are cool. Beware potential issue with lax requirements. Ensure conda_*.yaml contains correct pairs of :
- ipywidgets==6.0.0
- widgetsnbextension==2.*from ipywidgets import widgets
from IPython.display import display
text = widgets.Text()
text.on_submit(lambda x: print('hello {}'.format(x.value)))
display(text)Straight breakpoint:
import pdb; pdb.set_trace()Or print, often helpful if you can't set a breakpoint
for att in dir(myobject):
print(att, getattr(myobject, att))See data/README_DATA.md
Jonathan Sedar © 2020