ilastik depends on 60+ packages. Most of those packages are already provided for us by the Anaconda Python distribution. For 20+ the packages that aren't provided by Anaconda, we use the recipes in this repo.
These recipes are built using the conda-build tool. The resulting binaries are uploaded to the ilastik anaconda channel, and can be installed using the conda package manager.
- Installing ilastik for development
- Generating a release binary
- How to build these packages yourself
- Appendix: Writing a new recipe
- Appendix: Compiler details
- Appendix: Linux VM Details
- Appendix: TODO/TBD
Preamble: Depending on what you are trying to do, you may not need to follow any of these steps. The ilastik binary is shipped with a complete conda environment, .py
files, etc. For many purposes, simply downloading the binary and editing the (python) source code by hand may suffice. However, these instructions can give you a more complete developer setup, suitable e.g. for C++ development.
Here's how to install everything you need to develop ilastik.
0. Prerequisite: Install Miniconda
# Install miniconda to the prefix of your choice, e.g. /my/miniconda
# LINUX:
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
# MAC:
wget https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
bash Miniconda3-latest-MacOSX-x86_64.sh
# Activate conda
CONDA_ROOT=`conda info --root`
source ${CONDA_ROOT}/bin/activate root
NOTE:
When using conda
, make sure you are not using any of python's site-specific or user-specific customization features. In particular, make sure the following environment variables are not defined in your terminal:
PYTHONPATH
PYTHONUSERSITE
PYTHONUSERBASE
Also, make sure there are no python-related directories in ~/.local/
.
Some ilastik workflows require commercial solvers, for which one must purchase or obtain an academic license. If you don't have CPLEX and Gurobi on your machine, you can install everything else with this command:
conda create -n ilastik-devel ilastik-dependencies-no-solvers -c ilastik-forge -c conda-forge
If you have both CPLEX and Gurobi on your machine, you can install the full ilastik development setup, including full support for tracking and multicut.
First, define these environment variables:
export CPLEX_ROOT_DIR=/path/to/ibm/ILOG/CPLEX_Studio1251
export GUROBI_ROOT_DIR=/path/to/gurobi650/linux64
Now you can install the ilastik-dependencies
package:
conda create -n ilastik-devel ilastik-dependencies -c ilastik-forge -c conda-forge
Note: To be really sure that you're getting the right version of ilastik-dependencies
, you can require a specific version and build of the package with PKGNAME=VERSION=BUILD
syntax:
conda create -n ilastik-devel ilastik-dependencies=1.2.0=6 -c ilastik-forge -c conda-forge
If you only have one of CPLEX or Gurobi, and you're seeking to develop for a workflow that requires it, you must install some dependencies of that workflow individually. For example, to install tracking with CPLEX, but not Gurobi:
conda create -n ilastik-devel install ilastik-dependencies-no-solvers -c ilastik-forge -c conda-forge
conda install -n ilastik-devel multi-hypotheses-tracking-with-cplex -c ilastik-forge -c conda-forge
For example, to install multicut with Gurobi support:
conda create -n ilastik-devel install ilastik-dependencies-no-solvers -c ilastik-forge -c conda-forge
conda install -n ilastik-devel nifty-with-gurobi -c ilastik-forge -c conda-forge
${CONDA_ROOT}/envs/ilastik-devel/run_ilastik.sh --debug
So far, our environment contains the ilastik source, but not the git repos.
If you need to edit the ilastik python code,
replace the ilastik-meta
directory with the full git repo.
Note: This will remove both ilastik-meta
and ilastik-dependencies
, but all of the other dependencies in your environment will remain.
CONDA_ROOT=`conda info --root`
DEV_PREFIX=${CONDA_ROOT}/envs/ilastik-devel
conda remove -n ilastik-devel ilastik-meta
# Re-install ilastik-meta.pth
cat > ${DEV_PREFIX}/lib/python2.7/site-packages/ilastik-meta.pth << EOF
../../../ilastik-meta/lazyflow
../../../ilastik-meta/volumina
../../../ilastik-meta/ilastik
EOF
# Option 1: clone a fresh copy of ilastik-meta
git clone http://github.com/ilastik/ilastik-meta ${DEV_PREFIX}/ilastik-meta
cd ${DEV_PREFIX}/ilastik-meta
git submodule update --init --recursive
git submodule foreach "git checkout master"
# Option 2: Symlink to a pre-existing working copy, if you have one.
cd ${DEV_PREFIX} && ln -s /path/to/ilastik-meta
-
(Prerequisite) Update the version number.
-
Edit
ilastik.__version_info__
(inilastik/__init__.py
) and commit your change. -
Commit to
ilastik-meta
and add a matching git tag:cd ${DEV_PREFIX}/ilastik-meta git commit -m "Alpha Release 1.2.3a4" ilastik lazyflow volumina git push origin master git tag -m "Alpha Release" -a 1.2.3a4 git push --tags origin
-
Double-check your conda configuration (
.condarc
). You should allow access to theilastik
channel anddefaults
, but nothing else:$ cat ~/.condarc channels: - ilastik - defaults
-
Build
ilastik-meta
andilastik-dependencies
packages, and upload to theilastik
anaconda channel.WITH_SOLVERS=1 conda build ilastik-meta ilastik-dependencies anaconda upload -u ilastik ${CONDA_ROOT}/conda-bld/linux-64/ilastik-meta*.tar.bz2 anaconda upload -u ilastik ${CONDA_ROOT}/conda-bld/linux-64/ilastik-dependencies*.tar.bz2
Troubleshooting Tip: If the ilastik-meta
tag has been relocated since you last built the ilastik-meta
package, you should probably clear conda's git cache for that repo, to ensure you have the new tags: rm -rf $(conda info --root)/conda-bld/git_cache/github.com/ilastik/ilastik-meta
-
(Optional) Install to a local environment and test
conda create -n test-env ilastik-dependencies=1.2.3a4 -c ilastik-forge -c conda-forge cd ${CONDA_ROOT}/envs/test-env ./run_ilastik.sh
-
Create tarball/app
Linux:
$ grep Usage ./create-tarball.sh ## Usage: create-tarball.sh [--skip-tar] [--git-latest] [--no-tracking] [... extra install-args, e.g. --use-local or -c ilastik ...] $ ./create-tarball.sh -c ilastik-forge -c conda-forge
Mac:
$ grep Usage ./osx-packages/create-osx-app.sh ## Usage: create-osx-app.sh [--compress] [--git-latest] [--no-tracking] [... extra install-args, e.g. --use-local or -c ilastik ...] $ ./osx-packages/create-osx-app.sh --compress -c ilastik-forge -c conda-forge
If any options are used in the Linux or Mac binary creation scripts above, they must be passed in this order:
--skip-tar
: (Linux only) Create theilastik-release
environment, but don't compress it into a .tar.bz2 file.--compress
: (Mac only) After creating the.app
bundle, compress it into a.tar.bz2
file.--git-latest
: Use the latestmaster
branch ofilastik
,lazyflow
, andvolumina
instead of the most recent tag. (Don't use for official releases.)--no-tracking
: Omit tracking-specific dependencies--use-local
: Tells conda to use your custom builds of each package, if available.-c ilastik-forge
: Tells conda to use packages from the ilastik-forge channel (in case it's missing from~/.condarc
).
Windows:
## create new environment for packaging and activate it $ conda create -n ilastik-release ilastik-dependencies -c ilastik-forge -c conda-forge $ activate ilastik-release ## install exe and preconfigured installer generation script $ conda install ilastik-exe ilastik-package -c ilastik-forge -c conda-forge ## build the installer using Inno setup ## ACTION REQUIRED: open the file ${YOUR_CONDA_ENV_PATH}/package/ilastik.iss in Inno Setup ## and build the installer. ## delete environment $ activate root $ conda env remove -n ilastik-release
Note: see https://github.com/ilastik/ilastik-publish-packages for an automated way of building all packages required by ilastik
Warning: the description below is outdated
All of the recipes in this repo should already be uploaded to the ilastik anaconda channel.
The linux packages were built on CentOS 5.11, so they should be compatible with most modern distros.
The Mac packages were built with MACOSX_DEPLOYMENT_TARGET=10.7
, so they should theoretically support OSX 10.7+.
But if, for some reason, you need to build your own binary packages from these recipes, it should be easy to do so:
# Prerequisite: Install conda-build
source activate root
conda install conda-build
# Clone the ilastik build recipes
git clone http://github.com/ilastik/ilastik-build-conda
cd ilastik-build-conda
# Build a recipe, e.g:
conda build --numpy=1.11 vigra
# Now install your newly built package, directly from your local build directory:
conda install --use-local -n ilastik-devel vigra
Now run ilastik from with your ilastik-meta
repo:
cd /path/to/ilastik-meta
# Run ilastik
PYTHONPATH="ilastik:lazyflow:volumina" python ilastik/ilastik.py
As mentioned above, some packages require CPLEX and Gurobi. To build those packages, you must define some environment variables first:
# Configure environment for building with solvers active
export WITH_SOLVERS=1
export CPLEX_ROOT_DIR=/path/to/ibm/ILOG/CPLEX_Studio1251
export GUROBI_ROOT_DIR=/path/to/gurobi650/linux64
# Build some recipes that depend on solvers
conda build ilastik-dependencies
The conda documentation explains in detail how to create a new package, but here's a quick summary:
source activate root
conda install conda-build
Add a directory to this repo:
cd ilastik-build-conda
mkdir somepackage
cd somepackage
A complete recipe has at least 3 files:
meta.yaml
build.sh
(used for both Mac and Linux)bld.bat
(used for Windows)
...additional files (such as patches) may be needed for some recipes.
Write meta.yaml:
$ cat > meta.yaml
package:
name: somepackage
version: 1.2.3
source:
fn: somepackage-1.2.3.tar.bz2
url: http://www.randompackages.org/somepackage/somepackage-1.2.3.tar.bz2
md5: b060bb137d6bd8accf8f0c4c59d2746d
requirements:
build:
- zlib
- python
run:
- zlib
- python
about:
home: http://www.somepackage.com
license: WYSIWYG v3
Write build.sh:
$ cat > build.sh
# configure, make, and install
configure --prefix=$PREFIX --with-zlib=$PREFIX
make -j${CPU_COUNT}
make install
Write bld.bat:
$ cat > bld.bat
mkdir build
cd build
REM Configure step
cmake -DCMAKE_PREFIX_PATH=%LIBRARY_PREFIX% -DCMAKE_INSTALL_PREFIX=%LIBRARY_PREFIX% -DCMAKE_BUILD_TYPE=Release %SRC_DIR%
REM Build step
devenv SomePackage.sln /Build "%RELEASE_TARGET%"
if errorlevel 1 exit 1
REM Install step
devenv SomePackage.sln /Build "%RELEASE_TARGET%" /Project INSTALL
if errorlevel 1 exit 1
# Switch back to the `ilastik-build-conda` directory
$ cd ../
# Build the package
$ conda build somepackage
3. Upload the package to your anaconda channel.
conda install anaconda-client
# Upload to your personal channel:
anaconda upload /my/miniconda/conda-bld/osx-64/somepackage-1.2.3-0.tar.bz2
# Or to ilastik's anaconda channel:
anaconda upload -u ilastik /my/miniconda/conda-bld/osx-64/somepackage-1.2.3-0.tar.bz2
When writing your own recipes, use gcc provided by conda.
Instead of using your system compiler, all of our C++ packages use the gcc
package provided by conda
itself (or our own variation of it). On Mac, we use LLVM's clang instead to get C++11 features. On Linux, using conda's gcc-4.8 is an easy way to get C++11 support on old OSes, such as our CentOS 5.11 build VM.
To use the gcc package, add these requirements to your meta.yaml
file:
requirements:
build:
- gcc 4.8.5 # [linux]
run:
- libgcc # [linux]
And in build.sh
, make sure you use the right gcc
executable. For example:
export CC=${PREFIX}/bin/gcc
export CXX=${PREFIX}/bin/g++
# conda provides default values of these on Mac OS X,
# but we don't want them when building with gcc
export CFLAGS=""
export CXXFLAGS=""
export LDFLAGS=""
./configure --prefix=${PREFIX} ...etc...
make
make install
# Or, for cmake-based packages:
mkdir build
cd build
cmake .. \
-DCMAKE_C_COMPILER=${PREFIX}/bin/gcc \
-DCMAKE_CXX_COMPILER=${PREFIX}/bin/g++ \
-DCMAKE_INSTALL_PREFIX=${PREFIX} \
...etc...
make
make install
The Anaconda distribution is built on a CentOS 5.11 VM. To build the ilastik stack on that OS, you'll need to install the following:
cmake
,git
,conda
,gcc
- VTK dependencies:
- OpenGL:
yum install mesa-libGL-devel
- X11:
yum groupinstall "X Software Development"
- OpenGL:
- CPLEX (optional)
- Recommended: VirtualBox Guest additions
- Register external package repository "rpmforge"
- Install package
dkms
- In VBox menu, select
Devices
>Insert Guest Additions CD
- From disk image, install Guest Additions from command line
-
General
-
In cases where we provide an alternative build of a package that Continuum already provides, we need to make sure our special channel takes priority over the
defaults
channel used by conda. (Edit: Ideally, we could just use a custom "build string", but due to conda/conda#918, that doesn't work. Instead, we just use a deliberately strange version number in our custom packages, e.g.version: 5.10.1.99
.) -
It would be nice if we built "debug" versions of important packages (e.g. Python, vigra, Qt) and attached them to the
[debug]
conda-build "feature". -
The final binaries produced via
create-Linux-tarball.sh
andcreate-osx-app.sh
are quite large. They could be reduced by excluding unecessary dylibs and stripping the remaining dylibs. Also, directories likeinclude
, etc. should be excluded. -
Mac
-
For unknown reasons, the
py2app
module does not work "out-of-the-box" for this conda build. (The resulting app crashes frequently.) It probably has something to do with our new dependency ongcc-4.8
andlibgcc
. The current version ofcreate-osx-app.sh
uses a hacky workaround for this issue. It would be nice if we could figure out what the real issue is. -
Windows
-
So far, this repo includes no package build scripts for Windows.
-
Generate a final binary package from the built dependencies
-
Should we attempt to track different versions of the MSVC++ std library via a conda "feature"?