The CaImAn team is hiring! We're looking for a data scientist/software engineer with a strong research component. For more information please follow this link.
A Python toolbox for large scale Calcium Imaging data Analysis and behavioral analysis.
CaImAn implements a set of essential methods required in the analysis pipeline of large scale calcium imaging data. Fast and scalable algorithms are implemented for motion correction, source extraction, spike deconvolution, and component registration across multiple days. It is suitable for both two-photon and one-photon fluorescence microscopy data, and can be run in both batch and online modes. CaImAn also contains some routines for the analysis of behavior from video cameras. A list of features as well as relevant references can be found here.
A paper explaining most of the implementation details and benchmarking can be found here.
@article{giovannucci2019caiman,
title={CaImAn: An open source tool for scalable Calcium Imaging data Analysis},
author={Giovannucci, Andrea and Friedrich, Johannes and Gunn, Pat and Kalfon, Jeremie and Brown, Brandon L and Koay, Sue Ann and Taxidis, Jiannis and Najafi, Farzaneh and Gauthier, Jeffrey L and Zhou, Pengcheng and Khakh, Baljit S and Tank, David W and Chklovskii, Dmitri B and Pnevmatikakis, Eftychios A},
journal={eLife},
volume={8},
pages={e38173},
year={2019},
publisher={eLife Sciences Publications Limited}
}
All the results and figures of the paper can be regenerated using this package. For more information visit this page.
Beginning in August 2019 we have an experimental binary release of the software in the conda-forge package repos. This is intended for people who can use CaImAn as a library, interacting with it as the demos do. It also does not need a compiler. It is not suitable for people intending to change the CaImAn codebase. Comfort with conda is still required. If you wish to use the binary package, you do not need the sources (including this repo) at all. With a functional install of conda, you can do (on any platform):
conda create -n caiman -c conda-forge caimanYou will still need to use caimanmanager.py afterwards to create a data directory. If you install this way, do not follow any of the other install instructions below.
You can now use the save method included in both the CNMF and OnACID objects to export the results (and parameters used) of your analysis. The results are saved in an HDF5 file that you can then load in a graphical user interface for more inspection. The GUI will allow you to inspect the results and modify the selected components based on the various quality metrics. For more information click here
The Neurodata Without Borders (NWB) file format is now supported by CaImAn. You read and analyze NWB files and can save the results of the analysis (Estimates object) back to the original NWB file. Consult this demo for an example on how to use this feature.
To use CaImAn with these additional features you'll need to create a new environment following the usual instructions.
To circumvent a problem arising in Windows installation, we recently removed keras from the list of dependencies. Keras was being used to deploy the pretrained neural network models for component screening. The neural network models are being deployed through tensorflow and for that we included tensorflow compatible versions of the models inside the folder model. Existing users who already have keras in their environment will continue using keras as it is slightly faster. However if you create an environment without it you may want to either reinstall caimanmanager or simply copy the files model/*.pb into the folder caiman_data/model/ for the files to be discoverable. New CaImAn users do not need to do anything as this is being taken care of during the installation process.
Installation and updating instructions can be found here.
Documentation of the code can be found here. Moreover, our wiki page covers some aspects of the code.
- Installation on Linux (Windows and MacOS are problematic with anaconda at the moment)
- create a new environment (suggested for safety) and follow the instructions for the calcium imaging installation
- Install spams, as explained here. Installation is not straightforward and it might take some trials to get it right
-
Notebooks: The notebooks provide a simple and friendly way to get into CaImAn and understand its main characteristics. They are located in the
demos/notebooks. To launch one of the jupyter notebooks:source activate CaImAn jupyter notebook --NotebookApp.iopub_data_rate_limit=1.0e10and select the notebook from within Jupyter's browser. The argument
--NotebookApp.iopub_data_rate_limit=1.0e10will prevent any memory issues while plotting on a notebook. -
demo files are also found in the demos/general subfolder. We suggest trying demo_pipeline.py first as it contains most of the tasks required by calcium imaging. For behavior use demo_behavior.py
-
If you modify the demos to use them for your own data it is recommended that you save them in a different file to avoid file conflicts during updating.
-
If you want to directly launch the python files, your python console still must be in the CaImAn directory.
- All diffs must be tested before asking for a pull request. Call
python caimanmanager.py testfrom outside of your CaImAn folder to look for errors (you need to pass the path to the caimanmanager.py file).
- Eftychios A. Pnevmatikakis, Flatiron Institute, Simons Foundation
- Andrea Giovannucci, University of North Carolina, Chapel Hill
- Johannes Friedrich, Flatiron Institute, Simons Foundation
- Pat Gunn, Flatiron Institute, Simons Foundation
A complete list of contributors can be found here.
The following references provide the theoretical background and original code for the included methods.
If you use this code please cite the corresponding papers where original methods appeared (see References below), as well as:
[1] Giovannucci A., Friedrich J., Gunn P., Kalfon J., Brown, B., Koay S.A., Taxidis J., Najafi F., Gauthier J.L., Zhou P., Baljit, K.S., Tank D.W., Chklovskii D.B., Pnevmatikakis E.A. (2019). CaImAn: An open source tool for scalable Calcium Imaging data Analysis. eLife 8, e38173. [paper]
[2] Pnevmatikakis, E.A., Soudry, D., Gao, Y., Machado, T., Merel, J., ... & Paninski, L. (2016). Simultaneous denoising, deconvolution, and demixing of calcium imaging data. Neuron 89(2):285-299, [paper], [Github repository].
[3] Pnevmatikakis, E.A., Gao, Y., Soudry, D., Pfau, D., Lacefield, C., ... & Paninski, L. (2014). A structured matrix factorization framework for large scale calcium imaging data analysis. arXiv preprint arXiv:1409.2903. [paper].
[4] Zhou, P., Resendez, S. L., Stuber, G. D., Kass, R. E., & Paninski, L. (2016). Efficient and accurate extraction of in vivo calcium signals from microendoscopic video data. arXiv preprint arXiv:1605.07266. [paper], [Github repository].
[5] Friedrich J. and Paninski L. Fast active set methods for online spike inference from calcium imaging. NIPS, 29:1984-1992, 2016. [paper], [Github repository].
[6] Giovannucci, A., Friedrich J., Kaufman M., Churchland A., Chklovskii D., Paninski L., & Pnevmatikakis E.A. (2017). OnACID: Online analysis of calcium imaging data in real data. NIPS 2017, pp. 2378-2388. [paper]
[7] Pnevmatikakis, E.A., and Giovannucci A. (2017). NoRMCorre: An online algorithm for piecewise rigid motion correction of calcium imaging data. Journal of Neuroscience Methods, 291:83-92 [paper], [Github repository].
[8] Giovannucci, A., Pnevmatikakis, E. A., Deverett, B., Pereira, T., Fondriest, J., Brady, M. J., ... & Masip, D. (2017). Automated gesture tracking in head-fixed mice. Journal of Neuroscience Methods, 300:184-195. [paper].
[9] Tepper, M., Giovannucci, A., and Pnevmatikakis, E (2018). Anscombe meets Hough: Noise variance stabilization via parametric model estimation. In ICASSP, 2018. [paper]. [Github repository]
- Running CaImAn on a Cluster
- More detailed install instructions for Windows
- Install quirks on some Linux Distributions
- How CaImAn can use your GPUs
- The CaImAn GUI
The implementation of this package is developed in parallel with a MATLAB toobox, which can be found here.
Some tools that are currently available in Matlab but have not been ported to CaImAn are
A list of dependencies can be found in the environment file.
Please use the gitter chat room for questions and comments and create an issue for any bugs you might encounter.
Special thanks to the following people for letting us use their datasets for our various demo files:
- Weijian Yang, Darcy Peterka, Rafael Yuste, Columbia University
- Sue Ann Koay, David Tank, Princeton University
- Manolis Froudarakis, Jake Reimers, Andreas Tolias, Baylor College of Medicine
- Clay Lacefield, Randy Bruno, Columbia University
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.