microns_phase3_nda

nda schema for MICrONS phase3. For more on the MICrONS project please see: MICrONS Explorer

Interactive Environment

Here are some options that provide a great experience:

Cloud-based IDE: (recommended)
- Launch using GitHub Codespaces using the + option which will Create codespace on main in the codebase repository on your fork with default options. For more control, see the ... where you may create New with options....
- Build time for a codespace is ~5m. This is done infrequently and cached for convenience.
- Start time for a codespace is ~30s. This will pull the built codespace from cache when you need it.
- Tip: Each month, GitHub renews a free-tier quota of compute and storage. Typically we run into the storage limits before anything else since Codespaces consume storage while stopped. It is best to delete Codespaces when not actively in use and recreate when needed. We'll soon be creating prebuilds to avoid larger build times. Once any portion of your quota is reached, you will need to wait for it to be reset at the end of your cycle or add billing info to your GitHub account to handle overages.
- Tip: GitHub auto names the codespace but you can rename the codespace so that it is easier to identify later.
Local IDE:
- Ensure you have Git
- Ensure you have Docker
- Ensure you have VSCode
- Install the Dev Containers extension
- git clone the codebase repository and open it in VSCode
- Use the Dev Containers extension to Reopen in Container (More info in the Getting started included with the extension)

You will know your environment has finished loading once you either see a terminal open related to Running postStartCommand with a final message: Done or the README.md is opened in Preview.

Technical documentation

Technical documentation on the functional data can be found here.

Installation Instructions

This package requires access to the functional data. To download the SQL database get started on the microns-nda-access repo here.

This package requires PyTorch. Windows users may need to install from the PyTorch website: PyTorch installation

This package requires the em_coregistration package from the Allen Institute:

pip3 install git+https://github.com/AllenInstitute/em_coregistration.git@phase3

Install this package:

pip3 install git+https://github.com/cajal/microns_phase3_nda.git

Import Instructions

Import datajoint. Configuration instructions: https://docs.datajoint.io/python/setup/01-Install-and-Connect.html

import datajoint as dj

In a jupyter notebook:

from phase3 import nda, func, utils

Using the schema

To view schema ERD:

dj.ERD(nda)

Tutorial:

Using DataJoint to Access Functional Data Tutorial

nda table descriptions

nda.Scan: Information on completed scans. Cajal Pipeline: meso.ScanInfo

nda.ScanInclude: Scans determined suitable for analysis.

nda.Field: Individual fields of scans. Cajal Pipeline: meso.ScanInfo.Field

nda.RawManualPupil: Pupil traces for each scan from the left eye collected at ~20 Hz and semi-automatically segmented.

nda.ManualPupil: Manual pupil traces low-pass filtered with a hamming window to the scan frame rate and linearly interpolated to scan frame times.

nda.RawTreadmill: Cylindrical treadmill rostral-caudal position extracted with a rotary optical encoder at ~100Hz and converted into velocity.

nda.Treadmill: Treadmill velocities low-pass filtered with a hamming window to the scan frame rate then linearly interpolated to scan frame times.

nda.FrameTimes: Timestamps of scan frames in seconds relative to the start of the scan for the first pixel of the first imaging field.

nda.Stimulus: For each scan, contains the movie aligned to activity traces in nda.Activity.

nda.Trial: Contains information for each trial of the movie in nda.Stimulus. There are three types of trials, Clip, Monet2, and Trippy. Each unique trial has its own condition_hash. To get detailed information for each trial stimulus, join each condition_hash according to its corresponding type in one of: nda.Clip, nda.Monet2, or nda.Trippy.

nda.Clip: Detailed information for movie clips.

nda.Monet2: Detailed information for the Monet2 stimulus.

nda.Trippy: Detailed information for the Trippy stimulus.

nda.MeanIntensity: Mean intensity of imaging field over time. Cajal Pipeline: meso.Quality.MeanIntensity

nda.SummaryImages: Reference images of the scan field. Cajal Pipeline: meso.SummaryImages

nda.Stack: High-res anatomical stack information. Cajal Pipeline: stack.CorrectedStack

nda.Registration: Parameters of the affine matrix learned for field registration into the stack. Cajal Pipeline: stack.Registration.Affine

nda.Coregistration: Coregistration transform solutions from the Allen Institute. em_coregistration

nda.Segmentation: CNMF segmentation of a field using CaImAn package (https://github.com/simonsfoundation/CaImAn). It records the masks of all segmented cells. Mask_id's are unique per field. Cajal Pipeline: meso.Segmentation.Mask

nda.Fluorescence: Records the raw fluorescence traces for each segmented mask. Cajal Pipeline: meso.Fluorescence.Trace

nda.ScanUnit: Unit_id assignment that is unique across the entire scan. Includes info about each unit. Cajal Pipeline: meso.ScanSet.Unit / meso.ScanSet.UnitInfo

nda.Activity: Deconvolved spike trace from the fluorescence trace. Cajal Pipeline: meso.Activity.Trace

nda.StackUnit: Unit coordinates in stack reference frame after field registration. np_x, np_y, np_z should be used for transformation to EM space using Coregistration. meso.StackCoordinates.UnitInfo

nda.AreaMembership: Visual area labels for all units.

nda.MaskClassification: Classification of segmented masks into soma or artifact. Uses CaImAn package (https://github.com/simonsfoundation/CaImAn). Cajal Pipeline: meso.MaskClassification.Type

nda.Oracle: Leave-one-out correlation for repeated videos in stimulus.

For more documentation see: Cajal Pipeline Documentation

Acknowledgement of Government Sponsorship

Supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior / Interior Business Center (DoI/IBC) contract number D16PC00003. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/IBC, or the U.S. Government.

yambottle/microns_phase3_nda