HybridContents

The HybridContentManager was originally created by Quantopian as part of pgcontents; however, the usage of HybridContentsManager was restricted to the compatibility requirements of pgcontents. These restrictions included postgres dependencies and no support for the latest notebook version (>6).

At Viaduct we used pgcontents exclusively for the HybridContentsManager and wanted to extend its functionality, so we created this fork hybridcontents.

See related pgcontents issues:

Getting Started

Prerequisites:

A Python installation with Jupyter Notebook >= 4.0.

Installation:

pip

pip install hybridcontents

Anaconda

conda install -c viaduct hybridcontents

conda-forge

See instructions here

Featues

Mix and match different content managers for different directories
Easily move files between different content managers (i.e local files to s3 backed manager)
Path validation to keep consistent naming scheme and/or prevent illegal characters

Usage

For a detailed example see, hybrid_manager_example.py

The following code snippet creates a HybridContentsManager with two directories with different content managers.

c = get_config()

c.NotebookApp.contents_manager_class = HybridContentsManager

c.HybridContentsManager.manager_classes = {
    # NOTE: LargFileManager only exists in notebook > 5
    # If using notebook < 5, use FileContentManager instead
    "": LargeFileManager,
    "shared": S3ContentsManager
}

# Each item will be passed to the constructor of the appropriate content manager.
c.HybridContentsManager.manager_kwargs = {
    # Args for root LargeFileManager
    "": {
        "root_dir": read_only_dir
    },
    # Args for the shared S3ContentsManager directory
    "shared": {
        "access_key_id": ...,
        "secret_access_key": ...,
        "endpoint_url":  ...,
        "bucket": ...,
        "prefix": ...
    },
}

def only_allow_notebooks(path):
  return path.endswith('.ipynb')

# Only allow notebook files to be stored in S3
c.HybridContentsManager.path_validators = {
    "shared": only_allow_notebooks
}

Testing

To run unit tests,

tox

This will run all unit tests for python versions 2.7, 3.6, 3.7 and jupyter notebook versions 4, 5, and 6.

Publishing a Release

Create a new release on Github
Update the version in setup.py
Run ./scripts/pip_publish.sh
Update the version meta.yaml
Update the sha256 in meta.yaml
Run ./scripts/anaconda_publish.sh
Update on Conda Forge

RamakrishnaChilaka/hybridcontents