`edlfs`: a fsspec back-end that supports NASA's EDL

Accessing and using NASA's Earth observing data with fsspec, or the software built on top of it like the Pangeo stack, is harder than it should be. This package aims to abstract those complications away, and provide a convenient Python filesystem interface to NASA's Earth observing data.

Challenges

The challenges this pacakge aims to overcome are detailed in our overview document, and briefly restated here:

Most NASA Earth observing dataset require authenticated HTTP access via NASA's Earthdata Login (EDL). However, fsspec does not support EDL/OAuth2 out of the box.
NASA supports different access patterns for cloud-based and on-prem datasets hosted at the various Distributed Active Archive Centers (DAACs), where each DAAC may support only certain access patterns and auth mechanisms.
Handling the above two challenges for large-scale, distributed workflows with tools like Dask adds additional complications.

edlfs is being developed to hide those complications for users so interacting with NASA's Earth observing data, even at global-scale, is straightforward, much like how s3fs hides the complications of working with cloud-data from users.

Usage

Quick start

import edlfs

print(edlfs.__version__)

Installation

In order to easily manage dependencies, we recommend using dedicated project environments via Anaconda/Miniconda or Python virtual environments.

NOTE: edlfs will be available on PyPI and conda-forge with the v0.1.0 release, which is coming soon! Until then, use the Development install.

edlfs can be installed into a conda environment with

conda install -c conda-forge edlfs

or into a virtual environment with

python -m pip install edlfs

Using the Docker image

edlfs provides a docker container image with all the necessary dependencies pre-installed. To get the latest released version:

docker pull ghcr.io/nasa-openscapes/edlfs:latest

a specific release version (>=v0.1.0 only):

docker pull ghcr.io/nasa-openscapes/edlfs:0.1.0

or the current development version:

docker pull ghcr.io/nasa-openscapes/edlfs:test

To run the container and jump into a bash shell inside:

docker run -it --rm ghcr.io/nasa-openscapes/edlfs:latest

To mount your current directory inside the container so that files will be written back to your local machine:

docker run -it -v ${PWD}:/home/conda/work --rm ghcr.io/nasa-openscapes/edlfs:latest
cd work

For more docker run options, see: https://docs.docker.com/engine/reference/run/.

Contact us!

Found a bug? Want to request a feature? Open an issue

General questions? Suggestions? Or anything else? Start a discussion

Don't hesitate to reach out; we would love to hear from you!

Development

To contribute to edlfs, first check out our Code of Conduct and our contributing guide.

Development install

To create a development environment for edlfs, we recommend using conda/mamba to create a development environment. First fork the repo and then:

git clone https://github.com/[OWNER]/edlfs.git
cd edlfs

mamba env update -f environment.yml  # will create if env. doesn't already exist
mamba activate edlfs

python -m pip install -e .

Note: Each time you go to make new changes/create new feature branches, you may want to ensure the environment and install are up-to-date by running:

# from the repository root
mamba env update -f environment.yml
mamba deactivate && mamba activate edlfs
python -m pip install -e .

Collaborators

Feel free to add your name here, or if you want to sign up to be a maintainer, in the package authors.

NASA-Openscapes/edlfs

edlfs: a fsspec back-end that supports NASA's EDL