/amazon-asdi

Docs and supporting material for the Amazon Sustainability Data Initiative.

Primary LanguageJupyter NotebookApache License 2.0Apache-2.0

Amazon Sustainability Data Initiative

The Amazon Sustainability Data Initiative (ASDI) seeks to accelerate sustainability research and innovation by minimizing the cost and time required to acquire and analyze large sustainability datasets. ASDI supports innovators and researchers with the data, tools, and technical expertise they need to move sustainability to the next level.

This repo contains docs, examples, and supporting material for the Amazon Sustainability Data Initiative (ASDI), an initiative to provide access to large datasets in the cloud to help researchers and innovators address a wide range of sustainability challenges.

Working with data in ASDI

Getting started

The Getting Started with the Amazon Sustainability Data Initiative (ASDI) has information on setuping up an AWS account, relevant AWS services, accessing the datasets, and an overview of AWS basics for working with data.

Finding ASDI datasets

Datasets from ASDI are listed on the Registry of Open Data on AWS's ASDI collaboration page

ASDI workshops and events

Event Description
Code Green 2019 Code Green is both a hackathon and workshop, originally created for re:Invent 2019, centered around exploring Amazon Sustainability Data Initiative (ASDI) datasets.
Code Green 2021 Code Green is both a hackathon and workshop. This version was created for re:Invent 2021, and centers around exploring Amazon Sustainability Data Initiative (ASDI) and AWS Data Exchange (ADX) datasets.

Examples

These are examples of how to set up infrastructure on AWS for working with ASDI datasets.

Example AWS Services ASDI Datasets Description
Simple CMIP6 Amazon SageMaker Notebooks CMIP6 Example of how to filter ensemble members in CMIP6, plot a spatial map, and extract a time series of air temperature
Fargate Dask Cluster AWS Fargate, Amazon SageMaker Notebooks CMIP6, ERA5, GOES-16 Example of how to set up a Dask cluster and Jupyter notebook for scalable analysis of datasets
GFS to Parquet Conversion Amazon SageMaker Notebooks, Amazon Athena, AWS Glue GFS Example of how to subset GFS grib2 files by variable, and convert them to the parquet file format for querying
GHCN Growing Degree Days Amazon Athena, AWS Glue GHCN Example of how to query the NOAA GHCN data using Amazon Athena to return the number of growing degree days
ISD Example Amazon SageMaker Notebooks, Amazon Athena, AWS Glue ISD An example of how to work with the NOAA Integrated Surface Database using Jupyter notebooks
Julia GOES-16 Amazon EC2 GOES-16 An example showing multi-cpu computation against the GOES-16 precipitation files using Julia
OpenAQ Example Amazon Athena, AWS Glue OpenAQ Creates AWS Glue tables for easily querying the OpenAQ data in Amazon Athena

Contributing to ASDI

There are two main ways to contribute to the ASDI Program:

  1. If you have sustainability-related data you would like to share with the Amazon Sustainability Data Initiative (ASDI), please refer to the contribution guidelines on the Registry of Open Data GitHub repository, or contact us at sustainability-data-initiative@amazon.com. You may be able to offset the storage and egress costs of your data by submitting an application to the AWS Open data Program. Check here for guidance on how to optimally share data in the AWS cloud. Data providers are responsible for maintaining and supporting the data that they share.

  2. If you have tutorials, open code, or publications that exemplify how existing ASDI datasets can be used for sustainability related work, you can add them directly to the Registry of Open Data on AWS by clicking the link at the bottom of a dataset page named 'Edit this dataset entry on GitHub.' For example see the link at the bottom of this dataset.

ASDI data for supporting research

AWS Cloud Credits

If you would like to test a cloud-based idea for sustainability, consider requesting a cloud grant to offset the costs of experimentation. The AWS Promotional Credit Program managed by the Amazon Sustainability Data Initiative awards grants of AWS credits to support those interested in prototyping new sustainability related solutions on AWS, or transfer existing workflows from on-premise to the cloud.

AWS Research Cloud Program

The AWS Research Cloud Program was built by scientists at AWS to enable easy use of AWS resources by their fellow researchers in the scientific community around the globe. By signing up you can receive a free copy of the AWS Researcher's Handbook, a comprehensive guide to doing research on AWS.

For more information about the Amazon Sustainability Data Initiative, visit www.AmazonSDI.com.

⚠️ License

This project is licensed under the Apache License 2.0.