felixsteinke/cloud-provider-dataset

CSV Datasets for Cloud Providers of Azure and Google Cloud. Contains regions, datacenter sustainability, machines, used CPUs, and embodied emissions.

Cloud Provider Dataset

Dataset for the Cloud Providers Microsoft Azure and Google Cloud Platform (GCP).

Table of Content

• Raw Datasets
• Azure Web Scraper
• Google Cloud Platform Web Scraper

Installation

Tested on OpenJDK 17.0.2 & Maven 3.6.3.

mvn clean install -DskipTests

Dataset Update

Azure Machines

Extracted from https://learn.microsoft.com/en-us/azure/virtual-machines/linux/compute-benchmark-scores with the responsible Web Scraper.

cd azure-web-scraper
java -jar ./target/exectuable.jar

Azure Regions

• https://www.azurespeed.com/Information/AzureRegions
• azure-data-regions.html

Copy the <table> element of the table on the website and use it in the resources with minimal reformatting.

Website Element Screenshot

Then run the responsible Web Scraper.

cd azure-web-scraper
java -jar ./target/exectuable.jar

Azure Sustainability

• https://datacenters.microsoft.com/globe/fact-sheets/
• azure-data-sustainability.html

Copy the <div> element of the table on the website and use it in the resources with minimal reformatting.

Website Element Screenshot

Then run the responsible Web Scraper.

cd azure-web-scraper
java -jar ./target/exectuable.jar

Hardware Embodied Emissions

Copied from the https://docs.google.com/spreadsheets of https://www.cloudcarbonfootprint.org/docs/embodied-emissions/.

GCP CPUs

Extracted from https://cloud.google.com/compute/docs/cpu-platforms with the responsible Web Scraper.

cd gcp-web-scraper
java -jar ./target/exectuable.jar

GCP Data Center CFE

• https://cloud.google.com/sustainability/region-carbon
• gcp-data-center-cfe.csv

Queried from BigQuery and saved as CSV:

SELECT year,cloud_region,location,zone_id,cfe_region,google_cfe FROM `bigquery-public-data.google_cfe.datacenter_cfe` WHERE year=2021 LIMIT 1000

GCP Data Center PUE

• https://www.google.com/about/datacenters/efficiency/
• gcp-data-centers-pue.csv

Created by hand and matched with the cloud regions. A lot of table entries could not find a match with the regions:

• Lenoir, North Carolina
• Montgomery County, Tennessee
• Jackson County, Alabama
• Papillion, Nebraska
• Mayes County, Oklahoma

GCP Machines

• https://cloud.google.com/compute/docs/machine-resource
• gcp-region-hardware.html

Copy the <table> element of the table on the website and use it in the resources with minimal reformatting.

Website Element Screenshot

Then run the responsible Web Scraper.

cd gcp-web-scraper
java -jar ./target/exectuable.jar

GCP Region Hardware

Extracted from https://cloud.google.com/compute/docs/regions-zones with the responsible Web Scraper.

cd gcp-web-scraper
java -jar ./target/exectuable.jar