/magic_castle

Terraform modules to replicate the HPC user experience in the cloud

Primary LanguageHCLMIT LicenseMIT

Magic Castle

DOI Build Status

The Digital Research Alliance of Canada provides HPC infrastructure and support to every academic research institution in Canada. The Alliance uses CVMFS, a software distribution system developed at CERN, to make its research software stack available on its HPC clusters, and anywhere else with internet access. This enables replication of the user experience outside of The Alliance physical infrastructure.

From these new possibilities emerged an open-source software project named Magic Castle, which aims to recreate the HPC user experience in public clouds. Magic Castle uses the open-source software Terraform and HashiCorp Language (HCL) to define the virtual machines, volumes, and networks that are required to replicate a virtual HPC infrastructure. The infrastructure definition is packaged as a Terraform module that users can customize as they require. After deployment, the user is provided with a complete HPC cluster software environment including a Slurm scheduler, a Globus Endpoint, JupyterHub, LDAP, DNS, and over 3000 research software applications compiled by experts with EasyBuild. Magic Castle is compatible with AWS, Microsoft Azure, Google Cloud, OpenStack, and OVH.

Setup

How Magic Castle Works

This software project integrates multiple parts that come into play at different steps of spawning the cluster. The following list enumerates the steps involved in order for users to better grasp what is happening when they create clusters.

We will refer to the user of Magic Castle as the operator.

  1. After downloading the latest release of the cloud provider of choice and adapting the main configuration file, the operator launches Terraform and accepts the proposed plan.
  2. Terraform communicates with the cloud provider REST API and requests the creation of the virtual machines.
  3. For each virtual machine creation request, Magic Castle provides a cloud-init file. This file is used to initialize the virtual machine base configuration and installs puppet agent. The cloud-init file of the puppet tagged virtual machine installs and configures a Puppet primary server.
  4. Terraform uploads on the Puppet primary server instance a YAML file containing information about the roles of each instances specified as tags.
  5. The Puppet agents communicate with the Puppet primary server to retrieve and apply their configuration based on the tags defined in the preceding YAML file.

Talks, slides and videos

List of other cloud HPC cluster open-source projects

When I think about the DevOps landscape, we have so many people just like chefs in a restaurant that are experimenting with different ways of doing things. Once they get it, then they create those recipes. Those recipes in our world is source code. [...] That's why we will always have duplicates and similar projects, because there's going to be one ingredient that's going to be slightly different to make you preferred over something else

Kelsey Hightower, Sourcegraph Podcast, Episode 16, 2020

Contributing / Customizing

Refer to the reference design and the developer documentation.