/dodas-apps

DODAS Applications

Primary LanguageShell

Dynamic On Demand Analysis Service

What's DODAS

Dynamic On Demand Analysis Service (DODAS) is a Platform as a Service tool built combining several solutions and products developed by the INDIGO-DataCloud H2020 project and now part of the EOSC-hub H2020 Project.

What you can do with DODAS?

DODAS allows to instantiate on-demand complex infrastructures over any cloud with almost zero effort and with very limited knowledge of the underlying technical details. In particular DODAS provides the end user with all the support to deploy from scratch a variety of solution dedicated (but not limited) to scientific data analysis. For instance, with pre-compiled templates the users can create a K8s cluster and deploy on top of it their preferred Helm charts all in one step. DODAS provides three principal baselines ready to be used and to be possibly extended:

  • an HTCondor batch system
  • a Spark+Jupyter cluster for interective and big-data analysis
  • a Caching on demand system based on XRootD

DODAS targets multiple users:

  • Researchers possibly with requirement specific workflows,
  • Big Communities, Small groups
  • Resource Providers

Designed to be experiment agnostic

  • Flexible enough to support multiple and diverse use cases Highly Customizable to accommodate needs from diverse communities
  • Built on top of modern industry standards

Commuties are already adopting DODAS

DODAS has been integrated by the Submission Infrastructure of Compact Muon Solenoid CMS, one of the two bigger and general purposes experiments at LHC of CERN, as well as by the Alpha Magnetic Spectrometer AMS-02 computing environment.

DODAS schema

DODAS, as a Thematic Services in the context of EOSC-hub project, is financially supported by European Union’s Horizon 2020 research and innovation programme, grant agreement RIA 777536.

You can find a more detailed overview of the stack here

Quick start

Before starting pleas note that all the DODAS templates uses the helm charts to deploy application on top of Kubernetes. You can find the helm chart defined and documented here. Therefore all applications can be installed also on top of any pre-existing k8s instance with Helm.

In the quick-start guide you will learn to use the basic functionalities and deployments modes of DODAS. As an example you will be guided through the creation of a kubernetes cluster with an instance of Jupyter and Spark.

Supported apps

Developers guide

Preview feature: DODAS Kubernetes operator

If you already have a Kubernetes cluster and you want to manage your infrastructures as Kubernetes resources the DODAS Kubernetes operator is what you are looking for.

Please refer to the documentation here for a quick start guide.

From HELM to template

If you are interested in package your working helm chart in a template you can find useful this section.

Roadmap

  • WN pod Autoscaler based on condor_q
  • Cluster autoscaling based on monitoring metrics
  • HTCondor integration wiht IAM

DODAS adopters

Contributing

  1. create a branch
  2. upload your changes
  3. create a pull request

Thanks!

Render the page using Mkdocs

You will need mkdocs installed on your machine. You can install it with pip:

pip install mkdocs mkdocs-material

To start a real time rendering of the doc just type:

mkdocs serve

The web page generated will be now update at each change you do on the local folder.

Acknowledgement

This work is co-funded by the EOSC-hub project (Horizon 2020) under Grant number 777536.

Publications and presentations

Papers

  • D. Spiga et al. “DODAS: How to effectively exploit heterogeneous clouds for scientific computations”, PoS(ISGC 2018 & FCDD)024, DOI: https://doi.org/10.22323/1.327.0024
  • Using DODAS as deployment manager for smart caching of CMS data management system (ACAT, 2019), D. Spiga et al. Sep.2019
  • Exploiting private and commercial clouds to generate on-demand CMS computing facilities with DODAS, https://doi.org/10.1051/epjconf/201921407027

Training and talks

Contact us

DODAS Team provides two support channels, email and Slack channel.