Dynamic On Demand Analysis Service (DODAS) is a Platform as a Service tool built combining several solutions and products developed by the INDIGO-DataCloud H2020 project and now part of the EOSC-hub H2020 Project.
DODAS allows to instantiate on-demand complex infrastructures over any cloud with almost zero effort and with very limited knowledge of the underlying technical details. In particular DODAS provides the end user with all the support to deploy from scratch a variety of solution dedicated (but not limited) to scientific data analysis. For instance, with pre-compiled templates the users can create a K8s cluster and deploy on top of it their preferred Helm charts all in one step. DODAS provides three principal baselines ready to be used and to be possibly extended:
- an HTCondor batch system
- a Spark+Jupyter cluster for interective and big-data analysis
- a Caching on demand system based on XRootD
- Researchers possibly with requirement specific workflows,
- Big Communities, Small groups
- Resource Providers
- Flexible enough to support multiple and diverse use cases Highly Customizable to accommodate needs from diverse communities
- Built on top of modern industry standards
DODAS has been integrated by the Submission Infrastructure of Compact Muon Solenoid CMS, one of the two bigger and general purposes experiments at LHC of CERN, as well as by the Alpha Magnetic Spectrometer AMS-02 computing environment.
DODAS, as a Thematic Services in the context of EOSC-hub project, is financially supported by European Union’s Horizon 2020 research and innovation programme, grant agreement RIA 777536.
You can find a more detailed overview of the stack here
Before starting pleas note that all the DODAS templates uses the helm charts to deploy application on top of Kubernetes. You can find the helm chart defined and documented here. Therefore all applications can be installed also on top of any pre-existing k8s instance with Helm.
In the quick-start guide you will learn to use the basic functionalities and deployments modes of DODAS. As an example you will be guided through the creation of a kubernetes cluster with an instance of Jupyter and Spark.
- Full-fledged HTCondor cluster:
- Spark cluster with JupyterHub Kubespawner (under construction)
- CachingOnDemand (under construction)
- IAM-integrated MINIO S3 instance (under construction)
If you already have a Kubernetes cluster and you want to manage your infrastructures as Kubernetes resources the DODAS Kubernetes operator is what you are looking for.
Please refer to the documentation here for a quick start guide.
If you are interested in package your working helm chart in a template you can find useful this section.
- WN pod Autoscaler based on condor_q
- Cluster autoscaling based on monitoring metrics
- HTCondor integration wiht IAM
- create a branch
- upload your changes
- create a pull request
Thanks!
You will need mkdocs installed on your machine. You can install it with pip:
pip install mkdocs mkdocs-materialTo start a real time rendering of the doc just type:
mkdocs serveThe web page generated will be now update at each change you do on the local folder.
This work is co-funded by the EOSC-hub project (Horizon 2020) under Grant number 777536.
- D. Spiga et al. “DODAS: How to effectively exploit heterogeneous clouds for scientific computations”, PoS(ISGC 2018 & FCDD)024, DOI: https://doi.org/10.22323/1.327.0024
- Using DODAS as deployment manager for smart caching of CMS data management system (ACAT, 2019), D. Spiga et al. Sep.2019
- Exploiting private and commercial clouds to generate on-demand CMS computing facilities with DODAS, https://doi.org/10.1051/epjconf/201921407027
- The DODAS experience with the integration of multiple scientific communities and infrastructures
- DODAS: How to effectively exploit heterogeneous clouds for scientific computations
- Exploiting private and commercial clouds to generate on-demand CMS computing facilities with DODAS
- BoF: HPC, Containers and Big Data Analytics: How can Cloud Computing contribute to the New Challenges
- The AMS and DAMPE computing models and their integration into DODAS
- Training event in the context of SOS18 school
- INFN Training event
- Training course on Batch As a System
- Training course on Big Data Clusters
- Using DODAS as deployment manager for smart caching of CMS data management system
- Dynamic On Demand Analysis Service
- Vacuum model for job execution
- DODAS as no CE solution
- The DODAS Experience on the EGI Federated Cloud
- Dynamic integration of distributed, Cloud-based HPC and HTC resources using JSON Web Tokens and the INDIGO IAM Service
- Talk at K8s WLCG
DODAS Team provides two support channels, email and Slack channel.
- mailing list: send a message to the following list dodas-support@lists.infn.it
- slack channel: join us on Slack Channel