Infrastructure as code
Opened this issue · 46 comments
References:
- https://en.wikipedia.org/wiki/Configuration_management
- https://en.wikipedia.org/wiki/Version_control
- https://en.wikipedia.org/wiki/Infrastructure_as_code
- GitOps
I don't think it is possible to have a "dev ops" environment without some kind of Configuration Management Tool.
They make it possible to configure an environment through scripts.
The big advantage of a tool like this is that new virtual machine or container instances can be easily created or updated.
The most common solutions here I think are Ansible, Puppet, Chef and Salt.
It is probably best to pick one tool and have practical scripting exercises how to an enviroment can be configured programmatically.
Hi Göran,
Thanks for your suggestion.
Among Ansible, Puppet, Chef and Salt, what's the one with the best documentation, in particular getting started and tutorials pages to your opinion?
Reading material:
GitOps: A Path to More Self-Service IT
https://cacm.acm.org/magazines/2018/9/230599-gitops/fulltext
"Use Cases for GitOps. DNS is an obvious place to start, as are VM creation, container maintenance and orchestration, firewall rules, website updates, blog posts, email aliases and mailing lists, and just about any virtual infrastructure or one with a configuration file or API."
We've done this with Ansible and Salt in a couple of big projects, only to end up with huge amounts of stuff just to get the actual stuff working.
Terraform
The move to Terraform and Hashicorp Configuration Language (HCL) made life a lot easier https://www.terraform.io/.
We're running Salt extensively on thousands of nodes. It's working great for us and gives us great power in controlling our "standard" environments
I have seen this implemented with puppet, both badly and very well, in different organisations.
How good is your puppet? an empirically defined and validated quality model for puppet
https://pure.tudelft.nl/portal/files/37386939/how_good_is_your_puppet.pdf
Molecule is a tool for testing ansible scripts
https://molecule.readthedocs.io/en/stable/#
Using Testinfra with Ansible to verify server state
https://opensource.com/article/19/5/using-testinfra-ansible-verify-server-state
Python library to create AWS CloudFormation descriptions
https://github.com/cloudtools/troposphere
Pulumi - Infrastructure as Code
https://www.pulumi.com/
The 'as Code' Activities: Development Anti-patterns for Infrastructure as Code
http://arxiv.org/abs/2006.00177
Automating web applications proxying, DNS registration and TLS termination with ansible
Free version of Morris' book on infrastructure as code: https://us-east-1.linodeobjects.com/marketing-assets/Infrastructure_as_Code_2E-ER_Linode.pdf
See accepted papers at CONFLANG, workshop on the design, the theory, the practice and the future evolution of configuration languages.
https://2021.splashcon.org/home/conflang-2021#event-overview
Luke Hoban on Infrastructure as Code IEEE Software
CUE: Configure Unify Execute "Validate, define, and use dynamic and text-based data"
https://cuelang.org/
Dhall is a programmable configuration language that you can think of as: JSON + functions + types + imports
https://dhall-lang.org/
Tool for vulnerability scanning of Infrastructure as Code https://www.checkov.io/
Modus is a language for building Docker/OCI container images, it uses logic programming to express interactions among build parameters, specify complex build workflows, automatically parallelise and cache builds, help to reduce image size, and simplify maintenance.
https://modus-continens.com/
Paper: "Modus: a Datalog dialect for building container images."
Nickel's purpose is to automate the generation of static configuration files - think JSON, YAML, XML, or your favorite data representation language - that are then fed to another system. It is designed to have a simple, well-understood core: it is in essence JSON with functions.
https://nickel-lang.org/
This is relevant to Dhall.
Earthly is a CI/CD framework that allows you to develop pipelines locally and run them anywhere. Earthly leverages containers for the execution of pipelines. This makes them self-contained, repeatable, portable and parallel.
HashiCorp Packer
Packer is a free and open source tool for creating golden images for multiple platforms from a single source configuration.
https://www.packer.io
GitOps: The Evolution of DevOps? (IEEE Software)
Material for the Ansible Up & Running book: https://github.com/ansiblebook
Skaffold handles the workflow for building, pushing and deploying your application, allowing you to focus on what matters most: writing code.
https://skaffold.dev/
Mars is an infrastructure-as-code tool for Ethereum
https://github.com/TrueFiEng/Mars
CircleCI orbs: package management ecosystem for CircleCI configuration
https://circleci.com/docs/orb-intro/
Mining for Cost Awareness in the Infrastructure as Code Artifacts of Cloud-based Applications: an Exploratory Study.
http://arxiv.org/abs/2304.07531
Testing idempotence for infrastructure as code
https://dsg.tuwien.ac.at/team/hummer/docs/2013-middleware-iac.pdf
werf: CD and gitops for Kubernetes
https://werf.io/
tutorial by https://killercoda.com/jarns-zeiher/scenario/Simple-End-to-End-GitOps-with-Werf
OpenTofu lets you declaratively manage your cloud infrastructure.
https://github.com/opentofu/opentofu
Infrastructure-as-Code Ecosystems
https://link.springer.com/chapter/10.1007/978-3-031-36060-2_9
Starlark (formerly known as Skylark) is a language intended for use as a configuration language. It was designed for the Bazel build system, but may be useful for other projects as well.
https://github.com/bazelbuild/starlark
Heavily used in the tensorflow repo
Rego is a general-purpose policy language, which means that it works for any layer of the stack and any domain. The primary purpose of Rego is to accept JSON/YAML inputs and data that are evaluated to make policy-enabled decisions about infrastructure resources, identities, and operations.
Crossplane is a framework for building cloud native control planes without needing to write code.
https://www.crossplane.io/
qq, a query language that supports the main formats for config files
https://github.com/JFryy/qq