PaCRD (a combination of "Pipelines as Code" and "Custom Resource Definition") is a Kubernetes controller that manages the lifecycle of SpinnakerTM applications and pipelines as objects within your cluster. PaCRD extends Kubernetes functionality to support Spinnaker Application and Pipeline objects that can be observed for changes through a mature lifecycle management API.
With PaCRD you can:
- Maintain your Spinnaker pipelines as code with the rest of your Kubernetes manifests.
- Persist Pipeline and Application changes with confidence to your Spinnaker cluster.
- Leverage existing tools like Helm and Kustomize to template your pipelines across teams and projects.
To get started right away, check out the Quick Start section for installation instructions.
To use PaCRD, make sure you meet the following requirements:
- Have a working Kubernetes 1.11+ cluster
- Have a working Spinnaker installation
- Although there is no minimum version required for this experiment, Armory recommends using the latest release
- Have permissions to install CRDs, create RBAC roles, and create service accounts
Download the current pacrd manifest to your local machine:
curl -fsSL https://engineering.armory.io/manifests/pacrd-1.0.1.yaml > pacrd-1.0.1.yamlThen, inspect the manifest to make sure it is compatible with your cluster.
Create the following files in the directory where you downloaded the pacrd
manifest to customize the installation: kustomization.yaml and patch.yaml.
Start by creating a kustomization.yaml file, which contains
the installation settings:
# file: kustomization.yaml
resources:
- pacrd-1.0.1.yaml
patchesStrategicMerge:
- patch.yaml
namespace: spinnaker # Note: you should change this value if you are _not_ deploying into the `spinnaker` namespace.Next, create a patch.yaml file that contains your pacrd config. If you are
not deploying into the spinnaker namespace, update the front50 and orca
keys:
# file: patch.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: pacrd-config
namespace: spinnaker
data:
pacrd.yaml: |
spinnakerServices:
# NOTE: change `spinnaker` to your namespace name here
front50: http://spin-front50.spinnaker:8080
orca: http://spin-orca.spinnaker:8083
# OPTIONAL: uncomment the next line to configure a Fiat service account, it should be at the same level that spinnakerServices.
# fiatServiceAccount: my-service-accountWhen you are ready, apply the pacrd manifest to your cluster:
# If using `kubectl` >= 1.14
kubectl apply -k .
# Otherwise, use `kustomize` and `kubectl` toegether
kustomize build | kubectl apply -f -Once you have PaCRD installed and running in your cluster, you can define your applications and pipelines. Then apply them to the cluster.
While this product is in an [Experimental]({{< ref "release-definitions" >}}) state, kind objects for PaCRD
live under the pacrd.armory.spinnaker.io/v1alpha1 version moniker.
In Spinnaker, an Application is a logical construct that allows you to group resources under a single name. You can read more about applications in the Spinnaker docs.
For available Application configuration options check out the {{< linkWithTitle "pacrd-crd-docs.md" >}}.
In Kubernetes, define your application in an application.yaml file. The configuration fields are the same as what you see when you create an application using the Spinnaker UI. The following example defines an application named "myapplicationname".
Note: Application names must adhere to both Kubernetes and Spinnaker name standards.
#file: application.yaml
apiVersion: pacrd.armory.spinnaker.io/v1alpha1
kind: Application
metadata:
name: pacrd-pipeline-stages-samples
spec:
email: test@armory.io
description: DescriptionCreate the application in your cluster by running:
kubectl apply -f application.yamlCheck on the status of your application by using either the get or describe
commands. kubectl recognizes either app or application for the resource
kind:
kubectl get app myapplicationname
# or kubectl get application myapplicationnameThe command returns information similar to the this:
NAME URL LASTCONFIGURED STATUS
myapplicationname http://spinnaker.io/#/applications/myapplicationname/clusters 7m26s Created
You can update in one of two ways:
- Reapply the application manifest in your repository
kubectl apply -f application.yaml
- Edit the application manifest in-cluster
kubectl edit app myapplicationname
When you update your application in Kubernetes, the changes propagate into
Spinnaker. If an error occurs during the update, your application may show an
ErrorFailedUpdate state. You can see the details of that failure by describing
the resource and looking in the "Events" section:
kubectl describe app myapplicationnameYou can delete an application in one of two ways:
- Reapply the application manifest in your repository
kubectl delete -f application.yaml
- Delete the application directly
kubectl delete app myapplicationname
When you delete your application in Kubernetes, the deletion propagates into
Spinnaker. If an error occurs during deletion, your application may show an
ErrorFailedDelete state. You can see the details of that failure by describing
the resource and looking in the "Events section":
kubectl describe app myapplicationnamePipelines allow you to encode the process that your team follows to take a service from commit to a desired environment, such as production. You can read more in the Spinnaker Pipelines guide.
View Pipeline configuration options in the {{< linkWithTitle "pacrd-crd-docs.md" >}}.
In Kubernetes, define your pipeline in a pipeline.yaml file. The configuration fields are the same as what you see when you create a pipeline using the Spinnaker UI. The following example defines a simple pipeline named "myapplicationpipeline", which bakes a manifest and prompts for a manual judgment.
Pipeline names should follow the Kubernetes Object Names and IDs naming conventions.
This example assumes that you've created the myapplicationname
application from the previous section. Create one before
proceeding if you have not done so already.
# file: deploy-nginx.yaml
apiVersion: pacrd.armory.spinnaker.io/v1alpha1
kind: Pipeline
metadata:
name: pacrd-deploymanifest-integration-samples
spec:
description: A sample showing how to define artifacts.
application: &app-name pacrd-pipeline-stages-samples
stages:
- type: deployManifest
properties:
name: Deploy text manifest
refId: "1"
requisiteStageRefIds: [ ]
account: spinnaker
cloudProvider: kubernetes
moniker:
app: *app-name
skipExpressionEvaluation: true
source: text
manifests:
- |
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80Create your pipeline in your cluster:
kubectl apply -f pipeline.yamlCheck on the status of your pipeline by using either the get or describe
commands. kubectl will recognize either pipe or pipeline for the resource
kind:
kubectl get pipe myapplicationpipeline
# or ... kubectl get pipeline myapplicationpipelineThe command returns information similar to the this:
NAME STATUS LASTCONFIGURED URL
myapplicationpipeline Updated 5s http://spinnaker.company.com/#/applications/myapplicationname/executions/configure/f1eb82ce-5a8f-4b7a-9976-38e4aa022702
A describe call can give you additional contextual information about the
status of your pipeline:
kubectl describe pipeline myapplicationpipelineThe command returns information similar to the this::
Name: myapplicationpipeline
API Version: pacrd.armory.spinnaker.io/v1alpha1
Kind: Pipeline
Metadata:
# omitted for brevity
Spec:
# omitted for brevity
Status:
Id: f1eb82ce-5a8f-4b7a-9976-38e4aa022702
Last Configured: 2020-03-09T15:55:27Z
Phase: Updated
URL: http://localhost:9000/#/applications/myapplicationname/executions/configure/f1eb82ce-5a8f-4b7a-9976-38e4aa022702
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Updated 94s pipelines Pipeline successfully created in Spinnaker.
Warning ErrorUpdatingPipeline 93s pipelines Bad Request: The provided id f1eb82ce-5a8f-4b7a-9976-38e4aa022702 doesn't match the pipeline id null
Normal Updated 91s (x2 over 91s) pipelines Pipeline successfully updated in Spinnaker.
You can update a pipeline in one of two ways:
- Reapply the pipeline manifest in your repository
kubectl apply -f pipeline.yaml
- Edit the pipeline manifest in-cluster
kubectl edit pipeline myapplicationpipeline
When you update your pipeline in Kubernetes, the changes propagate into
Spinnaker. If an error occurs during the update, your pipeline may show an
ErrorFailedUpdate state. You can see the details of that failure by describing
the resource and looking in the "Events" section:
kubectl describe pipeline myapplicationpipelineYou can delete a pipeline in one of two ways:
- Delete the pipeline manifest from your repository definition
kubectl delete -f pipeline.yaml
- Delete the pipeline directly
kubectl delete pipeline myapplicationpipeline
When you delete your pipeline in Kubernetes, the deletion propagates into
Spinnaker. If an error occurred during deletion, then your pipeline may show an
ErrorFailedDelete state. You can see the details of that failure by describing
the resource and looking in the "Events section":
kubectl describe pipeline myapplicationpipelineAn artifact is an object that references an external resource. Examples include a Docker container, a file in source control, an AMI, or a binary blob in S3. Artifacts in PaCRD come in two types:
- Definitions contain all necessary information to locate an artifact.
- References contain enough information to find a Definition.
Define your pipeline artifacts in a section called expected artifacts. The
following example defines a single container image that the pipeline expects as
an input to the BakeManifest stage:
apiVersion: pacrd.armory.spinnaker.io/v1alpha1
kind: Pipeline
metadata:
name: my-pipeline
spec:
description: A sample showing how to define artifacts.
application: my-application
expectedArtifacts:
- id: &image-id my-application-docker-image
displayName: *image-id
matchArtifact:
type: docker/image
properties:
name: my-organization/my-container
artifactAccount: docker-registry
stages:
- type: bakeManifest
properties:
name: Bake Application
refId: "1"
outputName: myManifest
templateRenderer: helm2
inputArtifacts:
- id: *image-idEach matchArtifact block contains:
type: required; the artifact classification; see the Types of Artifacts section in the Spinnaker documentation for supported typesproperties: dictionary of key-value pairs appropriate for that artifact
PaCRD only validates officially supported artifacts. PaCRD does not validate custom artifacts or artifacts defined via Plugins.
Reference artifacts in the inputArtifacts section of a pipeline stage. You
can use either the artifact id or displayName. If you are new to using
artifacts, you can use the displayName value, which is most often what appears
when the Spinnaker UI displays your pipeline.
The following example defines two artifacts in the expectedArtifacts section. Each artifact is then referenced in the inputArtifacts section of the bakeManifest stage. The first is declared with id and the second with displayName.
apiVersion: pacrd.armory.spinnaker.io/v1alpha1
kind: Pipeline
metadata:
name: my-pipeline
spec:
description: A sample showing how to reference artifacts.
application: my-application
expectedArtifacts:
- id: first-inline-artifact-id
displayName: My First Inline Artifact Id
matchArtifact:
type: embedded/base64
properties:
name: my-inline-artifact
- id: second-inline-artifact-id
displayName: My Second Inline Artifact
matchArtifact:
type: embedded/base64
properties:
name: my-second-inline-artifact
stages:
- type: bakeManifest
properties:
name: Bake Application
refId: "1"
outputName: myManifest
templateRenderer: helm2
inputArtifacts:
- id: first-inline-artifact-id
- displayName: My Second Inline ArtifactPaCRD validates that the inputArtifacts referenced in the bakeManifest stage
correspond to exactly one artifact declared in the expectedArtifacts section
of the CRD.
PaCRD throws a PipelineValidationFailed error when it can't find an input
artifact in the list of expected artifacts. You can see which input artifact
failed validation by executing a describe call against the pipeline under
creation. If you use the above example but replace the id reference with
a-nonsense-value, pipeline validation fails.
Execute kubectl describe:
kubectl describe pipeline my-pipelineExpected output displays which input artifact failed validation:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Updated 2m53s (x2 over 2m54s) pipelines Pipeline successfully updated in Spinnaker.
Warning PipelineValidationFailed 0s (x4 over 3s) pipelines artifact with id "a-nonsense-value" and name "" could not be found for this pipeline
If you want to monitor how PaCRD is used and the errors encountered, you can enable New Relic integration. You can either send this data to your New Relic account or to Armory's New Relic account. If you choose to share the data with Armory, it helps us improve the product and provide better support. For information about what data is sent and how it is used, contact Armory, or simply enable it with your New Relic account first.
To enable this integration, add the New Relic license to the patch.yaml file
as shown below. If you send it to Armory's New Relic account, we will give you
a license to use, otherwise use your New Relic account's license.
# file: patch.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: pacrd-config
namespace: spinnaker
data:
pacrd.yaml: |
spinnakerServices:
# NOTE: change `spinnaker` to your namespace name here
front50: https://spin-front50.spinnaker:8080
orca: https://spin-orca.spinnaker:8083
# fiatServiceAccount: <fiatServiceAccount>
newRelicLicense: <newRelicLicense>Since reconciliation happens multiple times per minute, Armory sends metrics only during the first three minutes of each hour. Error messages contain obfuscated URLs, application names, and pipeline names.
By default the application name will be pacrd, if you want to change this you
can add NewRelicAppName property at the same level of newRelicLicense and
add your own custom application name.
Here's an example of error stack traces:
-
Deleting an application in Kubernetes triggers the following behavior:
- Delete the application in Kubernetes.
- Delete the application in Spinnaker.
- Delete pipelines associated with the application in Spinnaker only.
- Pipeline stages must be defined with a
typekey for the stage name and a key of the same name where all stage options live. For example, for the "Bake Manifest" stage you would structure your definition like this:
# ...
stages:
- type: BakeManifest
bakeManifest:
name: Bake the Bread
# ...
# ...- Documentation for available Application spec fields must be
found in the installation manifest for this controller. You can do so by
grepping for
applications.pacrd.armory.spinnaker.ioin the installation manifest. Fields are documented underspec.validation.openAPIV3Schema.
- Documentation for available Pipeline spec fields must be
found in the installation manifest for this controller. You can do so by
grepping for
pipelines.pacrd.armory.spinnaker.ioin the installation manifest. Fields are documented underspec.validation.openAPIV3Schema.
Please see the CONTRIBUTING.md doc.