StackPulse Steps

This repository is the official library of StackPulse steps.

StackPulse steps are containerized applications that can be composed together to form a playbook. Steps can perform any arbitrary computational task but are most often calls to different services and APIs, allowing the playbook composer to interact with their stack in an automated way. This enables faster and more streamlined handling of alerts and remediation of incidents.

Just like any other container, steps can receive input in the form of run arguments or environment variables and have output directed to stdout and stderr. The standard form for a step to issue its output is as a JSON object as defined in the StackPulse steps Golang SDK.

Development

Build tool

The build tool used by the steps repository is called baur. You must install it on your machine from here. The currently supported version we use is 2.0.

Baur requires PostgreSQL to operate. After building a step once, the step's file hashes will be stored into postgres. Subsequent attempts to build the step will not do anything unless one of the files the step depends on are changed.

Building all steps locally

In order to start postgres docker container on you must run make pg. This will start a docker container on your machine listening on port 15432. Baur is configured to use postgres running on that port to store its data.

You can build all steps by executing make local command. This process takes a while. Once all steps are built you will enjoy faster subsequent builds.

Building a specific step locally

If you want to build a single step you can run baur run step/name, for example baur run redis/get. If the step or its dependencies were changed the step will be rebuilt. The result of a successful build is a docker image with a tag named of the current branch (i.e: us-docker.pkg.dev/stackpulse/public/redis/keys:MyNewStep)

Step directory structure

Step normally belongs to a family, a family (i.e redis) will contain several steps that are related to it. It will normally be grouped by some common service all steps operate on.

Often times, there will be shared code for all the steps. It can either be a base Docker image that all steps inherit from, some common Go code, etc.

./steps/redis
|-- base
|   |-- base.go
|   |-- go.mod
|   `-- go.sum
|-- bigkeys
|   |-- Dockerfile
|   |-- big-keys.py
|   `-- manifest.yaml
|-- client-list
|   |-- Dockerfile
|   |-- go.mod
|   |-- go.sum
|   |-- main.go
|   `-- manifest.yaml

Any change to base will result all sibling steps to be rebuilt. A change to a specific step (get) will rebuild only that particular step.

Creating a new step

Decide to which family the step belongs, does such family already exist? Check the ./steps/ directory
If such directory already exists, review steps currently in that family. Aknowledge the base directory and see what is already implemented there.
Create a your step under ./steps/family/, it should contain a Dockerfile.
Once your step contains Dockerfile run make apps to generate baur application file for the newly created steps, the file created will be called .app.toml
After running the command you should be able to run baur run family/my-new-step to build it.

Adding integration tests to steps

Integration tests are meant to test the step container works as expected. Their purpose is to use the step docker image externally and validate its operation.

Integration tests should be implemented in the same directory of the step.
The test file should be called <step>_integration_test.go.
Integration tests should have a build tag // +build integration.
Integration tests run as a part of the build process and require the step image to be build in order to operate. In order to execute them run baur run <step_name>. Once the step is built the tests will be executed.
It is possible to run integration tests locally by passing the step image name using the STEP_IMAGE environment variable.

Contributing

Pull requests adding new steps and submitting bug fixes are welcome.

Please note that all new steps must use the StackPulse steps Golang SDK and be submitted with a step manifest containing input and output examples.

In the case of bug fixes, where appropriate, please provide an accompanying unit test that validates and verifies the proposed fix.

Troubleshooting

Please feel free to open an issue for any problem or you may encounter while using one of the steps in the repo. We welcome requests and suggestions for new steps as well.

License

Distributed under the BSD-3-Clause License. See LICENSE for more information.

bogdanciobanu/steps