Manage Datadog Monitors / Dashboards / Slos as code
- DRY, searchable, audited, documented
- Changes are PR reviewed and applied on merge
- Updating shows diff before applying
- Automated import of existing resources
- Resources are grouped into projects that belong to teams and inherit tags
- No copy-pasting of ids to create new resources
- Automated cleanup when removing code
- Helpers for automating common tasks
# teams/foo.rb
module Teams
class Foo < Kennel::Models::Team
defaults(mention: -> { "@slack-my-team" })
end
end
# projects/bar.rb
class Bar < Kennel::Models::Project
defaults(
team: -> { Teams::Foo.new }, # use mention and tags from the team
parts: -> {
[
Kennel::Models::Monitor.new(
self, # the current project
type: -> { "query alert" },
kennel_id: -> { "load-too-high" }, # pick a unique name
name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
message: -> {
<<~TEXT
This is bad!
#{super()} # inserts mention from team
TEXT
},
query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" },
critical: -> { 20 }
)
]
}
)
end
- create a new private
kennel
repo for your organization (do not fork this repo) - use the template folder as starting point:
git clone git@github.com:your-org/kennel.git git clone git@github.com:grosser/kennel.git seed mv seed/template/* kennel/ cd kennel && git add . && git commit -m 'initial'
- add a basic projects and teams so others can copy-paste to get started
- setup CI build for your repo (travis and Github Actions supported)
- uncomment
.travis.yml
section for datadog updates on merge (TODO: example setup for Github Actions) - follow
Setup
in your repos Readme.md
projects/
monitors/dashboards/etc scoped by projectteams/
team definitionsparts/
monitors/dashboards/etc that are used by multiple projectsgenerated/
projects as json, to show current state and proposed changes in PRs
mention
is used for all team monitors viasuper()
renotify_interval
is used for all team monitors (defaults to0
/ off)tags
is used for all team monitors/dashboards (defaults toteam:<team-name>
)
# teams/my_team.rb
module Teams
class MyTeam < Kennel::Models::Team
defaults(
mention: -> { "@slack-my-team" }
)
end
end
- use datadog monitor UI to create a monitor
- see below
- use datadog monitor UI to find a monitor
- get the
id
from the url - run
URL='https://app.datadoghq.com/monitors/123' bundle exec rake kennel:import
and copy the output - import task also works with SLO alerts, e.g.
URL='https://app.datadoghq.com/slo/edit/123abc456def123/alerts/789' bundle exec rake kennel:import
- find or create a project in
projects/
- add the monitor to
parts: [
list, for example:
# projects/my_project.rb
class MyProject < Kennel::Models::Project
defaults(
team: -> { Teams::MyTeam.new }, # use existing team or create new one in teams/
parts: -> {
[
Kennel::Models::Monitor.new(
self,
id: -> { 123456 }, # id from datadog url, not necessary when creating a new monitor
type: -> { "query alert" },
kennel_id: -> { "load-too-high" }, # make up a unique name
name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
message: -> {
# Explain what behavior to expect and how to fix the cause
# Use #{super()} to add team notifications.
<<~TEXT
Foobar will be slow and that could cause Barfoo to go down.
Add capacity or debug why it is suddenly slow.
#{super()}
TEXT
},
query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" }, # replace actual value with #{critical} to keep them in sync
critical: -> { 20 }
)
]
}
)
end
- run
PROJECT=my_project bundle exec rake plan
, an Update to the existing monitor should be shown (not Create / Delete) - alternatively:
bundle exec rake generate
to only locally update the generatedjson
files - review changes then
git commit
- make a PR ... get reviewed ... merge
- datadog is updated by CI
- go to datadog dashboard UI and click on New Dashboard to create a dashboard
- see below
- go to datadog dashboard UI and click on New Dashboard to find a dashboard
- get the
id
from the url - run
URL='https://app.datadoghq.com/dashboard/bet-foo-bar' bundle exec rake kennel:import
and copy the output - find or create a project in
projects/
- add a dashboard to
parts: [
list, for example:
class MyProject < Kennel::Models::Project
defaults(
team: -> { Teams::MyTeam.new }, # use existing team or create new one in teams/
parts: -> {
[
Kennel::Models::Dashboard.new(
self,
id: -> { "abc-def-ghi" }, # id from datadog url, not needed when creating a new dashboard
title: -> { "My Dashboard" },
description: -> { "Overview of foobar" },
template_variables: -> { ["environment"] }, # see https://docs.datadoghq.com/api/?lang=ruby#timeboards
kennel_id: -> { "overview-dashboard" }, # make up a unique name
layout_type: -> { "ordered" },
definitions: -> {
[ # An array or arrays, each one is a graph in the dashboard, alternatively a hash for finer control
[
# title, viz, type, query, edit an existing graph and see the json definition
"Graph name", "timeseries", "area", "sum:mystats.foobar{$environment}"
],
[
# queries can be an Array as well, this will generate multiple requests
# for a single graph
"Graph name", "timeseries", "area", ["sum:mystats.foobar{$environment}", "sum:mystats.success{$environment}"],
# add events too ...
events: [{q: "tags:foobar,deploy", tags_execution: "and"}]
]
]
}
)
]
}
)
end
Setting id
makes kennel take over a manually created datadog resource.
When manually creating to import, it is best to remove the id
and delete the manually created resource.
When an id
is set and the original resource is deleted, kennel will fail to update,
removing the id
will cause kennel to create a new resource in datadog.
Some validations might be too strict for your usecase or just wrong, please open an issue and
to unblock use the validate: -> { false }
option.
Link resources with their kennel_id in the format project kennel_id
+ :
+ resource kennel_id
,
this should be used to create dependent resources like monitor + slos,
so they can be created in a single update and can be re-created if any of them is deleted.
Resource | Type | Syntax |
---|---|---|
Dashboard | uptime | monitor: {id: "foo:bar"} |
Dashboard | alert_graph | alert_id: "foo:bar" |
Dashboard | slo | slo_id: "foo:bar" |
Monitor | composite | query: -> { "%{foo:bar} && %{foo:baz}" } |
Monitor | slo alert | query: -> { "error_budget(\"%{foo:bar}\").over(\"7d\") > 123.0" } |
Slo | monitor | monitor_ids: -> ["foo:bar"] |
- rebase on updated
master
to not undo other changes - figure out project name by converting the class name to snake-case
- run
PROJECT=foo bundle exec rake kennel:update_datadog
to test changes for a single project (monitors: remove mentions while debugging to avoid alert spam)
Add to parts/<folder>
.
module Monitors
class LoadTooHigh < Kennel::Models::Monitor
defaults(
name: -> { "#{project.name} load too high" },
message: -> { "Shut it down!" },
type: -> { "query alert" },
query: -> { "avg(last_5m):avg:system.load.5{hostgroup:#{project.kennel_id}} by {pod} > #{critical}" }
)
end
end
Reuse it in multiple projects.
class Database < Kennel::Models::Project
defaults(
team: -> { Kennel::Models::Team.new(mention: -> { '@slack-foo' }, kennel_id: -> { 'foo' }) },
parts: -> { [Monitors::LoadTooHigh.new(self, critical: -> { 13 })] }
)
end
Run rake kennel:alerts TAG=service:my-service
to see all un-muted alerts for a given datadog monitor tag.
rake kennel:validate_mentions
should run as part of CI
rake kennel:dump > tmp/dump
cat tmp/dump | grep foo
focus on a single type: TYPE=monitors
Show full resources or just their urls by pattern:
rake kennel:dump_grep DUMP=tmp/dump PATTERN=foo URLS=true
https://foo.datadog.com/dasboard/123
https://foo.datadog.com/monitor/123
rake kennel:nodata TAG=team:foo
Setting FORCE_GET_CACHE=true
will cache all get requests, which makes benchmarking improvements more reliable.
rake play
cd template
rake plan
Then make changes to play around, do not commit changes and make sure to revert with a rake kennel:update_datadog
after deleting everything.
To make changes via the UI, make a new free datadog account and use it's credentaisl instead.
Michael Grosser
michael@grosser.it
License: MIT