Playbook Dispatcher
Playbook Dispatcher is a service for running Ansible Playbooks on hosts connected via Cloud Connector.
Playbook Dispatcher takes care of:
- dispatching the request to run a Playbook
- tracking the progress of a Playbook run
Playbook Dispatcher consists of 3 parts:
- API, which can be used to dispatch a Playbook or to query it's state
- validator, which validates archives uploaded via ingress service
- response consumer, which processes validated archives and updates the internal state accordingly
Public REST interface
Information about Playbook runs can be queried using the REST API.
Filtering
The API offers rich filtering capabilities that enable resources to be filtered by their primary attributes as well as user-defined labels.
Examples:
/api/playbook-dispatcher/v1/runs?filter[status]=timeout
- filter runs based on the built-instatus
field/api/playbook-dispatcher/v1/runs?filter[labels][state_id]=0fdeeaa3-44e7-459b-9c14-cee42ec39287
- filter runs based on a service-specificstate_id
label/api/playbook-dispatcher/v1/run_hosts?filter[inventory_id]=e72d440b-0128-48fa-9bcc-b964eb8edab0
filter run hosts based on the given host inventory id
More information about supported filters can be found in the API schema
Representations
The API supports client-specific representations using sparse fieldsets.
This allows the client to control which fields are returned in the response.
For example, the shape of the representation of the /api/playbook-dispatcher/v1/runs
resource can be defined using:
/api/playbook-dispatcher/v1/runs?fields[data]=id,url,name
which yields
{
"data": [
{
"id": "5a9d54f5-06c2-46fe-a85e-dcc278cdce44",
"url": "https://cert.cloud.redhat.com/api/config-manager/v1/states/99d4ff95-b2de-41c2-b717-f6261cc5e51b/playbook",
"name": "Sample Playbook"
}
// left out for brevity
More examples:
/api/playbook-dispatcher/v1/runs?fields[data]=id,labels,name,service
/api/playbook-dispatcher/v1/run_hosts?fields[data]=host,status,stdout,links
Default and available fields for each resource can be found in the API schema
Note: Users interested in the up to date output of running playbooks should include the stdout
field
in their requests to the /v1/run_hosts
endpoint.
For example:
/api/playbook-dispatcher/v1/run_hosts?fields[data]=host,status,stdout
Authentication
The API is placed behind a web gateway (3scale). Authentication with the web gateway (basic auth, jwt cookie) is required to access the API. The API is only accessible to principals of type User (i.e. cert auth is not sufficient).
For in-cluster access, a valid x-rh-identity
header is required.
Authorization
The API resources are subject to role based access control.
The playbook-dispatcher:run:read
permission is required to access the API.
The permission always defines an attribute filter which specifies which playbook runs the given principal is authorized to access.
For example, a principal granted the following permission:
{
"permission": "playbook-dispatcher:run:read",
"resourceDefinitions": [
{
"attributeFilter": {
"key": "service",
"operation": "equal",
"value": "remediations"
}
}
]
}
will have access to information about playbook runs initiated by the remediations service.
Information about playbook runs initiated by services for which the principal does not have the corresponding permission will be filtered out of API responses.
Internal REST interface
In addition to the public REST interface, an internal REST interface is available. This API can only be accessed from within the cluster where playbook-dispatcher is deployed.
In order to authenticate, the client is required to hold an app-specific pre-shared key and use it in HTTP requests in the Authorization
header.
POST /internal/dispatch HTTP/1.1
Authorization: PSK <pre-shared key>
playbook-dispatcher supports multiple pre-shared keys to be configured at the same time.
The keys are configured via environment variables in form of PSK_AUTH_<service id>=<pre-shared key>
where service id
identifes the service that's been given the given key and pre-shared key
is the key itself. For example:
PSK_AUTH_REMEDIATIONS=xwKhCUzgJ8 ./app run
Dispatching of playbooks
Use the /internal/v2/dispatch
operation to dispatch a playbook.
Sample request:
POST /internal/v2/dispatch
[
{
"recipient": "dd018b96-da04-4651-84d1-187fa5c23f6c",
"org_id": "5318290",
"url": "http://console.redhat.com/api/remediations/v1/remediations/ddf9196f-4df9-4c7d-9443-98a6f328e256/playbook",
"name":"Apply fix",
"principal": "jharting",
"web_console_url": "http://console.redhat.com/insights/remediations/ddf9196f-4df9-4c7d-9443-98a6f328e256",
"recipient_config": {
"sat_id":"16372e6f-1c18-4cdb-b780-50ab4b88e74b",
"sat_org_id":"6826"
},
"hosts": [
{
"inventory_id": "16372e6f-1c18-4cdb-b780-50ab4b88e74b",
"ansible_host": "01.example.com"
},
{
"inventory_id": "ed36792e-241d-4c12-a74a-a5adffbb436b",
"ansible_host": "02.example.com"
}
]
}
]
In case of a recipient that's using rhc-worker-playbook
(also known as "directly connected host") this can be reduced to:
POST /internal/v2/dispatch
[
{
"recipient": "dd018b96-da04-4651-84d1-187fa5c23f6c",
"org_id": "5318290",
"url": "http://console.redhat.com/api/remediations/v1/remediations/ddf9196f-4df9-4c7d-9443-98a6f328e256/playbook",
"name":"Apply fix",
"principal": "jharting",
"web_console_url": "http://console.redhat.com/insights/remediations/ddf9196f-4df9-4c7d-9443-98a6f328e256"
}
]
as the host is assumed implicitly and recipient config is not needed.
The recipient
field contains the identifier that Cloud Connector uses to route the signal.
Hosts that are using rhc-worker-playbook
have their recipient identifier stored as rhc_client_id in the system profile. The recipient id of Satellite hosts can be obtained from Sources.
Both types of recipients can be used in a single dispatch operation.
See API schema for more details.
Sample response:
[
{
"code": 201,
"id": "c7450666-5614-4826-9961-26c05968fece"
}
]
Canceling of playbooks
Use the /internal/v2/cancel
operation to cancel a playbook.
Sample request:
POST /internal/v2/cancel
[
{
"run_id": "dd018b96-da04-4651-84d1-187fa5c23f6c",
"org_id": "5318290",
"principal": "jharting"
}
]
Sample response:
[
{
"code": 202,
"run_id": "dd018b96-da04-4651-84d1-187fa5c23f6c"
}
]
See API schema for more details.
Recipient status
One of the operations available in the internal API is the recipient status. This operation allows the client to obtain connection information for a given set of recipients.
Sample request:
POST /internal/v2/recipients/status
[
{
"recipient": "35720ecb-bc23-4b06-a8cd-f0c264edf2c1",
"org_id": "5318290"
}, {
"recipient": "73dca8b6-cc11-4954-8f6c-9ecc732ec212",
"org_id": "5318290"
}
]
Sample response:
[
{
"recipient": "35720ecb-bc23-4b06-a8cd-f0c264edf2c1",
"org_id": "5318290",
"connected": true
}, {
"recipient": "73dca8b6-cc11-4954-8f6c-9ecc732ec212",
"org_id": "5318290",
"connected": false
}
]
See API schema for more details.
Event interface
Run Event
Services can integrate with Playbook Dispatcher by consuming its event interface.
Playbook Dispatcher emits an event for every state change.
These events are produced to platform.playbook-dispatcher.runs
topic in Kafka from where integrating services can consume them.
The key of each event is the id of the given playbook run.
The value of each event is described by a JSON schema.
{
"event_type": "create",
"payload": {
"id": "6555d6f7-8dc1-4dec-9d1e-0ef8a02d7d43",
"org_id": "5318290",
"recipient": "dd018b96-da04-4651-84d1-187fa5c23f6c",
"correlation_id": "fbf49ad9-ea79-41fb-9f6c-cb13307e993d",
"service": "remediations",
"url": "http://example.com",
"labels": {
"remediation_id": "1234"
},
"name": "Apply fix",
"web_console_url": "http://example.com/remediations/1234",
"recipient_config": {
"sat_id": "16372e6f-1c18-4cdb-b780-50ab4b88e74b",
"sat_org_id": "6826"
},
"status": "running",
"timeout": 3600,
"created_at": "2022-04-22T11:15:45.429294Z",
"updated_at": "2022-04-22T11:15:45.429294Z"
}
}
An event can be of type:
- create - when a new playbook run is created
- update - when the state of a playbook run changes
- delete - when a playbook run is removed
- read - special type of event used to re-populate the topic (i.e. to remind the consumers of the latest state of a given run). This event does not indicate state change but can be used to populate caches, etc.
In addition, each event contains the following headers:
event_type
- the type of the event (see above)service
- the service that created the given playbook runstatus
- current of the playbook runorg_id
- the account (tenant) the given playbook run belong to
The event headers make it possible to filter events without the need to parse the value of each event.
Run Hosts Event
These events are produced to platform.playbook-dispatcher.run-hosts
topic in Kafka from where integrating services can consume them.
The key of each event is the id of the given playbook run.
The value of each event is described by a JSON schema.
{
"event_type": "create",
"payload": {
"id": "6555d6f7-8dc1-4dec-9d1e-0ef8a02d7d43",
"run_id": "6555d6f7-8dc1-4dec-9d1e-0ef8a02d7d43",
"inventory_id": "16372e6f-1c18-4cdb-b780-50ab4b88e74b",
"host": "01.example.com",
"stdout": "output from the playbook",
"status": "running",
"created_at": "2022-04-22T11:15:45.429294Z",
"updated_at": "2022-04-22T11:15:45.429294Z"
}
}
An event can be of type:
- create - when a new playbook run is created
- update - when the state of a playbook run changes
- delete - when a playbook run is removed
- read - special type of event used to re-populate the topic (i.e. to remind the consumers of the latest state of a given run). This event does not indicate state change but can be used to populate caches, etc.
In addition, each event contains the following headers:
event_type
- the type of the event (see above)status
- current state of the playbook run
The event headers make it possible to filter events without the need to parse the value of each event.
Expected input format
The service expects the uploaded files to be in one of the two supported formats.
Ansible Runner events
The uploaded file contains Ansible Runner job events. Job events should be stored in the newline-delimited JSON format. Each line in the file matches one job event.
{"event": "executor_on_start", "uuid": "4533e4d7-5034-4baf-b578-821305c96da4", "counter": -1, "stdout": "", "start_line": 0, "end_line": 0, "event_data": {"dispatcher_correlation_id": "c37278ac-f41c-424a-8461-7f41b4b87c8e"}}
{"event": "playbook_on_start", "uuid": "cb93301e-5ff8-4f75-ade6-57d0ec2fc662", "counter": 0, "stdout": "", "start_line": 0, "end_line": 0}
{"event": "playbook_on_stats", "uuid": "998a4bd2-2d6b-4c31-905c-2d5ad7a7f8ab", "counter": 1, "stdout": "", "start_line": 0, "end_line": 0}
The structure of each event is validated using a JSON Schema defined in ansibleRunnerJobEvent.yaml. Note that additional attributes (not defined by the schema) are allowed.
For plain files, the expected content type of the uploaded file is application/vnd.redhat.playbook.v1+jsonl
.
For compressed files, the expected content type of the uploaded file is application/vnd.redhat.playbook.v1+gzip
for gzip compressed files and application/vnd.redhat.playbook.v1+xz
for xz compressed files.
Non-standard event types
Besides Ansible Runner event types (playbook_*
and runner_*
) the services recognizes two additional event types.
Firstly, executor_on_start
event type is produced before Ansible Runner is invoked.
{"event": "executor_on_start", "uuid": "4533e4d7-5034-4baf-b578-821305c96da4", "counter": -1, "stdout": "", "start_line": 0, "end_line": 0, "event_data": {
"crc_dispatcher_correlation_id": "c37278ac-f41c-424a-8461-7f41b4b87c8e"
}}
The event carries the correlation id in the event_data
dictionary.
This identifier is necessary to tie the uploaded file to a Playbook run.
Secondly, executor_on_failed
event type may be produced should it not be possible to start a Playbook run for some reason.
This may happen e.g. if the Playbook signature is not valid, Ansible Runner binary is not available/executable, etc.
{"event": "executor_on_failed", "uuid": "14f07467-0020-4ab7-a075-d7d8c96d6fdc", "counter": -1, "stdout": "", "start_line": 0, "end_line": 0, "event_data": {
"crc_dispatcher_correlation_id": "c37278ac-f41c-424a-8461-7f41b4b87c8e",
"crc_dispatcher_error_code": "SIGNATURE_INVALID",
"crc_dispatcher_error_details": "Signature \"783701f7599830824fa73488f80eb79894f6f14203264b6a3ac3f0a14012c25f\" is not valid for Play \"run insights to obtain latest diagnosis info\"",
}}
Again, the correlation id is defined. In addition, an error code and detailed information should be provided.
Satellite Events
In addition to Ansible Runner events, playbook-dispatcher also supports events produced by satellite. Similar to Ansible Runner events, these events from satellite should be stored in newline-delimited JSON, and should match the Satellite job events schema.
{"type": "playbook_run_update", "version": 3, "correlation_id": "0465783c-2e36-4e57-8514-c2cb962d323a", "sequence": 0, "host": "0e4eac4a-e303-4c83-9f6b-2152d4cfa242", "console": "host1"}
{"type": "playbook_run_finished", "version": 3, "correlation_id": "0465783c-2e36-4e57-8514-c2cb962d323a", "host": "0e4eac4a-e303-4c83-9f6b-2152d4cfa242", "status": "success", "connection_code": 0, "execution_code": 0}
{"type": "playbook_run_update", "version": 3, "correlation_id": "0465783c-2e36-4e57-8514-c2cb962d323a", "sequence": 4, "host": "e8f05d27-5af9-4547-a30c-5be04e099ffe", "console": "host2"}
{"type": "playbook_run_finished", "version": 3, "correlation_id": "0465783c-2e36-4e57-8514-c2cb962d323a", "host": "e8f05d27-5af9-4547-a30c-5be04e099ffe", "status": "success", "connection_code": 0, "execution_code": 0}
{"type": "playbook_run_completed", "version": 3, "correlation_id": "0465783c-2e36-4e57-8514-c2cb962d323a", "status": "success", "satellite_connection_code": 0, "satellite_infrastructure_code": 0}
playbook_run_update
and playbook_run_finished
events are bound to hosts, while playbook_run_completed
events are bound to playbook runs and are sent out only once after the playbook run has concluded on all involved hosts. The sequence
and console
fields are not mandatory for playbook_run_finished
and playbook_run_completed
events.
The content type of plain files of uploads need to be application/vnd.redhat.playbook-sat.v3+jsonl
. For compressed files, the content type is expected to be either application/vnd.redhat.playbook-sat.v3+gzip
or application/vnd.redhat.playbook-sat.v3+xz
.
Partial Satellite Response
When playbook-dispatcher is deployed with response_full
set to false
, updates from all hosts involved in a playbook run are not expected with each upload. Satellite do not need to provide the entire console log with each update from a host, instead, they can provide the difference relative to the last playbook_run_update
from a host and playbook-dispatcher will concatenate these logs and record it in the database.
Cloud Connector integration
Playbook Dispatcher uses Cloud Connector to invoke Playbooks on connected hosts. Depending on the type of RHC worker used on the recipient, the message will be in one of the following formats.
rhc-worker-playbook
For each Playbook run request a message with the following format is sent to Cloud Connector:
{
"directive":"rhc-worker-playbook",
"metadata":{
"crc_dispatcher_correlation_id":"e957564e-b823-4047-9ad7-0277dc61c88f", // see Non-standard event types for more details
"response_interval":"600", // how often the recipient should send back responses
"return_url":"https://cloud.redhat.com/api/ingress/v1/upload" // URL to post responses to
},
// playbook to execute
"payload": "https://cloud.redhat.com/api/v1/remediations/1234/playbook?hosts=8f876606-5289-47f7-bb65-3966f0ba3ae1"
}
See rhc-worker-playbook for details.
foreman_rh_cloud
For each Playbook run request a message with the following format is sent to Cloud Connector:
"directive":"foreman_rh_cloud",
"metadata":{
"operation": "run",
// URL to post responses to
"return_url":"https://cloud.redhat.com/api/ingress/v1/upload"
// identifier used to correlate the initial signal with response data messages
"correlation_id": "e957564e-b823-4047-9ad7-0277dc61c88f",
// human-readable name of the playbook run
"playbook_run_name": "Human-readable playbook run name",
// URL that the user can be redirected to for more information about the playbook run
"playbook_run_url": "https://console.redhat.com/insights/remediations/1234",
// identifier of the Satellite instance
"sat_id": "aa3b1faa-56f3-4d14-8258-615d11e20060",
// identifier of the organization within Satellite
"sat_org_id": "12345",
// the SHA-256 hash of the username of the user that initiated the playbook run. May be empty string if the playbook is not initiated by a user
"initiator_user_id": "4efca34c6d9ae05ef7c3d7a7424e6370d198159a841ae005084888a9a4529e27",
// the list of hosts (their host inventory identifiers as defined by host-inventory service) involved in the playbook run
"hosts": "4df562cb-596c-4f2b-9ed1-ff2e920706c5,b562a345-dba5-4f77-a7c9-a4e5ddb4a61a",
// how often the recipient should send back responses
"response_interval": "30",
// indicates whether the playbook run update data message should contain the full console output (true) or only the diff relative to the previous playbook run update message sent for the given host (false)
"response_full": "false"
},
// playbook to execute
"payload": "https://cloud.redhat.com/api/v1/remediations/1234/playbook"
See foreman_rh_cloud for details.
Onboarding guide
New application onboarding guide can be found here.
Development
Prerequisities
- Golang >= 1.14
- golangci-lint
- docker-compose
Running the service
Run docker-compose up --build
to start the service and its dependencies
The API can be accessed at http://localhost:8000/api/playbook-dispatcher/v1/runs
curl -v -H "x-rh-identity: eyJpZGVudGl0eSI6eyJpbnRlcm5hbCI6eyJvcmdfaWQiOiI1MzE4MjkwIn0sImFjY291bnRfbnVtYmVyIjoiOTAxNTc4IiwidXNlciI6e30sInR5cGUiOiJVc2VyIn19Cg==" http://localhost:8000/api/playbook-dispatcher/v1/runs
Inspecting the event interface
-
Download and unpack Kafka
-
Start kafka-console-consumer configured to consume the event interface topic
/bin/kafka-console-consumer.sh --bootstrap-server=localhost:29092 --from-beginning --topic platform.playbook-dispatcher.runs
-
Trigger a state change (e.g.
make sample_request_sat
)
Useful commands:
make sample_request
can be used to dispatch a new sample Playbookmake sample_upload
can be used to upload a sample archive via Ingress
Running tests
make test
Running linter
install and run linter
make run-lint