Add an HTTP API to configure contacts and contactgroups

Question

Add an HTTP API to configure contacts and contactgroups

Opened this issue 7 months ago · 14 comments

Current State

At the moment contacts can only be configured by using the UI. Contactgroups cannot be configured at all. (see #174)

Problem

We can safely assume that consumers already have their users and usergroups defined somewhere. Maintaining them again for Icinga Notifications shouldn't be necessary.

Solution

We should provide a way to automate their creation. The easiest way is to provide a basic HTTP API to do so:

Authorization

The API must require the notifications/api/v1 permission.

Endpoints

Users: notifications/api/v1/contacts[/<identifier> | ?<filter>]
Groups: notifications/api/v1/contactgroups[/<identifier> | ?<filter>]

Parameters

identifier
This is a UUID. For resources created in the UI, this is a UUIDv4.

filter
A usual filter query string

Request Body Validity

A request's body is, if accepted, always expected to be JSON. Since it always describes a single resource, a JSON object. No envelope structure is required. The object is the resource definition itself
Every key not followed by a question mark (?) is mandatory
The default channel of a user must be an existing channel's name (created manually in the UI)
Referenced resources (contact -> groups, group -> contacts) must already exist

Resource Structure (Request and Response)

Contact

{
    id: identifier,
    full_name: string,
    username?: string,
    default_channel: string,
    groups?: identifier[],
    addresses?: {}
}

Contactgroup

{
    id: identifier,
    name: string,
    users?: identifier[]
}

Methods

GET

If a filter is passed
- Respond with a HTTP 200 code and a list of resources, even if empty
If no filter and no identifier is passed
- Respond with a HTTP 200 code and a full list of resources, even if empty
If an identifier is passed
- Respond with a HTTP 200 code and the resource, if it exists
- Respond with a HTTP 404 code, if it does not exist

POST

If a filter or an invalid request body is passed
- Respond with a HTTP 400 code
If an identifier is passed
- Respond with a HTTP 404 code if the resource is not found
- Respond with a HTTP 422 code if the resource already exists and is not replaced (id in the body is equal)
- Respond with a HTTP 201 code if the resource is replaced and include the resulting resource endpoint (with new identifier) in the Location header
If no identifier is passed
- Respond with a HTTP 422 code if the resource already exists
- else, respond with a HTTP 201 code, create the resource and include the resulting resource endpoint (with identifier) in the Location header

PUT

If no identifier or an invalid request body is passed (different id in the body is invalid)
- Respond with a HTTP 400 code
If a valid request body is passed
- Respond with a HTTP 201 code, if the resource is created
- Respond with a HTTP 204 code, if the resource is updated

DELETE

If no identifier is passed
- Respond with a HTTP 400 code
If the resource is not found
- Respond with a HTTP 404 code
If the resource exists
- Respond with a HTTP 204 code and remove it

Schema Changes

Icinga/icinga-notifications#192

Implementation Requirements

A single controller for each endpoint
- Don't rely on other parts of this project (e.g. forms)
- Straight forward code, no unnecessary optimizations or modularisation attempts
- Quick, but not dirty
Bypass the ORM, exclusively utilize ipl-sql
- Implement GET ?filter responses like https://github.com/Icinga/icingadb-web/blob/v1.1.2/library/Icingadb/Data/JsonResultSetUtils.php#L70-L100 and disable PHP's output buffering

Answer 1 · 2024-04-11T15:07:18.000Z

User

{
    full_name: string,
    username: identifier,
    default_channel: string,
    groups?: identifier[]
}

Group

{
    name: identifier,
    users?: identifier[]
}

So there will be two places where group memberships can be updated? Will the behavior be that on an update, memberships that aren't given, will be removed, i.e. to add a group to a user, you have to query the user first? Does omitting the groups/users attribute mean, that memberships remain unchanged?

The username column of the contact table must not be nullable

That column is used to store an optional reference to an Icinga Web user. That would imply that each contact is linked to one then.

The name column of the contactgroup table must be unique

That column currently stores a display name. Sure, that can be encoded as part of an URL, but is this desired here?

But in general, sounds like you want/need a user-chosen primary key for those tables. I wouldn't rule out just doing this instead.

Answer 2 · 2024-04-12T07:52:55.000Z

So there will be two places where group memberships can be updated? Will the behavior be that on an update, memberships that aren't given, will be removed, i.e. to add a group to a user, you have to query the user first? Does omitting the groups/users attribute mean, that memberships remain unchanged?

Yes. Yes. Yes. My expectation is that whoever uses this API, just imports data from another source, so there's no need to fetch anything first, as all information is already available.

But in general, sounds like you want/need a user-chosen primary key for those tables. I wouldn't rule out just doing this instead.

I had a discussion with Eric about the use of UUIDs, which I'd chosen first. Though, they'd still need to reference an identifier (UUIDv5) that is known to both sides, i.e. mandatory in any case. Otherwise (UUIDv4) we'd had to prevent changes in the UI to resources created through the API and vice versa. The primary key isn't an option, hence the username. The group name is indeed more of a label right now.

My goal was, not to limit edits in any way. A resource created through the API should be changeable in the UI. This means, by creating one in the UI, the identifier must be provided, just the same as when creating it through the API.

Answer 3 · 2024-04-12T08:35:19.000Z

I had a discussion with Eric about the use of UUIDs, which I'd chosen first. Though, they'd still need to reference an identifier (UUIDv5) that is known to both sides, i.e. mandatory in any case. Otherwise (UUIDv4) we'd had to prevent changes in the UI to resources created through the API and vice versa. The primary key isn't an option, hence the username. The group name is indeed more of a label right now.

My goal was, not to limit edits in any way. A resource created through the API should be changeable in the UI. This means, by creating one in the UI, the identifier must be provided, just the same as when creating it through the API.

Sounds like you think something is a problem where I'd say it's perfectly fine. UUIDs are an obvious choice, so let's stick to that. I'd say it's perfectly fine to make the primary key a UUID without any restrictions on the version. If a contact ist created using Web, it just gets a random UUID (v4) assigned. If a contact is created using the API, the client chooses the UUID however it desires. If it syncs from somewhere else that already uses an UUID to identify the source object, use that, otherwise, use a hash-based UUID (v5) or even use a random UUID and keep some state, doesn't matter for us, that would be a decision for the API client author.

Creating a contact in Web and then updating it using the API should be possible, yes, but if your sync source is the primary data source anyways, why wouldn't you create all contacts using the API? If your use-case is to just update individual attributes like an e-mail address for existing contacts, the API client could still query the contact by username using the filter mechanism and then update it by the returned ID. Or it could just query all contacts and then update those, where an update is necessary. So that would still be possible without a predictable ID.

Answer 4 · 2024-04-12T09:33:25.000Z

Fine. Let's use UUIDs as identifier. contact.username and contactgroup.name are left untouched then. The UUID of a resource will be part of the structure as id key. But I wouldn't make the UUID the new primary key. We'd have to change every reference in the schema to contacts and groups then.

Oh, btw, I forgot to include addresses. -.-

Answer 5 · 2024-04-12T12:36:19.000Z

But I wouldn't make the UUID the new primary key. We'd have to change every reference in the schema to contacts and groups then.

Yes, but it sounds like the way cleaner option, doesn't it? Would this result in an unreasonable amount of work in Notifications Web?

Answer 6 · 2024-04-12T12:56:39.000Z

Yes, but it sounds like the way cleaner option, doesn't it?

It's not required. I'd rather add a new column for this, as a start. We still don't have versioning, so the schema isn't stable anyway..

Answer 7 · 2024-04-16T09:46:02.000Z

However, if we just replace the primary key type of these two columns, it's a bit of an arbitrary mix.

Another idea for how two different columns could make sense in my opinion: keep the current numeric ID as-is and add a second column, something like external_id or external_key that is nullable and unique. For objects created using the web interface, this value is just not set, but if objects are created via the API, it's an optional field the client can use to store auxiliary information to later identify the same record. I.e. the default identifier stays the numeric ID, but if desired, the API client could access the objects also by this external ID (or username for that matter), all of these would be fast due to the existence of a corresponding index. The type could either be UUID/16 bytes or even just some string type so that the client could store whatever they want, like LDAP DN, ID reference to whatever other database, etc. without requiring any hashing, thus allowing lookups in the other direction as well.

Answer 8 · 2024-04-16T11:15:09.000Z

the default identifier stays the numeric ID

Nope. That goes against everything I read. If we keep our numeric ID, it's won't be exposed in the API.

The type could either be...

A single type. I really don't want to support multiple ways if UUID is one of them. It should be the only one.

thus allowing lookups in the other direction as well.

This is something we already solved and agreed on:

Creating a contact in Web and then updating it using the API should be possible, yes, but if your sync source is the primary data source anyways, why wouldn't you create all contacts using the API? If your use-case is to just update individual attributes like an e-mail address for existing contacts, the API client could still query the contact by username using the filter mechanism and then update it by the returned ID.

However, if we just replace the primary key type of these two columns, it's a bit of an arbitrary mix.

Then let's introduce a new column of type UUID and make it required.

Answer 9 · 2024-04-16T11:29:46.000Z

the default identifier stays the numeric ID

Nope. That goes against everything I read. If we keep our numeric ID, it's won't be exposed in the API.

Why not? Also, I'm not really sure what you read.

The type could either be...

A single type. I really don't want to support multiple ways if UUID is one of them. It should be the only one.

Of course we should pick one. Just wanted to say that the exact choice won't matter for the rest I wrote.

thus allowing lookups in the other direction as well.

This is something we already solved and agreed on:

That's not what I wanted to say with that. If you have a field large enough to store an unhashed reference, an API client could retrieve a list of all contacts, look them up by say a stored LDAP DN, and check if anything needs to be updated. Not sure how commonly someone would want to build something like this, I just wanted to say that's something that would additionally be possible if it was a "store whatever you want" type.

Answer 10 · 2024-04-16T11:53:21.000Z

Why not? Also, I'm not really sure what you read.

To prevent enumeration attacks.

If you have a field large enough to store an unhashed reference, an API client could retrieve a list of all contacts, look them up by say a stored LDAP DN, and check if anything needs to be updated.

If the identifier is a UUID, chosen by the client, no checks are necessary. The client just PUTs the data it has and nothing else. And all with a "store whatever you want" type.

Answer 11 · 2024-04-16T15:28:43.000Z

Why not? Also, I'm not really sure what you read.

To prevent enumeration attacks.

Yes, in general, unpredictable IDs add an additional layer of defense. But isn't the API supposed to allow listing all objects anyways?

If the identifier is a UUID, chosen by the client, no checks are necessary. The client just PUTs the data it has and nothing else. And all with a "store whatever you want" type.

Syncing object deletion would be annoying that way though as you don't really have a way of telling how that UUID was generated. If you have an contact that says external_id = 'uid=poorguy,ou=something,dc=example,dc=com', a sync client could simply check if that still exists in LDAP and if it doesn't, issue a DELETE request.

Answer 12 · 2024-04-16T16:00:39.000Z

We concluded that two columns (pk + external_uuid), both required, are sufficient for now. Making the pk the UUID can be done at a later point, together with a custom fact for the client to identify its own resources. (To allow safe removals)

Answer 13 · 2024-07-04T17:47:05.000Z

Requests

Note

Captions: ${\texttt{\color{#CFBAF0}GET \color{#B9FBC0}POST \color{#FDE4CF}PUT \color{#FFCFD2}DELETE}}$
^{indicate the request method}
Captions: 🟢 🔴
^{declare whether a request is expected to return a success or a failure}

${\texttt{\color{#B9FBC0}POST\,\,\,\:\:}}$ 🔴 1. New contact group (invalid body format)

Description

Create a new contact group with a YAML payload, while declaring the body type as application/json.

Request

URL: http://localhost/icingaweb2/notifications/api/v1/contactgroups
Headers: Accept: application/json, Content-Type: application/json
Body:

---
payload: invalid

Response

Headers: 500 Internal Server Error
Body:

{
    "status": "error",
    "message": "Syntax error: '---\npayload: invalid\n'"
}

${\texttt{\color{#B9FBC0}POST\,\,\,\:\:}}$ 🔴 2. New contact group (invalid body type)

Description

Create a new contact group with a YAML payload.

Request

URL: http://localhost/icingaweb2/notifications/api/v1/contactgroups
Headers: Accept: application/json, Content-Type: text/yaml
Body:

---
payload: invalid

Response

Headers: 400 Bad Request
Body:

{
    "status": "error",
    "message": "No JSON content"
}

${\texttt{\color{#B9FBC0}POST\,\,\,\:\:}}$ 🔴 3. New contact group (incomplete body)

Description

Create a new contact group with a valid JSON payload, that is missing the mandatory name field.

Request

URL: http://localhost/icingaweb2/notifications/api/v1/contactgroups
Headers: Accept: application/json, Content-Type: application/json
Body:

{
    "id": "0817d973-398e-41d7-9ef2-61cdb7ef41a2", 
    "users": []
}

Response

Headers: 400 Bad Request
Body:

{
    "status": "error", 
    "message": "Invalid request body: the fields id and name must be present and of type string"
}

${\texttt{\color{#B9FBC0}POST\,\,\,\:\:}}$ 🔴 4. New contact group (with filter)

Description

Create a new contact group with a valid JSON payload, while providing a filter.

Request

URL: http://localhost/icingaweb2/notifications/api/v1/contactgroups?id=0817d973-398e-41d7-9ef2-61cdb7ef41a2
Headers: Accept: application/json, Content-Type: application/json
Body:

{
    "id": "0817d973-398e-41d7-9ef2-61cdb7ef41a2", 
    "name": "Test group",
    "users": []
}

Response

Headers: 400 Bad Request
Body:

{
    "status": "error",
    "message": "Filter is only allowed for GET requests"
}

${\texttt{\color{#B9FBC0}POST\,\,\,\:\:}}$ 🔴 5. New contact group (with identifier)

Description

Create a new contact group with a valid JSON payload, while providing an identifier.

Request

URL: http://localhost/icingaweb2/notifications/api/v1/contactgroups/0817d973-398e-41d7-9ef2-61cdb7ef41a2
Headers: Accept: application/json, Content-Type: application/json
Body:

{
    "id": "0817d973-398e-41d7-9ef2-61cdb7ef41a2", 
    "name": "Test group",
    "users": []
}

Response

Headers: 404 Not Found
Body:

{
	"status": "error",
	"message": "Contactgroup not found"
}

${\texttt{\color{#B9FBC0}POST\,\,\,\:\:}}$ 🟢 6. New contact group (create)

Description

Create a new contact group with a valid JSON payload.

Request

URL: http://localhost/icingaweb2/notifications/api/v1/contactgroups
Headers: Accept: application/json, Content-Type: application/json
Body:

{
    "id": "0817d973-398e-41d7-9ef2-61cdb7ef41a2", 
    "name": "Test group",
    "users": []
}

Response

Headers: 201 Created, Location: notifications/api/v1/contactgroups/0817d973-398e-41d7-9ef2-61cdb7ef41a2
Body:

{
	"status": "success",
	"data": null
}

${\texttt{\color{#B9FBC0}POST\,\,\,\:\:}}$ 🔴 7. New contact group (replace, equal identifier)

Description

Replace a contact group while providing the same identifier in both the JSON payload and Request-URI.

Request

URL: http://localhost/icingaweb2/notifications/api/v1/contactgroups/0817d973-398e-41d7-9ef2-61cdb7ef41a2
Headers: Accept: application/json, Content-Type: application/json
Body:

{
    "id": "0817d973-398e-41d7-9ef2-61cdb7ef41a2", 
    "name": "Test group",
    "users": []
}

Response

Headers: 422 Unprocessable Entity
Body:

{
	"status": "error",
	"message": "Contactgroup already exists"
}

${\texttt{\color{#B9FBC0}POST\,\,\,\:\:}}$ 🟢 8. New contact group (replace, new identifier)

Description

Replace a contact group while providing a new identifier in the JSON payload.

Request

URL: http://localhost/icingaweb2/notifications/api/v1/contactgroups/0817d973-398e-41d7-9ef2-61cdb7ef41a2
Headers: Accept: application/json, Content-Type: application/json
Body:

{
    "id": "0817d973-398e-41d7-9ef2-61cdb7ef41a3", 
    "name": "Test group (replaced)",
    "users": []
}

Response

Headers: 201 Created, Location: notifications/api/v1/contactgroups/0817d973-398e-41d7-9ef2-61cdb7ef41a3
Body:

{
	"status": "success",
	"data": null
}

${\texttt{\color{#FDE4CF}PUT\,\,\,\,\,\,\,\,\,}}$ 🔴 9. Update contact group (invalid body format)

Description

Update a contact group with a YAML payload, while declaring the body type as application/json.

Request

URL: http://localhost/icingaweb2/notifications/api/v1/contactgroups/0817d973-398e-41d7-9ef2-61cdb7ef41a3
Headers: Accept: application/json, Content-Type: application/json
Body:

---
payload: invalid

Response

Headers: 500 Internal Server Error
Body:

{
	"status": "error",
	"message": "Syntax error: '---\npayload: invalid\n'"
}

${\texttt{\color{#FDE4CF}PUT\,\,\,\,\,\,\,\,\,}}$ 🔴 10. Update contact group (without identifier)

Description

Update a contact group with a YAML payload, while not providing an identifier in the Request-URI.

Request

URL: http://localhost/icingaweb2/notifications/api/v1/contactgroups
Headers: Accept: application/json, Content-Type: application/json
Body:

{
    "id": "0817d973-398e-41d7-9ef2-61cdb7ef41a3", 
    "name": "Test group (replaced)",
    "users": []
}

Response

Headers: 400 Bad Request
Body:

{
	"status": "error",
	"message": "Identifier is required"
}

${\texttt{\color{#FDE4CF}PUT\,\,\,\,\,\,\,\,\,}}$ 🔴 11. Update contact group (identifier mismatch)

Description

Update a contact group with a YAML payload, while providing different identifiers in the request body and Request-URI.

Request

URL: http://localhost/icingaweb2/notifications/api/v1/contactgroups/0817d973-398e-41d7-9ef2-61cdb7ef41a3
Headers: Accept: application/json, Content-Type: application/json
Body:

{
    "id": "0817d973-398e-41d7-9ef2-61cdb7ef41a4", 
    "name": "Test group (replaced)",
    "users": []
}

Response

Headers: 400 Bad Request
Body:

{
	"status": "error",
	"message": "Identifier mismatch"
}

${\texttt{\color{#FDE4CF}PUT\,\,\,\,\,\,\,\,\,}}$ 🟢 12. Create contact group (with identifier)

Description

Create a new contact group with a YAML payload, while providing its identifier in the Request-URI.

Request

URL: http://localhost/icingaweb2/notifications/api/v1/contactgroups/0817d973-398e-41d7-9ef2-61cdb7ef41a4
Headers: Accept: application/json, Content-Type: application/json
Body:

{
    "id": "0817d973-398e-41d7-9ef2-61cdb7ef41a4",
    "name": "Test group 2",
    "users": []
}

Response

Headers: 201 Created
Body:

{
	"status": "success",
	"data": null
}

${\texttt{\color{#FDE4CF}PUT\,\,\,\,\,\,\,\,\,}}$ 🟢 13. Update contact group (with identifier)

Description

Update an existing contact group with a YAML payload, while providing its identifier in the Request-URI.

Request

URL: http://localhost/icingaweb2/notifications/api/v1/contactgroups/0817d973-398e-41d7-9ef2-61cdb7ef41a4
Headers: Accept: application/json, Content-Type: application/json
Body:

{
    "id": "0817d973-398e-41d7-9ef2-61cdb7ef41a4",
    "name": "Test group 2 (updated)",
    "users": []
}

Response

${\texttt{\color{#FFCFD2}DELETE}}$ 🟢 21. Delete contact group (same as request 20)

Description

Deletes a contact group, while providing an existing identifier.

Request

URL: http://localhost/icingaweb2/notifications/api/v1/contactgroups/0817d973-398e-41d7-9ef2-61cdb7ef41a4
Headers: Accept: application/json
Body: none

Response

Headers: 204 No Content
Body: none

Broken requests

Request 1
This request triggers an uncatched exception as it's passing a YAML, but declares it as application/json.
The endpoint shouldn't trust HTTP headers but check for the validity of the
payload before passing it to a JSON decoder.
Request 5
The RESTful API should deny this request as it provides a UUID in the Request-URI while using the POST method.
It currently accepts the request and returns a 404 Not Found as no group with the given UUID could be found.
Request 9
This request triggers an uncatched exception as it's passing a YAML, but declares it as application/json.
The endpoint shouldn't trust HTTP headers but check for the validity of the
payload before passing it to a JSON decoder.
Request 16
There's a few things wrong with this request.
First of all, the endpoint always returns a list, as there's a filter provided with the request.
This would be correct, if the filter would get processed as a LIKE (~) condition and not as a full comparison (===, equals).
It currently does a full match, which is why the result can only contain a single resource when providing an id= filter (since its unique) and multiple resources when providing a name= filter (multiple resources with the same name are technically possible).
Additionally, there's a problem with the filter parsing mechanism.
If the request gets executed with brackets in its filter name, the request fails with a 500 Internal Server Error.

_{Example URI}
http://localhost/icingaweb2/notifications/api/v1/contactgroups?name=Test%20group%20(replaced)
_Body
```
{
	  "status": "error",
	  "message": "Invalid filter \"name=Test group (replaced)\", unexpected ) at pos 26"
}
```

Answer 14 · 2024-07-05T08:59:04.000Z

Thanks for the tests. Please write directly in the PR next time.
Requests 1 and 9 now throw 400 Bad Request.
Request 5 is implemented as the requirements above. If the ID is specified, an attempt is made to replace the entry.