telekom-mms/ansible-collection-icinga-director

[Bug] Changes are reported in Icinga Activity log when nothing is different

Closed this issue · 10 comments

flkhndlr commented

Description

When rolling out the config with the collection Icinga shows changes in all object types (like commands or hostgroups). No changes to the config were made though and the rendered config is the same, too. This behavior is consistent over multiple runs. The objects shown in the diff inside the activity log are not there.

Reproduction steps

  1. Rollout multiple Objects with multiple commands, timeperiods, hostgroups, hosttemplates etc.
  2. Render config
  3. See Changes in activitiy log (new object, former object)

Current Behavior

Changes are reported in the activity log, but diff is non

Expected Behavior

No Changes in the activity log

Additional information

No response

rndmh3ro commented

Can you please provide the used versions of the collection, icinga and the icinga-director?

Also, do you have a minimal reproduction case?

schurzi commented

Minimal reproduction case for the icinga_user module:

Run local image with director 1.9.1.: docker run -p 8080:80 ghcr.io/t-systems-mms/icinga2:director-1.9.1

- hosts: localhost
  tasks:
  - name: Create timeperiod
    telekom_mms.icinga_director.icinga_timeperiod:
      state: present
      url: "http://localhost:8080/icingaweb2"
      url_username: "icingaadmin"
      url_password: "icinga"
      object_name: '24/7'
      ranges:
        monday: "00:00-23:59"
        tuesday: "00:00-23:59"
        wednesday: "00:00-23:59"
        thursday: "00:00-23:59"
        friday: "00:00-23:59"
        saturday: "00:00-23:59"
        sunday: "00:00-23:59"
  - name: Create user template
    telekom_mms.icinga_director.icinga_user_template:
      state: present
      url: "http://localhost:8080/icingaweb2"
      url_username: "icingaadmin"
      url_password: "icinga"
      object_name: "domon-user"
      enable_notifications: false
      period: '24/7'
  - name: Create user
    telekom_mms.icinga_director.icinga_user:
      state: present
      url: "http://localhost:8080/icingaweb2"
      url_username: "icingaadmin"
      url_password: "icinga"
      object_name: "rb"
      imports: domon-user
      display_name: "Rufbereitschaft_24x7"
      email: "alerts@mms-support.de"
      period: "24/7"

This will always show changes für the user and the user template.

schurzi commented

in verbose mode the icinga_user module shows a diff:

    "diff": {
        "after": {
            "groups": "None"
        },
        "before": {
            "groups": "[]"
        }
    },

When setting the default to an empty array, the diff vanishes but the tasks still reports changes.

In icinga_user.py:

groups=dict(type="list", elements="str", required=False, default=[]),

Playbook result:

changed: [localhost] => {
    "changed": true,
    "diff": {},
schurzi commented

HTTP flow from strace:

[pid 109728] sendto(5, "
    GET /icingaweb2/director/user?name=rb HTTP/1.1
    Content-Type: application/x-www-form-urlencoded
    Content-Length: 0
    Host: localhost:8080
    Accept: application/json
    X-Http-Method-Override: GET
    Connection: close

    ", 317, 0, NULL, 0) = 317
[pid 109728] recvfrom(5, "
    HTTP/1.1 200 OK
    Content-Length: 207
    Content-Type: application/json

    {\n    \"display_name\": \"Rufbereitschaft_24x7\",\n    \"email\": \"alerts@mms-support.de\",\n    \"imports\": [\n        \"domon-user\"\n    ],\n    \"object_name\": \"rb\",\n    \"object_type\": \"object\",\n    \"period\": \"24\\/7\"\n}\n", 8192, 0, NULL, NULL) = 367

[pid 109728] sendto(5, "
    GET /icingaweb2/director/user?name=rb&withNull HTTP/1.1
    Content-Type: application/x-www-form-urlencoded
    Content-Length: 0
    Accept: application/json
    X-Http-Method-Override: GET
    Connection: close

    ", 326, 0, NULL, 0) = 326
[pid 109728] recvfrom(5, "
    HTTP/1.1 200 OK
    Content-Length: 374
    Content-Type: application/json

    {\n    \"disabled\": false,\n    \"display_name\": \"Rufbereitschaft_24x7\",\n    \"email\": \"alerts@mms-support.de\",\n    \"enable_notifications\": null,\n    \"groups\": [],\n    \"imports\": [\n        \"domon-user\"\n    ],\n    \"object_name\": \"rb\",\n    \"object_type\": \"object\",\n    \"pager\": null,\n    \"period\": \"24\\/7\",\n    \"states\": null,\n    \"types\": null,\n    \"vars\": {},\n    \"zone\": null\n}\n", 8192, 0, NULL, NULL) = 534

[pid 109728] sendto(5, "
    POST /icingaweb2/director/user?name=rb HTTP/1.1
    Content-Type: application/x-www-form-urlencoded
    Content-Length: 213
    Accept: application/json
    X-Http-Method-Override: POST
    Connection: close

    ", 321, 0, NULL, 0) = 321
[pid 109728] sendto(5, "
    {\"object_name\": \"rb\", \"display_name\": \"Rufbereitschaft_24x7\", \"imports\": [\"domon-user\"], \"disabled\": false, \"email\": \"alerts@mms-support.de\", \"pager\": null, \"period\": \"24/7\", \"groups\": [], \"object_type\": \"object\"}", 213, 0, NULL, 0) = 213
[pid 109728] recvfrom(5, "
    HTTP/1.1 200 OK
    Content-Length: 207
    Content-Type: application/json

    {\n    \"display_name\": \"Rufbereitschaft_24x7\",\n    \"email\": \"alerts@mms-support.de\",\n    \"imports\": [\n        \"domon-user\"\n    ],\n    \"object_name\": \"rb\",\n    \"object_type\": \"object\",\n    \"period\": \"24\\/7\"\n}\n", 8192, 0, NULL, NULL) = 367
rndmh3ro commented

Info: This does not happen with the director version 1.8.1.

  • 👍1
schurzi commented

I've tried swapping and changing parameters in our request and it seems the Director will always return HTTP 200 (object changed) for the user endpoint.

schurzi commented

Looking at the SQL log I see the Director is only updating the timeperiod for the user:
UPDATE icinga_user SET period_id = '2' WHERE (uuid = '...')

rndmh3ro commented

I've tried swapping and changing parameters in our request and it seems the Director will always return HTTP 200 (object changed) for the user endpoint.

Yes, that's the reason. In 1.8.1 we correctly get a HTTP 304 (not modified). In 1.9.1 we always get a HTTP 200, no matter if it changes or not.

See my test-case:

With director 1.8.1:

Create a new object:

> curl -v -H "Accept: application/json" -L -u icingaadmin:icinga  "http://localhost:8081/icingaweb2/director/user" --data '{
    "enable_notifications": true,
    "object_name": "domon-user-test-user",
    "object_type": "object",
    "period": "24x7"
}'
*   Trying 127.0.0.1:8081...
* Connected to localhost (127.0.0.1) port 8081 (#0)
* Server auth using Basic with user 'icingaadmin'
> POST /icingaweb2/director/user HTTP/1.1
> Host: localhost:8081
> Authorization: Basic aWNpbmdhYWRtaW46aWNpbmdh
> User-Agent: curl/7.81.0
> Accept: application/json
> Content-Length: 130
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 201 Created
< Date: Thu, 08 Feb 2024 10:42:23 GMT
< Server: Apache/2.4.38 (Debian)
< Content-Length: 131
< Content-Type: application/json
<
{
    "enable_notifications": true,
    "object_name": "domon-user-test-user",
    "object_type": "object",
    "period": "24x7"
}
* Connection #0 to host localhost left intact

Update the Object with no changes:

> curl -v -H "Accept: application/json" -L -u icingaadmin:icinga  "http://localhost:8081/icingaweb2/director/user?name=domon-user-test-user" --data '{
    "enable_notifications": true,
    "object_name": "domon-user-test-user",
    "object_type": "object",
    "period": "24x7"
}'
*   Trying 127.0.0.1:8081...
* Connected to localhost (127.0.0.1) port 8081 (#0)
* Server auth using Basic with user 'icingaadmin'
> POST /icingaweb2/director/user?name=domon-user-test-user HTTP/1.1
> Host: localhost:8081
> Authorization: Basic aWNpbmdhYWRtaW46aWNpbmdh
> User-Agent: curl/7.81.0
> Accept: application/json
> Content-Length: 130
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 304 Not Modified
< Date: Thu, 08 Feb 2024 10:42:29 GMT
< Server: Apache/2.4.38 (Debian)
<
* Connection #0 to host localhost left intact

Returns 304.

With director 1.9.1:

Create a new object:

> curl -v -H "Accept: application/json" -L -u icingaadmin:icinga  "http://localhost:80/icingaweb2/director/user" --data '{
    "enable_notifications": true,
    "object_name": "domon-user-test-user",
    "object_type": "object",
    "period": "24x7"
}'
*   Trying 127.0.0.1:80...
* Connected to localhost (127.0.0.1) port 80 (#0)
* Server auth using Basic with user 'icingaadmin'
> POST /icingaweb2/director/user HTTP/1.1
> Host: localhost
> Authorization: Basic aWNpbmdhYWRtaW46aWNpbmdh
> User-Agent: curl/7.81.0
> Accept: application/json
> Content-Length: 130
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 201 Created
< Date: Thu, 08 Feb 2024 10:41:21 GMT
< Server: Apache/2.4.38 (Debian)
< Content-Length: 131
< Content-Type: application/json
<
{
    "enable_notifications": true,
    "object_name": "domon-user-test-user",
    "object_type": "object",
    "period": "24x7"
}
* Connection #0 to host localhost left intact

Update the Object with no changes:

> curl -v -H "Accept: application/json" -L -u icingaadmin:icinga  "http://localhost:80/icingaweb2/director/user?name=domon-user-test-user" --data '{
    "enable_notifications": true,
    "object_name": "domon-user-test-user",
    "object_type": "object",
    "period": "24x7"
}'
*   Trying 127.0.0.1:80...
* Connected to localhost (127.0.0.1) port 80 (#0)
* Server auth using Basic with user 'icingaadmin'
> POST /icingaweb2/director/user?name=domon-user-test-user HTTP/1.1
> Host: localhost
> Authorization: Basic aWNpbmdhYWRtaW46aWNpbmdh
> User-Agent: curl/7.81.0
> Accept: application/json
> Content-Length: 130
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Thu, 08 Feb 2024 10:41:35 GMT
< Server: Apache/2.4.38 (Debian)
< Content-Length: 131
< Content-Type: application/json
<
{
    "enable_notifications": true,
    "object_name": "domon-user-test-user",
    "object_type": "object",
    "period": "24x7"
}

Returns 200 instead of 304.

Bug-Reports:
Icinga/icingaweb2-module-director#2728
Icinga/icingaweb2-module-director#2660

schurzi commented

Looking at the objects in the code of Icinga director I can see a definitive problem. I have dumped the loaded and updated object in the REST call and what is obvious is, that the parameter for the timeperiod refference, which should be an id, is set to the name of the timeperiod. then later it is updated to hold the id. This leeds to problems with change detection and the indicator signifying a change is not even filled in the end.

Object dump:

Icinga\Module\Director\Objects\IcingaUser Object
(
    [table:protected] => icinga_user
    [defaultProperties:protected] => Array
        (
            [id] =>
            [uuid] =>
            [object_name] =>
            [object_type] =>
            [disabled] => n
            [display_name] =>
            [email] =>
            [pager] =>
            [enable_notifications] =>
            [period_id] =>
            [zone_id] =>
        )

    [uuidColumn:protected] => uuid
    [supportsGroups:protected] => 1
    [supportsCustomVars:protected] => 1
    [supportsFields:protected] => 1
    [supportsImports:protected] => 1
    [booleans:protected] => Array
        (
            [enable_notifications] => enable_notifications
        )

    [relatedSets:protected] => Array
        (
            [states] => StateFilterSet
            [types] => TypeFilterSet
        )

    [relations:protected] => Array
        (
            [period] => IcingaTimePeriod
            [zone] => IcingaZone
        )

    [keyName:protected] => object_name
    [autoincKeyName:protected] => id
    [supportsRanges:protected] =>
    [supportsApplyRules:protected] =>
    [supportsSets:protected] =>
    [supportsChoices:protected] =>
    [supportedInLegacy:protected] =>
    [rangeClass:protected] =>
    [type:protected] =>
    [multiRelations:protected] => Array
        (
        )

    [loadedMultiRelations:protected] => Array
        (
        )

    [unresolvedRelatedProperties:protected] => Array
        (
            [period_id] => 24/7
        )

...

    [loadedProperties:protected] => Array
        (
            [id] => 2
            [uuid] => ...
            [object_name] => rb
            [object_type] => object
            [disabled] => n
            [display_name] => Rufbereitschaft_24x7
            [email] => alerts@mms-support.de
            [pager] =>
            [enable_notifications] =>
            [period_id] =>
            [zone_id] =>
        )

    [hasBeenModified:protected] =>
    [loadedFromDb:protected] => 1
    [properties:protected] => Array
        (
            [id] => 2
            [uuid] => ...
            [object_name] => rb
            [object_type] => object
            [disabled] => n
            [display_name] => Rufbereitschaft_24x7
            [email] => alerts@mms-support.de
            [pager] =>
            [enable_notifications] =>
            [period_id] =>
            [zone_id] =>
        )

    [modifiedProperties:protected] => Array
        (
        )

    [protectAutoinc:protected] => 1
    [binaryProperties:protected] => Array
        (
        )

)

This also applies to Zones and maybe other objects. And this also explains, what the diff in the Director schowns changes to Zones and Timeperiods.

What we can do now:

  • wait for a fix in the Director
  • do a local compare of the datastructures (like with check-mode) and only issue a command to Director, if we detect changes
rndmh3ro commented

Amazingly, this seems to have been fixed in director version 1.11.1.

I just tested it locally and there both tasks are idempotent.