IFW API and Invoke-IcingaCheckService Invalid JSON primitiv

We have began migrating to the new IFW API feature in version 1.11.1 of the framework however we have encountered the following bug on the service check. Basically as we have added services to the service include we have begun recieving the following error in output:
Got JSON, but not an object, from IfW API on host 'localhost' port '5668': "Invalid JSON primitive: atus."
In event viewer we have:

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="IfW::RESTApi" /> 
  <EventID Qualifiers="0">2051</EventID> 
  <Level>2</Level> 
  <Task>1</Task> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2023-11-10T19:30:01.000000000Z" /> 
  <EventRecordID>274984</EventRecordID> 
  <Channel>Icinga for Windows</Channel> 
  <Computer>scommsccmt01</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data>Failed to execute API call An API call could not be processed due to an internal exception. Icinga for Windows exception report: Exception Message: Invalid JSON primitive: nning. Invocation Name: ConvertFrom-Json Command Origin: Internal Script Line Number: 24546 Exact Position: At C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\cache\framework_cache.psm1:24546 char:28 + $PSArguments = ConvertFrom-Json -InputObject $CheckConfig; + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ StackTrace: at System.Web.Script.Serialization.JavaScriptObjectDeserializer.DeserializePrimitiveObject() at System.Web.Script.Serialization.JavaScriptObjectDeserializer.DeserializeInternal(Int32 depth) at System.Web.Script.Serialization.JavaScriptObjectDeserializer.BasicDeserialize(String input, Int32 depthLimit, JavaScriptSerializer serializer) at System.Web.Script.Serialization.JavaScriptSerializer.Deserialize(JavaScriptSerializer serializer, String input, Type type, Int32 depthLimit) at Microsoft.PowerShell.Commands.JsonObject.ConvertFromJson(String input, ErrorRecord& error) at Microsoft.PowerShell.Commands.ConvertFromJsonCommand.ProcessRecord() at System.Management.Automation.CommandProcessor.ProcessRecord() Call Stack: Command Arguments Location ------- --------- -------- Get-IcingaExceptionString {ExceptionObject=Invali... framework_cache.psm1: ... Write-IcingaEventMessage {Namespace=RESTApi, Eve... framework_cache.psm1: ... New-IcingaForWindowsRES... {RequireAuth=, ThreadId=2} framework_cache.psm1: ... Object details: No additional object details provided.</Data> 
  </EventData>
  </Event>

To get rid of the error I have to either remove some services and or split the service into 2 seperate service checks. I have roughly 100 services to include so I immagine there may be some new limit. However this limit has been introduced with the API call from the icinga agent as previously it was working. This service check is slightly modified in that if a service is not found on the windows machine it does not require to alert or go into an unknown state, it remains ok if the service is not present.

Other parameters in use are the following:

I am using the latest of all components, IFW 1.11.1, icinga 2.14, IcingaDB 1.1.1 ecc. This specific test was performed on a Windows Server 2012 R2.

So is this error a wanted hard limit or an unwanted side effect?

This limit seems to be present for excluded services also.

This is not an Icinga for Windows limitation, but for some reason related to the Icinga configuration or the Icinga Agent API implementation.

You can test this with the following code:

Set-IcingaTLSVersion;
Enable-IcingaUntrustedCertificateValidation;
[array]$Services = (Get-Service).Name;
$Body = @{ 'Service' = $Services };
(Invoke-WebRequest -UseBasicParsing -Uri 'https://localhost:5668/v1/checker?command=service' -Method POST -ContentType 'application/json' -Body (ConvertTo-Json $Body) ).Content;

The Icinga for Windows API will process this request just fine, but the JSON argument the Icinga Agent is transmitting seems to be incomplete.

@Al2Klimov: Any idea how to resolve this?

Hello @drapiti and thanks for reporting!

Please query the service in question from the Icinga 2 API. I need this part:

                "last_check_result": {
                    "active": true,
                    "check_source": "Alexanders-MacBook-Pro-2.local",
                    "command": [
                        "curl",
                        "--verbose",
                        "--tlsv1.2",
                        "--fail-with-body",
                        "--connect-to",
                        "Alexanders-MacBook-Pro-2.local:5668:localhost:5668",
                        "--ciphers",
                        "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256",
                        "--cert",
                        "/Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/certs//Alexanders-MacBook-Pro-2.local.crt",
                        "--key",
                        "/Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/certs//Alexanders-MacBook-Pro-2.local.key",
                        "--cacert",
                        "/Users/aklimov/NET/WS/icinga2/prefix/var/lib/icinga2/certs//ca.crt",
                        "--request",
                        "POST",
                        "--url",
                        "https://Alexanders-MacBook-Pro-2.local:5668/v1/checker?command=ifw-api",
                        "--user-agent",
                        "Icinga/v2.14.0-136-g778b23087",
                        "--header",
                        "Accept: application/json",
                        "--header",
                        "Content-Type: application/json",
                        "--data-raw",
                        "{}"
                    ],

Best,
A/K

Here is the output:

        "last_check_result": {
          "active": true,
          "check_source": "servername.xxxx.xxxxxxx.xx",
          "command": [
            "curl",
            "--verbose",
            "--tlsv1.2",
            "--fail-with-body",
            "--connect-to",
            "servername.xxxx.xxxxxxx.xx:5668:localhost:5668",
            "--ciphers",
            "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256",
            "--cert",
            "C:\\ProgramData\\icinga2\\var\\lib\\icinga2/certs//servername.xxxx.xxxxxxx.xx.crt",
            "--key",
            "C:\\ProgramData\\icinga2\\var\\lib\\icinga2/certs//servername.xxxx.xxxxxxx.xx.key",
            "--cacert",
            "C:\\ProgramData\\icinga2\\var\\lib\\icinga2/certs//ca.crt",
            "--request",
            "POST",
            "--url",
            "https://servername.xxxx.xxxxxxx.xx:5668/v1/checker?command=Invoke-IcingaCheckService",
            "--user-agent",
            "Icinga/v2.14.2",
            "--header",
            "Accept: application/json",
            "--header",
            "Content-Type: application/json",
            "--data-raw",
            "{\"FilterStartupType\":\"Automatic\",\"MitigateUnknown\":true,\"OverrideNotFound\":\"Ok\",\"Service\":[\"AATPSensor\",\"AATPSensorUpdater\",\"ADWS\",\"Amsp\",\"AzureADPasswordProtectionDCAgent\",\"AppIDSvc\",\"BESClient\",\"BFE\",\"BrokerInfrastructure\",\"CcmExec\",\"ClusSvc\",\"CohesityAgent\",\"CoreMessagingRegistrar\",\"CryptSvc\",\"DcomLaunch\",\"Dfs\",\"DFSR\",\"Dhcp\",\"DhcpServer\",\"DNS\",\"DiagTrack\",\"Dnscache\",\"DPS\",\"ds_agent\",\"ds_monitor\",\"ds_notifier\",\"EventLog\",\"EventSystem\",\"WpnService\",\"gpsvc\",\"hpDiscAgent\",\"IISADMIN\",\"IKEEXT\",\"iphlpsvc\",\"Kdc\",\"LanmanServer\",\"LanmanWorkstation\",\"LSM\",\"MpsSvc\",\"MSDTC\",\"Netlogon\",\"NlaSvc\",\"NPSrvHost\",\"nsi\",\"NTDS\",\"NWEAgent\",\"PcaSvc\",\"Power\",\"ProfSvc\",\"RpcEptMapper\",\"RpcSs\",\"SamSs\",\"Schedule\",\"SCVMMAgent\",\"SENS\",\"Spooler\",\"SystemEventsBroker\",\"SplunkForwarder\",\"TrkWks\",\"UALSVC\",\"vmms\",\"UserManager\",\"VGAuthService\",\"VMTools\",\"VMware Physical Disk Helper Service\",\"VMwareCAFManagementAgentHost\",\"W32Time\",\"Wcmsvc\",\"Winmgmt\",\"WinRM\",\"WpnService\",\"Sense\",\"adfssrv\",\"AdHealthAdfsDiagnostics\",\"AdHealthAdfsInsights\",\"AdHealthAdfsMonitor\",\"FontCache\",\"lmhosts\",\"NetBackup Client Service\",\"NetBackup Discovery Framework\",\"NetBackup Legacy Client Service\",\"NetBackup Legacy Network Service\",\"SNMP\"],\"Status\":\"Running\",\"Verbosity\":\"0\"}"
          ],

I'm afraid this is an IfW problem, not an Icinga 2 one. For demonstration fire ANY valid request against the IfW API, e.g.:

curl -kH 'Content-Type: application/json' 'https://10.27.1.128:5668/v1/checker?command=Invoke-IcingaCheckCPU' -d '{}'

{"Invoke-IcingaCheckCPU":{"exitcode":0,"checkresult":"[OK] CPU Load\n\\_ [OK] Top 10 Process CPU usage\n \\_ [OK] Process Data: No process with high CPU usage found","perfdata":["\u00270_1::ifw_cpu::load\u0027=1.468133%;;;0;100","\u0027totalload::ifw_cpu::load\u0027=2.309196%;;;0;100","\u00270_0::ifw_cpu::load\u0027=3.15026%;;;0;100","\u00270_total::ifw_cpu::load\u0027=2.309196%;;;0;100"]}}

OK? OK. And now add --limit-rate 1 to the curl command line and IfW will complain: "Invalid JSON primitive: ."

icinga-powershell-framework/lib/webserver/Read-IcingaTCPStream.psm1

Line 20 in ca8e5bd

$MessageSize = $Stream.Read($bytes, 0, $ReadLength);

In all programming languages the regular read function returns the first chunk available, not necessarily all data.

I have provided a PR for this, which should resolve the issue. The following improvements are done with this change:

The Icinga for Windows REST-Api will now try to figure out, what was send already to the server
In case the message was containing a body before, it will try to read the rest of the message
In case no body was send before, it will read the entire message
To make sure you can't break the API with these sorts of calls, it will now terminate itself after 10 or 15 seconds, depending on the message size
REST-Threads are now set to active before starting to read a network message, ensuring that parallel calls to the API are scheduled on other, inactive threads

If session is terminated because the client didn't manage to send all configurations to the API, it will terminate with the following message in Icinga context:

This does not change the behavior of the scheduling of the check. This timeout only applies in case the client is unable to manage to send the entire REST-Call within 10 or 15 seconds.

Would you be able to test the linked PR and check if your issues is resolved by that and if everything else is still working as expected?

I have provided a PR for this, which should resolve the issue. The following improvements are done with this change:

The Icinga for Windows REST-Api will now try to figure out, what was send already to the server

In case the message was containing a body before, it will try to read the rest of the message

In case no body was send before, it will read the entire message

To make sure you can't break the API with these sorts of calls, it will now terminate itself after 10 or 15 seconds, depending on the message size

REST-Threads are now set to active before starting to read a network message, ensuring that parallel calls to the API are scheduled on other, inactive threads

If session is terminated because the client didn't manage to send all configurations to the API, it will terminate with the following message in Icinga context:
This does **not** change the behavior of the scheduling of the check. This timeout only applies in case the client is unable to manage to send the entire REST-Call within 10 or 15 seconds.
Would you be able to test the linked PR and check if your issues is resolved by that and if everything else is still working as expected?

Will do, I will get back to you on this.

Great it is working.

Thank you for the quick test!
@Al2Klimov Can you please test as well and try to break everything? 😊

I will prepare a v1.12.1 and get some other fixes into it as well. The release should be by the end of April at latest.