Explanation & configuration of DEDUPLICATION_FILTER_NAME missing

[jonathanrelf]

In setting up the Nagios XI integration yesterday, I followed the instructions and it failed to push a new event to xMatters. On inspection it turned out I was getting the following error:

2018/10/16 14:01:38.717 +0000 GMT [inbound.applications.nagios-service.normal-1] ERROR - Integration Service (applications, nagios-service) was unable to process a request due to
an unexpected exception: java.lang.RuntimeException: Requested filter 'sample-relevance-engine' is not defined. Request: <?xml version="1.0" encoding="UTF-8"?>
java.lang.RuntimeException: Requested filter 'sample-relevance-engine' is not defined

In both of the configuration_host.js and configuration_service.js files in NagiosXI-IAFiles.zip there's a section at the bottom of the files with the following definition :

// ----------------------------------------------------------------------------------------------------
// Filter to use in <IAHOME>/conf/deduplicator-filter.xml
// ----------------------------------------------------------------------------------------------------
DEDUPLICATION_FILTER_NAME = "sample-relevance-engine";

From what I can see, Integration Agent 5.2.1 doesn't have any reference to sample-relevance-engine and only has ping-plan, sample-integrated-properties, sample-plan, and sample-terminate applications by default.

The deduplication-filter.xml file only defines the following filters:

<?xml version="1.0"?>
<!--
 | EventDeduplicator Configuration file.
 | 
 | Filter attributes
 |   attribute: predicates
 |   comments : This is the list of predicates that are considered in event correlation. An event request will
 |              be considered for deduplication only if all of these predicates are present in the request, AND
 |              if all of the predicates' values match those of a previous request.
 |
 |   attribute: suppression_period
 |   comments : An event request will be deduplicated only if a matching request was processed less than this
 |              number of seconds ago.
 |
 |   attribute: window_size
 |   comments : This is the number of non-matching events before the suppression period timer is reset.
 |              An event request will not be deduplicated if this many non-matching requests have been received
 |              within the suppression period.
 +-->
<deduplicator xmlns="http://www.alarmpoint.com/schema">
  <!-- sample filter to prevent pinging more often than once every 5 minutes to any device -->
  <filter name="ping-plan">
    <predicates>
      <predicate>device</predicate>
    </predicates>
    <suppression_period>300</suppression_period>
    <window_size>2000</window_size>
  </filter>

  <filter name="sample-terminate">
    <predicates>
      <predicate>propertyName</predicate>
      <predicate>propertyValue</predicate>
    </predicates>
    <suppression_period>300</suppression_period>
    <window_size>2000</window_size>
  </filter>
  
  <filter name="sample-plan">
    <predicates>
      <predicate>building</predicate>
      <predicate>city</predicate>
    </predicates>
    <suppression_period>20</suppression_period>
    <window_size>100</window_size>
  </filter>
  
</deduplicator>

I think the instructions need updating to indicate that a deduplication filter edit is necessary or the NagiosXI-IAFiles.zip should either point to a suitable example filter.

Also, the way that this repository is set up with two Zip files as the main contents doesn't help making suggested changes to their contents by Pull Requests, like I did yesterday with the README.md. Do you think there might be a different way of setting this up to allow for that and still make deployment easy?

Cheers,
Jonathan

[xMTinkerer]

Ah, they removed the sample-relevance-engine at some point. Comm Plans used to be called Relevance Engines so I'm guessing this is an update from that.

I hear you about the zip. In the past we delivered the IA files by .zip so it made sense to carry that forward here, but I think it makes more sense to do both. I'll break it out and make the updates you've noted.

[xMTinkerer]

@jonathanrelf, do you think this would make for a good deduplication filter?

<deduplicator xmlns="http://www.alarmpoint.com/schema">
  <!-- sample filter to prevent pinging more often than once every 5 minutes to any device -->
  <filter name="nagiosxi-host">
    <predicates>
      <predicate>host</predicate>
    </predicates>
    <suppression_period>120</suppression_period>
    <window_size>25</window_size>
  </filter>

  <filter name="nagiosxi-service">
    <predicates>
      <predicate>service</predicate>
    </predicates>
    <suppression_period>120</suppression_period>
    <window_size>25</window_size>
  </filter>  
</deduplicator>

Edit: Reduced the suppression_period and the window_size

[jonathanrelf]

@xMTinkerer Yes, it gives people a starting point to at least consider what values they'd like the filter to suppress. 👍
At the moment I've left all the sample entries in <IAHOME>/conf/deduplicator-filter.xml but now I've proven the basic IA connectivity I don't need any of those on my Nagios box. So the instructions could suggest to replace the contents of that file; provided they haven't got any other applications on the same server pushing to xMatters of course :)

[xMTinkerer]

Nah, let's just overwrite it all! :D
Thanks. I'll update this and push it.

[xMTinkerer]

Updated. Thanks for the feedback!