contact based routing only accepts 1 email address - same as #55

Question

contact based routing only accepts 1 email address - same as #55

njomeara opened this issue 7 years ago · 15 comments

Hi,

I have the same problem as reported in ..
contact based routing only accepts 1 email address #55

The email contacts are configured in the check. Emails only get sent to the default address configured in the mymail_handler.json file and not to the contacts configured in the support.json file.

If i only put one address in "contact" field in the check it will send to that address, adding an array of addresses doesn't work.

mymail_handler.json

{
    "mailer" :  {
        "mail_from": "sensu@ieatoskca-sensu3",
        "mail_to": ["nj@company.com"],
        "delivery_method": "smtp",
        "smtp_address": "smtp.internal.company.com",
        "smtp_port": "25",
        "smtp_domain": "company.com"
    },
    "handlers": {
        "mailer": {
            "type": "pipe",
            "command": "/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html",
         "filter": "state_change_only",
	          "severities": ["ok","warning","critical","unknown"],
            "handle_silenced": true,
            "handle_flapping": true
	    
        }
    }
}

support.json

{
  "contacts": {
    "support": {
      "email": {
        "to": "nj@company.com"
      }
    },
    "team1": {
      "email": {
        "to": "team1@company.com"
      }
    },
    "team2": {
      "email": {
        "to": "team2@company.com"
      }
    }
  }
}

check_test.json

{
    "checks": {
        "test_ignore": {
            "command": "exit 2",
            "standalone": true,
            "interval": 30,
            "handlers": ["mailer"],
            "contact": [
		"support",
		"team2"
	    ]
         }
     }
}

I turned on debug and attached the output of:

grep -i test_ignore /var/log/sensu/sensu-server.log
[debug.txt](https://github.com/sensu-plugins/sensu-plugins-mailer/files/1291942/debug.txt)

Sensu was running on version 0.29. I updated to latest as below with same result...

[root@ieatoskca-sensu3 tmp]# gem list | grep sensu
sensu (1.0.3)
sensu-cli (0.8.0)
sensu-extension (1.5.1)
sensu-extensions (1.9.0)
sensu-extensions-check-dependencies (1.0.1)
sensu-extensions-debug (1.0.0)
sensu-extensions-json (1.0.0)
sensu-extensions-occurrences (1.2.0)
sensu-extensions-only-check-output (1.0.0)
sensu-extensions-ruby-hash (1.0.0)
sensu-json (2.1.0)
sensu-logger (1.2.1)
sensu-plugin (2.3.0, 1.4.5)
sensu-plugins-cpu-checks (1.1.2)
sensu-plugins-disk-checks (2.5.0)
sensu-plugins-logstash (1.1.1)
sensu-plugins-mailer (1.2.0)
sensu-plugins-memory-checks (3.0.2)
sensu-plugins-network-checks (2.0.1)
sensu-plugins-ntp (1.0.1)
sensu-redis (2.2.0)
sensu-settings (10.3.0)
sensu-spawn (2.2.1)
sensu-transport (7.0.2)

Regards,

Noel

Answer 1 · 2017-10-07T02:53:30.000Z

@stevenviola another report of this, if we can't get this working we will have to revert the change.

Answer 2 · 2017-10-07T14:59:33.000Z

Hey @njomeara, in check_test.json, could you try changing contact to contacts? Are you only seeing the issue when adding contacts to checks, but when having contacts listed for clients, it works, or is the same issue happening for both checks and clients.

Answer 3 · 2017-10-07T15:17:31.000Z

I don't honestly see the value in having contact and contacts I think we should combine them in a breaking change (major versioned) to avoid confusion in the future. You can always still have a single contact in an array.

Answer 4 · 2017-10-07T15:56:05.000Z

@majormoses I was emulating the syntax in the sensu reference guide for checks and clients. Both contact and contacts is listed. Now that I reread that, I think the issue is, contact is singular (taking one contact), and that like the logic I ported over. @njomeara is passing an array of contacts so the contacts value should be used.

If we want to keep the compatibility with Sensu Enterprise, I could document the difference between contacts and contact more clearly to avoid confusion.

Answer 5 · 2017-10-07T16:01:34.000Z

Ya it makes total sense to me and I feel the same way about that with handler vs handlers in check definitions. All it does is add confusion. I think we could catch this scenario and throw an unknown when passing a single one to it. Thoughts?

Answer 6 · 2017-10-07T16:42:40.000Z

@stevenviola I just opened a PR to help users with this type of misconfiguration as it is a fairly easy to trap. Once it has been reviewed, merged, and released we can have the user test and see if it catches their scenario and tells them how to update it. If that goes well and this works as expected hopefully we can put this to rest.

Answer 7 · 2017-10-21T18:42:40.000Z

@njomeara can you please try using 2.0.0 which I just released to attempt to catch this misconfiguration?

Answer 8 · 2017-10-24T08:38:47.000Z

@majormoses @stevenviola, I updated the sensu-plugins-mailer to 2.0.0 and tried again with this config in the check ...
"contact": [
"team1",
"team2"
]
Again it didn't work. I changed to "contacts" in the check:
"contacts": [
"team1",
"team2"
]
and now it works as expected. I tried both "contact" and "contacts" before upgrading the gem and it still didn't work so the new 2.0.0. version has fixed the issue. I agree that you should be able to use contact and contacts interchangeably, difficulty one to spot when you leave the "s" of contacts.

Thank you for your support

Answer 9 · 2017-10-24T15:47:24.000Z

@njomeara did you see anything in your server logs that would have indicated that there was an issue? The goal was to make it easier to spot the issue rather than merge the two. I added this: https://github.com/sensu-plugins/sensu-plugins-mailer/blob/2.0.0/bin/handler-mailer.rb#L148-L153 we should probably add a "Gotcha" in the readme to warn users about this behavior.

Answer 10 · 2017-11-03T16:48:18.000Z

@majormoses. Debug logs below. First run with "contact" and second run with "contacts". don't see anything in server.log (api.log had nothing either). Note the send run with "contacts" sent to two email addresses but the first run with "contact" only sent to one address

[root@ieatoskca-sensu3 scripts]# echo '{ "name": "dodgy_check", "output": "testing email alerts", "status": 1, "handler": "mailer","contact":["support","support"] }' > /dev/tcp/localhost/3030

[root@ieatoskca-sensu3 ~]# tail -f /var/log/sensu/sensu-server.log | grep dodgy
{"timestamp":"2017-11-03T16:36:10.492399+0000","level":"debug","message":"received result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contact":["support","support"],"executed":1509726970,"issued":1509726970}}}
{"timestamp":"2017-11-03T16:36:10.492487+0000","level":"debug","message":"processing result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contact":["support","support"],"executed":1509726970,"issued":1509726970}}}
{"timestamp":"2017-11-03T16:36:10.492856+0000","level":"debug","message":"storing check result","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contact":["support","support"],"executed":1509726970,"issued":1509726970,"type":"standard"}}
{"timestamp":"2017-11-03T16:36:10.494016+0000","level":"info","message":"processing event","event":{"client":{"name":"ieatoskca-sensu3","address":"xxx.xxx.xxx.xxx","environment":"production","mail_to":[" joeblogs@acme.com "],"subscriptions":["rabbitmq_servers","redis_servers","sensu_masters","client:ieatoskca-sensu3"],"version":"1.1.1","timestamp":1509726954},"check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contact":["support","support"],"executed":1509726970,"issued":1509726970,"type":"standard","history":["1","0","1"],"total_state_change":0},"occurrences":1,"occurrences_watermark":1,"action":"create","timestamp":1509726970,"id":"2044e967-1809-4684-a54b-bfff721626ed","last_ok":1509726970,"last_state_change":1509726970,"silenced":false,"silenced_by":[]}}
{"timestamp":"2017-11-03T16:36:10.494378+0000","level":"debug","message":"handling event","event_data":"{\"client\":{\"name\":\"ieatoskca-sensu3\",\"address\":\"xxx.xxx.xxx.xxx\",\"environment\":\"production\",\"mail_to\":[\" joeblogs@acme.com \"],\"subscriptions\":[\"rabbitmq_servers\",\"redis_servers\",\"sensu_masters\",\"client:ieatoskca-sensu3\"],\"version\":\"1.1.1\",\"timestamp\":1509726954},\"check\":{\"name\":\"dodgy_check\",\"output\":\"testing email alerts\",\"status\":1,\"handler\":\"mailer\",\"contact\":[\"support\",\"support\"],\"executed\":1509726970,\"issued\":1509726970,\"type\":\"standard\",\"history\":[\"1\",\"0\",\"1\"],\"total_state_change\":0},\"occurrences\":1,\"occurrences_watermark\":1,\"action\":\"create\",\"timestamp\":1509726970,\"id\":\"2044e967-1809-4684-a54b-bfff721626ed\",\"last_ok\":1509726970,\"last_state_change\":1509726970,\"silenced\":false,\"silenced_by\":[]}","event":{"id":"2044e967-1809-4684-a54b-bfff721626ed"},"handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html","filter":"state_change_only","severities":["ok","warning","critical","unknown"],"handle_silenced":true,"handle_flapping":true,"name":"mailer"}}
{"timestamp":"2017-11-03T16:36:11.286134+0000","level":"info","message":"handler output","handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html","filter":"state_change_only","severities":["ok","warning","critical","unknown"],"handle_silenced":true,"handle_flapping":true,"name":"mailer"},"event":{"id":"2044e967-1809-4684-a54b-bfff721626ed"},"output":["mail -- sent alert for ieatoskca-sensu3/dodgy_check to  joeblogs@acme.com \n"]}

[root@ieatoskca-sensu3 scripts]# echo '{ "name": "dodgy_check", "output": "testing email alerts", "status": 1, "handler": "mailer","contacts":["support","support"] }' > /dev/tcp/localhost/3030

[root@ieatoskca-sensu3 ~]# tail -f /var/log/sensu/sensu-server.log | grep dodgy
{"timestamp":"2017-11-03T16:39:10.513673+0000","level":"debug","message":"received result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509727150,"issued":1509727150}}}
{"timestamp":"2017-11-03T16:39:10.513844+0000","level":"debug","message":"processing result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509727150,"issued":1509727150}}}
{"timestamp":"2017-11-03T16:39:10.514320+0000","level":"debug","message":"storing check result","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509727150,"issued":1509727150,"type":"standard"}}
{"timestamp":"2017-11-03T16:39:10.515456+0000","level":"info","message":"processing event","event":{"client":{"name":"ieatoskca-sensu3","address":"xxx.xxx.xxx.xxx","environment":"production","mail_to":[" joeblogs@acme.com "],"subscriptions":["rabbitmq_servers","redis_servers","sensu_masters","client:ieatoskca-sensu3"],"version":"1.1.1","timestamp":1509727134},"check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509727150,"issued":1509727150,"type":"standard","history":["1","0","1","1","0","1"],"total_state_change":0},"occurrences":1,"occurrences_watermark":1,"action":"create","timestamp":1509727150,"id":"d6e32dc3-01a2-4fa7-8d79-63c9d64d223e","last_ok":1509727150,"last_state_change":1509727150,"silenced":false,"silenced_by":[]}}
{"timestamp":"2017-11-03T16:39:10.515817+0000","level":"debug","message":"handling event","event_data":"{\"client\":{\"name\":\"ieatoskca-sensu3\",\"address\":\"xxx.xxx.xxx.xxx\",\"environment\":\"production\",\"mail_to\":[\" joeblogs@acme.com \"],\"subscriptions\":[\"rabbitmq_servers\",\"redis_servers\",\"sensu_masters\",\"client:ieatoskca-sensu3\"],\"version\":\"1.1.1\",\"timestamp\":1509727134},\"check\":{\"name\":\"dodgy_check\",\"output\":\"testing email alerts\",\"status\":1,\"handler\":\"mailer\",\"contacts\":[\"support\",\"support\"],\"executed\":1509727150,\"issued\":1509727150,\"type\":\"standard\",\"history\":[\"1\",\"0\",\"1\",\"1\",\"0\",\"1\"],\"total_state_change\":0},\"occurrences\":1,\"occurrences_watermark\":1,\"action\":\"create\",\"timestamp\":1509727150,\"id\":\"d6e32dc3-01a2-4fa7-8d79-63c9d64d223e\",\"last_ok\":1509727150,\"last_state_change\":1509727150,\"silenced\":false,\"silenced_by\":[]}","event":{"id":"d6e32dc3-01a2-4fa7-8d79-63c9d64d223e"},"handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html","filter":"state_change_only","severities":["ok","warning","critical","unknown"],"handle_silenced":true,"handle_flapping":true,"name":"mailer"}}
{"timestamp":"2017-11-03T16:39:11.487711+0000","level":"info","message":"handler output","handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html","filter":"state_change_only","severities":["ok","warning","critical","unknown"],"handle_silenced":true,"handle_flapping":true,"name":"mailer"},"event":{"id":"d6e32dc3-01a2-4fa7-8d79-63c9d64d223e"},"output":["mail -- sent alert for ieatoskca-sensu3/dodgy_check to  joeblogs@acme.com , joeblogs@acme.com\n"]}

[root@ieatoskca-sensu3 scripts]# /opt/sensu/embedded/bin/gem list | grep mailer
sensu-plugins-mailer (2.0.0)
[root@ieatoskca-sensu3 scripts]# rpm -qa | grep sensu
sensu-1.1.1-1.el7.x86_64

Answer 11 · 2017-11-03T17:39:22.000Z

So the second one is working just fine, what is confusing me here is that you are not triggering this: https://github.com/sensu-plugins/sensu-plugins-mailer/blob/2.0.0/bin/handler-mailer.rb#L148-L153 on the first one.

Answer 12 · 2017-11-03T20:57:00.000Z

OK I think I figured it out thanks to @cwjohnston who helped me realize whats going on. As you are specifying it from the check as contact but in your config you have contacts the most reliable check that we can add is to inspect the event data as well to determine if you are overwriting the config in the event. I will try to get that done this weekend.

Answer 13 · 2017-11-04T02:27:37.000Z

@njomeara can you try pulling that handler in and seeing if we can trigger something to help user to properly configure.

Answer 14 · 2017-11-06T15:07:03.000Z

Now it looks like mail is not sent at all when "contact" is used

[root@ieatoskca-sensu3 bin]# echo '{ "name": "dodgy_check", "output": "testing email alerts", "status": 0, "handler": "mailer","contact":["support","support"] }' > /dev/tcp/localhost/3030

[root@ieatoskca-sensu3 ~]# tail -f /var/log/sensu/sensu-server.log | grep dodgy
{"timestamp":"2017-11-06T12:39:00.350826+0000","level":"debug","message":"received result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":0,"handler":"mailer","contact":["support","support"],"executed":1509971940,"issued":1509971940}}}
{"timestamp":"2017-11-06T12:39:00.351002+0000","level":"debug","message":"processing result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":0,"handler":"mailer","contact":["support","support"],"executed":1509971940,"issued":1509971940}}}
{"timestamp":"2017-11-06T12:39:00.351511+0000","level":"debug","message":"storing check result","check":{"name":"dodgy_check","output":"testing email alerts","status":0,"handler":"mailer","contact":["support","support"],"executed":1509971940,"issued":1509971940,"type":"standard"}}
{"timestamp":"2017-11-06T12:39:00.352529+0000","level":"info","message":"processing event","event":{"client":{"name":"ieatoskca-sensu3","address":"xxx.xxx.xxx.xxx","environment":"production","mail_to":[" joebloggs@acme.com "],"subscriptions":["rabbitmq_servers","redis_servers","sensu_masters","client:ieatoskca-sensu3"],"version":"1.1.1","timestamp":1509971936},"check":{"name":"dodgy_check","output":"testing email alerts","status":0,"handler":"mailer","contact":["support","support"],"executed":1509971940,"issued":1509971940,"type":"standard","history":["1","0","1","1","0","1","0","1","0","1","0","1","0","1","1","1","0"],"total_state_change":0},"occurrences":3,"occurrences_watermark":3,"action":"resolve","timestamp":1509971940,"id":"c86052cb-3b4d-4f04-9237-25610f6fba84","last_state_change":1509971940,"last_ok":1509971940,"silenced":false,"silenced_by":[]}}
{"timestamp":"2017-11-06T12:39:00.352967+0000","level":"debug","message":"handling event","event_data":"{"client":{"name":"ieatoskca-sensu3","address":"xxx.xxx.xxx.xxx","environment":"production","mail_to":[" joebloggs@acme.com "],"subscriptions":["rabbitmq_servers","redis_servers","sensu_masters","client:ieatoskca-sensu3"],"version":"1.1.1","timestamp":1509971936},"check":{"name":"dodgy_check","output":"testing email alerts","status":0,"handler":"mailer","contact":["support","support"],"executed":1509971940,"issued":1509971940,"type":"standard","history":["1","0","1","1","0","1","0","1","0","1","0","1","0","1","1","1","0"],"total_state_change":0},"occurrences":3,"occurrences_watermark":3,"action":"resolve","timestamp":1509971940,"id":"c86052cb-3b4d-4f04-9237-25610f6fba84","last_state_change":1509971940,"last_ok":1509971940,"silenced":false,"silenced_by":[]}","event":{"id":"c86052cb-3b4d-4f04-9237-25610f6fba84"},"handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html","filter":"state_change_only","severities":["ok","warning","critical","unknown"],"handle_silenced":true,"handle_flapping":true,"name":"mailer"}}

Changing to "contacts"

[root@ieatoskca-sensu3 bin]# echo '{ "name": "dodgy_check", "output": "testing email alerts", "status": 1, "handler": "mailer","contacts":["support","support"] }' > /dev/tcp/localhost/3030

[root@ieatoskca-sensu3 ~]# tail -f /var/log/sensu/sensu-server.log | grep dodgy
{"timestamp":"2017-11-06T12:40:08.647702+0000","level":"debug","message":"received result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509972008,"issued":1509972008}}}
{"timestamp":"2017-11-06T12:40:08.647818+0000","level":"debug","message":"processing result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509972008,"issued":1509972008}}}
{"timestamp":"2017-11-06T12:40:08.648562+0000","level":"debug","message":"storing check result","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509972008,"issued":1509972008,"type":"standard"}}
{"timestamp":"2017-11-06T12:40:08.649511+0000","level":"info","message":"processing event","event":{"client":{"name":"ieatoskca-sensu3","address":"xxx.xxx.xxx.xxx","environment":"production","mail_to":[" joebloggs@acme.com "],"subscriptions":["rabbitmq_servers","redis_servers","sensu_masters","client:ieatoskca-sensu3"],"version":"1.1.1","timestamp":1509971996},"check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509972008,"issued":1509972008,"type":"standard","history":["1","0","1","1","0","1","0","1","0","1","0","1","0","1","1","1","0","1"],"total_state_change":0},"occurrences":1,"occurrences_watermark":1,"action":"create","timestamp":1509972008,"id":"d1d751b6-1b39-4806-be38-c66abedc9a6d","last_ok":1509972008,"last_state_change":1509972008,"silenced":false,"silenced_by":[]}}
{"timestamp":"2017-11-06T12:40:08.649858+0000","level":"debug","message":"handling event","event_data":"{"client":{"name":"ieatoskca-sensu3","address":"xxx.xxx.xxx.xxx","environment":"production","mail_to":[" joebloggs@acme.com "],"subscriptions":["rabbitmq_servers","redis_servers","sensu_masters","client:ieatoskca-sensu3"],"version":"1.1.1","timestamp":1509971996},"check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509972008,"issued":1509972008,"type":"standard","history":["1","0","1","1","0","1","0","1","0","1","0","1","0","1","1","1","0","1"],"total_state_change":0},"occurrences":1,"occurrences_watermark":1,"action":"create","timestamp":1509972008,"id":"d1d751b6-1b39-4806-be38-c66abedc9a6d","last_ok":1509972008,"last_state_change":1509972008,"silenced":false,"silenced_by":[]}","event":{"id":"d1d751b6-1b39-4806-be38-c66abedc9a6d"},"handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html","filter":"state_change_only","severities":["ok","warning","critical","unknown"],"handle_silenced":true,"handle_flapping":true,"name":"mailer"}}
{"timestamp":"2017-11-06T12:40:09.603993+0000","level":"info","message":"handler output","handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html","filter":"state_change_only","severities":["ok","warning","critical","unknown"],"handle_silenced":true,"handle_flapping":true,"name":"mailer"},"event":{"id":"d1d751b6-1b39-4806-be38-c66abedc9a6d"},"output":["mail -- sent alert for ieatoskca-sensu3/dodgy_check to joebloggs@acme.com , joebloggs@acme.com\n"]}

i checked my handler-mailer.rb below to make sure it had the code change ..

[root@ieatoskca-sensu3 bin]# grep src /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-mailer-2.0.0/bin/handler-mailer.rb
def raise_contact_is_array(src)
msg = "you passed #{src} which is meant to be a string, "

Answer 15 · 2017-11-06T16:31:49.000Z

Well that is progress I guess, that means we have successfully trapped the scenario. The thing that does not make sense is that there is not a log messaging indicating what is happening. @cwjohnston any thoughts?