Proxy restriction

Question

Proxy restriction

Closed this issue 7 years ago · 35 comments

Greetings,

We tried to implement your method in our production environment. However, the Slack webhook has been restricted on ICINGA production box. The Slack webhook is however connected to a different server.
We are trying to connect it via Reverse proxy
`
#!/bin/bash

ICINGA_HOSTNAME="nodea.example.com"
SLACK_WEBHOOK_URL="https://hooks.slack.com.slack.xxxxx.xxxxx.net/services/xxxx/sdfdsff/xxxxx"
SLACK_CHANNEL="#icinga"
export https_proxy="http://xxxxxxx.example.com:9090"
#Set the message icon based on ICINGA service state
if [ "$SERVICESTATE" = "CRITICAL" ]
then
ICON=":bomb:"
elif [ "$SERVICESTATE" = "WARNING" ]
then
ICON=":warning:"
elif [ "$SERVICESTATE" = "OK" ]
then
ICON=":beer:"
elif [ "$SERVICESTATE" = "UNKNOWN" ]
then
ICON=":question:"
else
ICON=":white_medium_square:"
fi

#Send message to Slack
PAYLOAD="payload={"channel": "${SLACK_CHANNEL}", "username": "${SLACK_BOTNAME}", "text": "${ICON} HOST: http://${ICINGA_HOSTNAME}/icingaweb2/monitoring/host/services?host=${HOSTNAME}|${HOSTDISPLAYNAME} SERVICE: <http://${ICINGA_HOSTNAME}/icingaweb2/dashboard#!/icingaweb2/monitoring/service/show?host=${HOSTNAME}&service= ${SERVICEDESC} > STATE: ${SERVICESTATE}"}"

curl -x http://xxxx.example.com:9091 --connect-timeout 30 --max-time 60 -s -S -X POST --data-urlencode "${PAYLOAD}" "${SLACK_WEBHOOK_URL}"

`

I have placed the slack-service-notification in the server, where slack webhook URL is accessible. However, the notifications are going in as a link and not in the right format. This gives just a link name with the hostname of ICINGA URL and not the "service name" and the "service state" in it

Is there anyway to bring the format as like Before ( Hostname,Service name,Service State) .

Thanks
Aravind

Answer 1 · 2017-08-28T20:44:25.000Z

Hi @ananthaa-advisory

The values are exported as environment variables by Icinga as defined in the notification command configuration files. Example: https://github.com/jjethwa/icinga2-slack-notification/blob/master/slack-service-notification-command.conf

Is the script being executed on the same server as Icinga or on another server?

Answer 2 · 2017-08-29T04:15:20.000Z

The script is executed on another server

Thanks
Aravind

Answer 3 · 2017-08-29T04:37:03.000Z

Can I use CURL command to go through other server and resolves proxy from remote machine itself

Answer 4 · 2017-08-29T20:04:16.000Z

Hi @ananthaa-advisory

Is the proxy server accessible from the Icinga server? If so, run the script with the -x proxy option on the Icinga server and it will work. Otherwise, you need to pass the environment variables to the shell that is running the script on the other server.

Answer 5 · 2017-08-29T20:53:40.000Z

Hi @jethwa,

You mean declare a new environment variables in other severs. Can you show a couple of samples.

I tried with -x option and it was not working. You can see my script and it has -x flag in curl command.

Answer 6 · 2017-08-29T20:59:10.000Z

HI @ananthaa-advisory

The -x option should work if you can connect to the proxy. Have you verified that connectivity is working?

How are you executing the script on the remote server?

Answer 7 · 2017-08-30T18:48:44.000Z

Hi @jjethwa ,

Yes, The connectivity is working fine from the other server. It is throwing notifications but with a Wrong format.

From slack it should look like this:
⚠️ HOST: node1.example .com SERVICE: http STATE: WARNING``

Instead now it is loooking like,

In place of hostname, it throws only ICINGA link for HOSTNAME and for Service and service state , it is not showing up anything.

I tried to put the ENV variables and it is showing errors

Answer 8 · 2017-08-30T18:57:19.000Z

This is my script that I run from Remote server, where I have access to the PROXY URL.

#!/bin/bash

ICINGA_HOSTNAME="sample.com"
SLACK_WEBHOOK_URL="https://hooks.slack.com.slack.xxxx"
SLACK_BOTNAME="sample"
SLACK_CHANNEL="#icinga2"
export https_proxy="http://sample.com:9090"
#Set the message icon based on ICINGA service state
if [ "$SERVICESTATE" = "CRITICAL" ]
then
    ICON=":bomb:"
elif [ "$SERVICESTATE" = "WARNING" ]
then
    ICON=":warning:"
elif [ "$SERVICESTATE" = "OK" ]
then
    ICON=":beer:"
elif [ "$SERVICESTATE" = "UNKNOWN" ]
then
    ICON=":question:"
else
    ICON=":white_medium_square:"
fi

 env = {
    "NOTIFICATIONTYPE" = "$notification.type$"
    "SERVICEDESC" = "$service.name$"
    "HOSTALIAS" = "$host.display_name$",
    "HOSTNAME" = "$host.name$",
    "HOSTADDRESS" = "$address$",
    "SERVICESTATE" = "$service.state$",
    "LONGDATETIME" = "$icinga.long_date_time$",
    "SERVICEOUTPUT" = "$service.output$",
    "NOTIFICATIONAUTHORNAME" = "$notification.author$",
    "NOTIFICATIONCOMMENT" = "$notification.comment$",
    "HOSTDISPLAYNAME" = "$host.display_name$",
    "SERVICEDISPLAYNAME" = "$service.display_name$",
  }

#Send message to Slack
PAYLOAD="payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_BOTNAME}\", \"text\": \"${ICON} HOST: <http://${ICINGA_HOSTNAME}/icingaweb2/monitoring/host/services?host=${HOSTNAME}|${HOSTDISPLAYNAME}>   SERVICE: <http://${ICINGA_HOSTNAME}/icingaweb2/dashboard#!/icingaweb2/monitoring/service/show?host=${HOSTNAME}&service= ${SERVICEDESC} >  STATE: ${SERVICESTATE}\"}"

curl -x http://sample.com:9090 --connect-timeout 30 --max-time 60 -s -S -X POST --data-urlencode "${PAYLOAD}" "${SLACK_WEBHOOK_URL}"

Answer 9 · 2017-08-30T20:59:42.000Z

Hi @ananthaa-advisory

The problem is these variables will not get set as they are set by Icinga2 when executing the script locally

env = {
    "NOTIFICATIONTYPE" = "$notification.type$"
    "SERVICEDESC" = "$service.name$"
    "HOSTALIAS" = "$host.display_name$",
    "HOSTNAME" = "$host.name$",
    "HOSTADDRESS" = "$address$",
    "SERVICESTATE" = "$service.state$",
    "LONGDATETIME" = "$icinga.long_date_time$",
    "SERVICEOUTPUT" = "$service.output$",
    "NOTIFICATIONAUTHORNAME" = "$notification.author$",
    "NOTIFICATIONCOMMENT" = "$notification.comment$",
    "HOSTDISPLAYNAME" = "$host.display_name$",
    "SERVICEDISPLAYNAME" = "$service.display_name$",
  }

How are you executing the script remotely?

Answer 10 · 2017-08-31T19:09:30.000Z

Hi @jjethwa,

I have placed this script in remote server and not from Icinga and just calling the script in remote server only, since Curl command is calling the Icinga Url by post method.

Running the script in remote server by ./slack-service-notification only.

So, how do you want me to execute the script.

i really wanted to appreciate your help on this so far.

Thanks again,
Aravind

Answer 11 · 2017-09-01T00:28:09.000Z

Hi @ananthaa-advisory

NP, glad to help 😄

So you can either adjust the script to pass the values for those environment variables as arguments (to the script) or export the variables to shell you're executing the script in and that would do the trick 😄

Answer 12 · 2017-09-01T15:34:03.000Z

Hi @jjethwa ,

I tried to export the variables in ICINGA server and it did not do anything.. Could you please advise

Answer 13 · 2017-09-01T19:43:35.000Z

Hi @ananthaa-advisory

Can you explain what you did? The env block I posted above should set the environment variables when the script is executed on the Icinga server.

Answer 14 · 2017-09-01T21:17:10.000Z

Hi @jjethwa ,

I did like this

export https_proxy="http://sample.com:9090"
#Set the message icon based on ICINGA service state
if [ "$SERVICESTATE" = "CRITICAL" ]
then
    ICON=":bomb:"
elif [ "$SERVICESTATE" = "WARNING" ]
then
    ICON=":warning:"
elif [ "$SERVICESTATE" = "OK" ]
then
    ICON=":beer:"
elif [ "$SERVICESTATE" = "UNKNOWN" ]
then
    ICON=":question:"
else
    ICON=":white_medium_square:"
fi
 export "NOTIFICATIONTYPE" = "$notification.type$"
   export "HOSTALIAS" = "$host.display_name$"
   export  "HOSTADDRESS" = "$address$"
   export "HOSTNAME" = "$host.name$"
   export "HOSTSTATE" = "$host.state$"
   export  "LONGDATETIME" = "$icinga.long_date_time$"
  export  "HOSTOUTPUT" = "$host.output$"
   export "NOTIFICATIONAUTHORNAME" = "$notification.author$"
   export "NOTIFICATIONCOMMENT" = "$notification.comment$"
    export "HOSTDISPLAYNAME" = "$host.display_name$"
   export  "USEREMAIL" = "$user.email$"

#Send message to Slack
#PAYLOAD="payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_BOTNAME}\", \"text\": \"${ICON} HOST: <http://${ICINGA_HOSTNAME}/icingaweb2/monitoring/host/services?host=${HOSTNAME}|${HOSTDISPLAYNAME}>   SERVICE: <http://${ICINGA_HOSTNAME}/icingaweb2/dashboard#!/icingaweb2/monitoring/service/show?host=${HOSTNAME}&service= ${SERVICEDESC} >  STATE: ${SERVICESTATE}\"}" | cat /etc/icinga2/conf.d/commands.conf
#
PAYLOAD="payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_BOTNAME}\", \"text\": \"${ICON} HOST: <http://${ICINGA_HOSTNAME}/icingaweb2/monitoring/host/services?host=${HOSTNAME}|${HOSTDISPLAYNAME}>   SERVICE: <http://${ICINGA_HOSTNAME}/icingaweb2/monitoring/service/show?host=${HOSTNAME}&service=${SERVICENAME} >  STATE: ${SERVICESTATE}\"}"


curl -x http://sample.com:9090 --connect-timeout 30 --max-time 60 -s -S -X POST --data-urlencode "${PAYLOAD}" "${SLACK_WEBHOOK_URL}"

I executed the above script from the remote server. When I tried to export the "env" variables on the ICINGA server from the commands.conf file. it throwed an error. so I then, exported the variables from the remote script itself.

Thanks
Aravind

Answer 15 · 2017-09-01T21:38:59.000Z

Hi @ananthaa-advisory

That won't work as the variables like $host.name$ are set by Icinga so you need it to be parsed on the Icinga server itself then export it to the shell (or pass as a variable) running the script on the remote server.

I'm still unclear as to how the Icinga server is communicating with the remote server that the script is executed on. Can you describe that?

Answer 16 · 2017-09-01T21:47:46.000Z

Hi @jjethwa ,

so , how do you want to export the variable in ICINGA server,

Just on the shell prompt:

#export "HOSTALIAS" = "$host.display_name$",so I need to export that as a permanent variable as well ? should do ?

Answer 17 · 2017-09-01T21:49:36.000Z

Icinga server will not communicate with the remote server, we make Post methods calls for the ICINGA server URL , thats how we get the output, is it right?

thanks
Aravind

Answer 18 · 2017-09-01T22:12:29.000Z

All I m doing is just placing the slack-service-notication script in remote server and exciting it. The notification are posted as curl link to slack channel instead of host and service name.

If you parse the variable in icinga server. Then do you want to export the env variable in icinga server itself.

Answer 19 · 2017-09-01T22:50:04.000Z

@ananthaa-advisory

Sorry, I'm still confused as to the setup 😢 The script from #9 (comment) is that executing on the Icinga server?

Answer 20 · 2017-09-02T21:08:32.000Z

HI @jjethwa ,

I have tried adding the -x option and running it on ICINGA server.This is the error message I get

[aravind@ICINGA2 ~]$ ./service
curl: (56) Received HTTP code 403 from proxy after CONNECT

The below is the script I am running:

#!/bin/bash

ICINGA_HOSTNAME="sample.sample.com"
#SLACK_WEBHOOK_URL="https://hooks.slack.com.slack.sample.shnpoc.net/services/TEpqhlBMjVwuO1"
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/T56t1JUpYr56lo1K0tj"
SLACK_BOTNAME="try"
SLACK_CHANNEL="slack"
export https_proxy="https://proxy-appgw.sample.com:9090"
#Set the message icon based on ICINGA service state
if [ "$SERVICESTATE" = "CRITICAL" ]
then
    ICON=":x:"
elif [ "$SERVICESTATE" = "WARNING" ]
then
    ICON=":warning:"
elif [ "$SERVICESTATE" = "OK" ]
then
    ICON=":white_check_mark:"
elif [ "$SERVICESTATE" = "UNKNOWN" ]
then
    ICON=":question:"
else
    ICON=":white_medium_square:"
fi

#Send message to Slack
PAYLOAD="payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_BOTNAME}\", \"text\": \"${ICON} HOST: <http://${ICINGA_HOSTNAME}/icingaweb2/monitoring/host/services?host=${HOSTNAME}|${HOSTDISPLAYNAME}>   SERVICE: <http://${ICINGA_HOSTNAME}/icingaweb2/monitoring/service/show?host=${HOSTNAME}&service=${SERVICEDESC}|${SERVICEDISPLAYNAME}>  STATE: ${SERVICESTATE}\"}"

curl -x https://proxy-appgw.sample.com:9090 --connect-timeout 30 --max-time 60 -s -S -X POST --data-urlencode "${PAYLOAD}" "${SLACK_WEBHOOK_URL}"

Answer 21 · 2017-09-05T20:28:48.000Z

Hi @ananthaa-advisory

Getting closer! Looks like the proxy server is returning an HTTP 403. Do you need to pass credentials?

--proxy-user <USERNAME>:<PASSWORD>

Answer 22 · 2017-09-05T20:59:24.000Z

Hi @jjethwa,

No , we don't have credentials for proxy server. The same proxy server we use in Dev environment. It does not use any credentials.

As far as I know , curl command Wil only work if the fire wall is opened.

Answer 23 · 2017-09-05T21:03:10.000Z

Hi @ananthaa-advisory

Any errors in the proxy server log?

Answer 24 · 2017-09-05T21:16:41.000Z

Hi @jjethwa,

For proxy server only Network team has access. They won't share it easily.

I assume the logs would also say the production icinga box can't connect to proxy url.

Answer 25 · 2017-09-05T21:19:23.000Z

I googled about this. I thought of running the script From icinga and pass the proxy to remote server. As remote server can easily resolve or connect to proxy ip.

But no documents showing of connecting proxy ip , by passing two servers.

Answer 26 · 2017-09-05T21:28:52.000Z

Hi @ananthaa-advisory

You would need to chain the proxy server connection, but that's a messy setup. I think it would be best to find out why the proxy server is returning an HTTP 403. Maybe your source IP needs to be whitelisted, etc Not sure how they set up the proxy server 😞

Answer 27 · 2017-09-07T01:22:09.000Z

Hi @jjethwa,

Thanks for help. :) remember you said about second option to export the env variables.

Can you just Tel me how do export the icinga variables and Call those variable from remote server with the script.

Thanks

Answer 28 · 2017-09-07T18:42:25.000Z

Hi @ananthaa-advisory

Here are a few examples: https://superuser.com/questions/48783/how-can-i-pass-an-environment-variable-through-an-ssh-command

Answer 29 · 2017-09-07T21:16:11.000Z

Hi @jjethwa,

Thats fine, that works for a normal variable. however, when i tried to export the ICINGA variable. it throws an error:

# export "NOTIFICATIONTYPE" = "$notification.type$"
-bash: export: `=': not a valid identifier
-bash: export: `.type$': not a valid identifier

Answer 30 · 2017-09-07T21:17:48.000Z

I executed locally inside the ICINGA server only. For locally itself is not working for ICINGA variables.so , how do you exactly call them

IF you could just show me one example, that helps a lot. Thanks :)

Answer 31 · 2017-09-07T21:27:08.000Z

Hi @ananthaa-advisory

For my setup, I just use the conf and script files that are in in this git repo. Since the slack notification script is executed by Icinga2, it sets the variables properly. Can you try reverting the script on the Icinga2 server and trying again?

Answer 32 · 2017-09-08T03:51:10.000Z

HI @jjethwa ,

I m trying from ICINGA server only

Answer 33 · 2017-09-08T18:19:23.000Z

Hi @ananthaa-advisory

Let's try reverting the script back to what's in this repo then modify the vars at the top

ICINGA_HOSTNAME="nodea.example.com"
SLACK_WEBHOOK_URL="https://hooks.slack.com.slack.xxxxx.xxxxx.net/services/xxxx/sdfdsff/xxxxx"
SLACK_CHANNEL="#icinga"

You can then try adding the proxy flag to the curl at the end or maybe add an echo in front to confirm the variables are set, then we can take it from there 😄

Answer 34 · 2017-09-12T06:49:30.000Z

Good news, I have the ticket approved by the N/W team and the alerts are coming fine now.

Thank you so much for all your help sir :)

Answer 35 · 2017-09-12T19:15:52.000Z

That's great news, @ananthaa-advisory 😃