[BUG] Sending a message with double quotes in TeamsAlert fields breaks MS Teams communication
earthquakesan opened this issue · 1 comments
earthquakesan commented
Description
The following message from alertmanager is not communicated properly to the MS Teams webhook:
2021-08-12 12:42:35,292 - prom2teams - INFO - JSON received is:
{'receiver': 'teams', 'groupLabels': {'alertname': 'unknown', 'instance': 'unknown', 'severity': 'unknown'}, 'externalURL': 'http://rancher-monitoring-alertmanager.cattle-monitoring-system:9093', 'status': 'firing', 'commonAnnotations': {'summary': 'unknown', 'description': 'unknown'}, 'groupKey': '{}/{alertname="Watchdog"}:{}', 'alerts': [{'fingerprint': '7109567b88e9eb6c', 'startsAt': datetime.datetime(2021, 8, 11, 15, 27, 2), 'status': 'firing', 'annotations': {'summary': 'unknown', 'description': 'unknown', 'message': 'This is an alert meant to ensure that the entire alerting pipeline is functional.\nThis alert is always firing, therefore it should always be firing in Alertmanager\nand always fire against a receiver. There are integrations with various notification\nmechanisms that send a notification when this alert is not firing. For example the\n"DeadMansSnitch" integration in PagerDuty.\n'}, 'generatorURL': 'http://rancher-monitoring-prometheus.cattle-monitoring-system:9090/graph?g0.expr=vector%281%29&g0.tab=1', 'labels': {'alertname': 'Watchdog', 'instance': 'unknown', 'severity': 'none', 'prometheus': 'cattle-monitoring-system/rancher-monitoring-prometheus'}, 'endsAt': datetime.datetime(1, 1, 1, 0, 0)}], 'commonLabels': {'alertname': 'Watchdog', 'instance': 'unknown', 'severity': 'none'}, 'version': '4', 'truncatedAlerts': 0}
2021-08-12 12:42:35,394 - prom2teams - INFO - {
"@type": "MessageCard",
"@context": "http://schema.org/extensions",
"themeColor": " ",
"summary": "unknown",
"title": "Prometheus alert ",
"sections": [{
"activityTitle": "unknown",
"facts": [{
"name": "Alert",
"value": "Watchdog"
},{
"name": "In host",
"value": "unknown"
},{
"name": "Severity",
"value": "none"
},{
"name": "Description",
"value": "unknown"
},{
"name": "Status",
"value": "firing"
},{
"name": "prometheus",
"value": "cattle-monitoring-system/rancher-monitoring-prometheus"
} ,{
"name": "message",
"value": "This is an alert meant to ensure that the entire alerting pipeline is functional.
This alert is always firing, therefore it should always be firing in Alertmanager
and always fire against a receiver. There are integrations with various notification
mechanisms that send a notification when this alert is not firing. For example the
"DeadMansSnitch" integration in PagerDuty.
"
}],
"markdown": true
}]
}
The problem is that the double quotes around "DeadMansSnitch" are not escaped.
Steps to Reproduce
Configuration for the prom2teams:
[Microsoft Teams]
Connector: https://connection-uri
Script to reproduce the issue:
MESSAGE_2="{
\"receiver\": \"teams\",
\"commonAnnotations\": {\"summary\": \"unknown\", \"description\": \"unknown\"},
\"groupKey\": \"{}/{alertname=\\\"Watchdog\\\"}:{}\",
\"status\": \"firing\",
\"externalURL\": \"http://rancher-monitoring-alertmanager.cattle-monitoring-system:9093\",
\"groupLabels\": {\"instance\": \"unknown\", \"alertname\": \"unknown\", \"severity\": \"unknown\"},
\"alerts\": [
{
\"startsAt\": \"2020-09-16T07:38:01.586706006Z\",
\"status\": \"firing\",
\"generatorURL\": \"http://rancher-monitoring-prometheus.cattle-monitoring-system:9090/graph?g0.expr=vector%281%29&g0.tab=1\",
\"annotations\": {
\"summary\": \"unknown\",
\"description\": \"unknown\",
\"message\": \"This is an alert meant to ensure that the entire alerting pipeline is functional.\nThis alert is always firing, therefore it should always be firing in Alertmanager\nand always fire against a receiver. There are integrations with various notification\nmechanisms that send a notification when this alert is not firing. For example the\n\\\"DeadMansSnitch\\\" integration in PagerDuty.\n\"
},
\"fingerprint\": \"7109567b88e9eb6c\",
\"labels\": {
\"instance\": \"unknown\",
\"alertname\": \"Watchdog\",
\"severity\": \"none\",
\"prometheus\": \"cattle-monitoring-system/rancher-monitoring-prometheus\"
},
\"endsAt\": \"2020-09-16T07:38:01.586706006Z\"
}
],
\"version\": \"4\",
\"commonLabels\": {
\"instance\": \"unknown\",
\"alertname\": \"Watchdog\",
\"severity\": \"none\"
},
\"truncatedAlerts\": 0
}"
curl -w "%{http_code}\n" -X POST -H "accept: application/json" -H "Content-Type: application/json" -d "${MESSAGE_2}" "http://localhost:8089/v2/Connector"
Expected behavior:
I can see a message in our MS Teams channel
Actual behavior:
2021-08-12 12:42:45,414 - werkzeug - ERROR - Error on request:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/werkzeug/serving.py", line 306, in run_wsgi
execute(self.server.app)
File "/usr/local/lib/python3.8/site-packages/werkzeug/serving.py", line 294, in execute
application_iter = app(environ, start_response)
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2309, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2295, in wsgi_app
response = self.handle_exception(e)
File "/usr/local/lib/python3.8/site-packages/flask_restplus/api.py", line 583, in error_router
return original_handler(e)
File "/usr/local/lib/python3.8/site-packages/flask_restplus/api.py", line 583, in error_router
return original_handler(e)
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1741, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 35, in reraise
raise value
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.8/site-packages/flask_restplus/api.py", line 583, in error_router
return original_handler(e)
File "/usr/local/lib/python3.8/site-packages/flask_restplus/api.py", line 583, in error_router
return original_handler(e)
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 35, in reraise
raise value
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.8/site-packages/flask_restplus/api.py", line 325, in wrapper
resp = resource(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/flask/views.py", line 88, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/flask_restplus/resource.py", line 44, in dispatch_request
resp = meth(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prom2teams/app/versions/v2/namespace.py", line 27, in post
self.sender.send_alerts(alerts, app.config['MICROSOFT_TEAMS'][connector])
File "/usr/local/lib/python3.8/site-packages/prom2teams/app/sender.py", line 26, in send_alerts
self.teams_client.post(teams_webhook_url, team_alert)
File "/usr/local/lib/python3.8/site-packages/prom2teams/app/teams_client.py", line 41, in post
simple_post(teams_webhook_url, message)
File "/usr/local/lib/python3.8/site-packages/prom2teams/app/teams_client.py", line 35, in simple_post
self._do_post(teams_webhook_url, message)
File "/usr/local/lib/python3.8/site-packages/prom2teams/app/teams_client.py", line 55, in _do_post
raise MicrosoftTeamsRequestException(
prom2teams.app.exceptions.MicrosoftTeamsRequestException: Error performing request to: {}.
Returned status code: {}.
Returned data: {}
Sent message: {}
Reproduces how often:
In all the alerts where quotes are present.
Environment
- The release version/s you are using: latest develop
- OS: MacOs (virtualenv + python3.9)
- Others: initially issue is encountered in k8s environment
Additional Information
[Any additional information, configuration or data that might be necessary to reproduce the issue.]
stale commented
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.