AlertManager - Bad Endpoint

Describe the bug
Trying to find a good way to troubleshoot the integration with Alertmanager.

Falcosidekick throws the error below:

[falco-falcosidekick-5f4d5967f-4bj5w] 2022/02/06 17:03:32 [INFO]  : WebUI - Post OK (200) 
[falco-falcosidekick-5f4d5967f-75sjk] 2022/02/06 17:03:33 [INFO]  : WebUI - Post OK (200) 
[falco-falcosidekick-6477545dcc-mcg7r] 2022/02/06 17:03:39 [ERROR] : AlertManager - Bad Endpoint 
[falco-falcosidekick-6477545dcc-mcg7r] 2022/02/06 17:03:39 [INFO]  : Enabled Outputs : [WebUI] 
[falco-falcosidekick-5f4d5967f-4bj5w] 2022/02/06 17:03:33 [INFO]  : WebUI - Post OK (200) 
[falco-falcosidekick-5f4d5967f-75sjk] 2022/02/06 17:03:35 [INFO]  : WebUI - Post OK (200) 
[falco-falcosidekick-6477545dcc-mcg7r] 2022/02/06 17:03:39 [INFO]  : Falco Sidekick is up and listening on :2801 
[falco-falcosidekick-5f4d5967f-4bj5w] 2022/02/06 17:03:35 [INFO]  : WebUI - Post OK (200)

How to reproduce it

1 . Secret configured

apiVersion: v1
kind: Secret
metadata:
  name: falco-falcosidekick
  namespace: auditing
  labels:
    app.kubernetes.io/name: falcosidekick
type: Opaque
data:
  # AlertManager Output
  ALERTMANAGER_HOSTPORT: "YWxlcnRtYW5hZ2VyLW9wZXJhdGVkLm9ic2VydmFiaWxpdHkuc3ZjOjkwOTM="
  ALERTMANAGER_MINIMUMPRIORITY: "ZGVidWc="
  ALERTMANAGER_MUTUALTLS: "ZmFsc2U="
  ALERTMANAGER_CHECKCERT: "ZmFsc2U="
  ALERTMANAGER_ENDPOINT: "L2FwaS92Mi9hbGVydHM="   # /api/v2/alerts
  #ALERTMANAGER_ENDPOINT: "L2FwaS92MS9hbGVydHM=" # /api/v1/alerts

Checking the deployment, we can see the variable worked as expected

kubectl iexec -n auditing
Namespace: auditing | Pod: ✔ falco-falcosidekick-6477545dcc-mcg7r
/app $ env | grep ALERT
ALERTMANAGER_MINIMUMPRIORITY=debug
ALERTMANAGER_MUTUALTLS=false
ALERTMANAGER_HOSTPORT=alertmanager-operated.observability.svc:9093
ALERTMANAGER_ENDPOINT=/api/v2/alerts
ALERTMANAGER_CHECKCERT=false

It's is a valid namespace and service because if I create a new pod on the same namespace auditing it works.

$ kubectl run pod -n auditing --image=alpine:latest --command -- sh -c 'sleep 1d' 
pod/pod created

$ kubectl iexec -n auditing   
Namespace: auditing | Pod: ✔ pod
/ # curl -vvv http://alertmanager-operated.observability.svc:9093/api/v2/alerts | jq  | head 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.233.92.37:9093...
* Connected to alertmanager-operated.observability.svc (10.233.92.37) port 9093 (#0)
> GET /api/v2/alerts HTTP/1.1
> Host: alertmanager-operated.observability.svc:9093
> User-Agent: curl/7.80.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Vary: Origin
< Date: Sun, 06 Feb 2022 17:13:19 GMT
< Transfer-Encoding: chunked
< 
{ [1298 bytes data]
100  6092    0  6092    0     0  1541k      0 --:--:-- --:--:-- --:--:-- 1983k
* Connection #0 to host alertmanager-operated.observability.svc left intact
[
  {
    "annotations": {
      "description": "HTTP status code is not 200-399\n  VALUE = 0\n  LABELS: map[__name__:probe_http_status_code instance:http://content.default.svc job:blackbox_http_2xx_toca_ca]",
      "summary": "Blackbox probe HTTP failure (instance http://content.default.svc)"
    },
    "endsAt": "2022-02-06T17:16:55.143Z",
    "fingerprint": "15c3ade5fa8b069f",
    "receivers": [
      {

Same issue happens if I set /api/v1/alerts. I have not tested sending out a JSON manually because I have other services integrated with Alertmanager that are working as expected, however, I can try to create a JSON payload to test it.

Expected behaviour

Alertmanager to work

Environment

Falco version: 0.31.0

System info:

{
  "machine": "x86_64",
  "nodename": "falco-chjjf",
  "release": "3.10.0-1160.49.1.el7.x86_64",
  "sysname": "Linux",
  "version": "#1 SMP Tue Nov 30 15:51:32 UTC 2021"
}

Cloud provider or hardware configuration:
OS: CentOS7

Kernel: Linux p70 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 30 15:51:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Installation method:
Kubernetes

Additional context

Hello,

Just to be sure before digging, have you tried with ALERTMANAGER_HOSTPORT=http://alertmanager-operated.observability.svc:9093`?

Thanks

@Issif

BOOM!! Man, I can't believe it was that!

falco-falcosidekick-ui-674846548f-kjtxs
[falco-falcosidekick-6b95df7875-kqt6m] 2022/02/07 16:39:20 [INFO]  : Enabled Outputs : [AlertManager WebUI] 
[falco-falcosidekick-6b95df7875-kqt6m] 2022/02/07 16:39:20 [INFO]  : Falco Sidekick is up and listening on :2801

Should we add a note in the documentation? I would be happy

I'm happy it works now 👍

Yes please, feel free to propose an update in documentation. 🙏

Reviewing the documentation, it is already placed at

falcosidekick/README.md

Line 559 in 51ce72a

- **ALERTMANAGER_HOSTPORT** : AlertManager http://host:port, if not `empty`,

which I totally overlooked.

Things are good and thanks for the help!