crazy-max/diun

Error "invalid character '<' looking for beginning of value" when trying to use ntfy

Closed this issue · 5 comments

Kalooth commented

Support guidelines

I've found a bug and checked that ...

  • ... the documentation does not mention anything about my problem
  • ... there are no open or closed issues that are related to my problem

Description

When trying to use ntfy for notifications, it always throws the error:

ERR Ntfy notification failed error="invalid character '<' looking for beginning of value" image=docker.io/netdata/netdata:latest

This even occurs when attempting to use the test command:
docker compose exec diun diun notif test

diun: error: main.NotifTestCmd.Run(): rpc error: code = Unknown desc = invalid character '<' looking for beginning of value

Running via docker compose (replaced _ENDPOINT with my endpoint):
version: "3.5"

services:
diun:
image: crazymax/diun:latest
command: serve
volumes:
- "./data:/data"
- "/var/run/docker.sock:/var/run/docker.sock"
environment:
- "TZ=America/Chicago"
- "LOG_LEVEL=debug"
- "DIUN_WATCH_WORKERS=20"
- "DIUN_WATCH_SCHEDULE=0 */6 * * *"
- "DIUN_WATCH_JITTER=30s"
- "DIUN_PROVIDERS_DOCKER=true"
- "DIUN_PROVIDERS_DOCKER_WATCHBYDEFAULT=true"
- "DIUN_PROVIDERS_DOCKER_WATCHSTOPPED=true"
- "DIUN_WATCH_FIRSTCHECKNOTIF=true"
- "DIUN_NOTIF_NTFY_ENDPOINT=https://ntfy.example.com"
- "DIUN_NOTIF_NTFY_TOPIC=diun_updates"
labels:
- "diun.enable=true"
restart: unless-stopped

Expected behaviour

No errors when pushing notifications via ntfy

Actual behaviour

Error: code = Unknown desc = invalid character '<' looking for beginning of value

Steps to reproduce

  1. Setup self hosted ntfy instance
  2. Setup diun via docker compose, connect endpoint to ntfy instance
  3. Execute the test notify command - docker compose exec diun diun notif test
  4. Notice error regarding invalid character '<'

Diun version

4.26

Docker info

Client: Docker Engine - Community
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 23
  Running: 20
  Paused: 0
  Stopped: 3
 Images: 23
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.0-26-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.56GiB
 Name: p
 ID: eda8ca4c-10c4-4f07-8d29-179da8f2138d
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Docker Compose config

name: diun
services:
  diun:
    command:
    - serve
    environment:
      DIUN_NOTIF_NTFY_ENDPOINT: https://ntfy.example.com
      DIUN_NOTIF_NTFY_TOPIC: diun_updates
      DIUN_PROVIDERS_DOCKER: "true"
      DIUN_PROVIDERS_DOCKER_WATCHBYDEFAULT: "true"
      DIUN_PROVIDERS_DOCKER_WATCHSTOPPED: "true"
      DIUN_WATCH_FIRSTCHECKNOTIF: "true"
      DIUN_WATCH_JITTER: 30s
      DIUN_WATCH_SCHEDULE: 0 */6 * * *
      DIUN_WATCH_WORKERS: "20"
      LOG_LEVEL: debug
      TZ: America/Chicago
    image: crazymax/diun:latest
    labels:
      diun.enable: "true"
    networks:
      default: null
    restart: unless-stopped
    volumes:
    - type: bind
      source: /home/n/Documents/diun/data
      target: /data
      bind:
        create_host_path: true
    - type: bind
      source: /var/run/docker.sock
      target: /var/run/docker.sock
      bind:
        create_host_path: true
networks:
  default:
    name: diun_default

Logs

n/a

Additional info

No response

crazy-max commented

@Kalooth Do you repro with crazymax/diun:4.25.0?

Kalooth commented

Pulled 4.25.0 and getting the same error both naturally and through the test command.

crazy-max commented

I don't repro with https://ntfy.sh so might be an issue with your self-hosted instance: https://ntfy.example.com

sNiXx commented

I was seeing the same issue and it took me some time to figure out what was going on. The issue is related to the response parser. I tested with curl from within the diun container and received the following response body:

<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>

The parser was expecting a json, but got an html instead and thus complained about the "<" when it was trying to parse.

In my case, this was due to settings of my web server that disallowed connections from other containers. Hope this helps anyone who might be running into this issue.

@crazy-max: Does it make sense to improve response handling? Or is this an upstream issue? Sorry, I am not at all familiar with Go...

crazy-max commented

I was seeing the same issue and it took me some time to figure out what was going on. The issue is related to the response parser. I tested with curl from within the diun container and received the following response body:

<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>

The parser was expecting a json, but got an html instead and thus complained about the "<" when it was trying to parse.

In my case, this was due to settings of my web server that disallowed connections from other containers. Hope this helps anyone who might be running into this issue.

@crazy-max: Does it make sense to improve response handling? Or is this an upstream issue? Sorry, I am not at all familiar with Go...

Ah that makes sense and yes we could improve the error handling here. Can you open a new issue?