dgtlmoon/changedetection.io

BrotliDecompress failed on some pages

Closed this issue · 3 comments

jbd7 commented

Describe the bug
Some pages monitor fail with Exception: BrotliDecompress failed

Version
Exact version in the top right area: 47.04
(but has been happening with previous versions too)

To Reproduce

Steps to reproduce the behavior:

  1. Wait for changedetection to fetch the monitored page, or click on "Recheck"
  2. Refresh changedetection homepage
  3. See error Exception: BrotliDecompress failed

Logs

Using docker logs -f <container id>, I see the following:

2024-11-05 18:39:59.804 | DEBUG    | changedetectionio.processors.text_json_diff.processor:run_changedetection:350 - Watch UUID f97ab8da-5d0c-4e3e-8b25-c58842da78fb content check - Previous MD5: 209cc19efa0f71bddc65bb9a6faa46d5, Fetched MD5 f01d5f462bc688efd381e9b7436399c6
2024-11-05 18:39:59.812 | DEBUG    | changedetectionio.model.Watch:history:184 - Reading watch history index for f97ab8da-5d0c-4e3e-8b25-c58842da78fb
2024-11-05 18:39:59.813 | DEBUG    | changedetectionio.model.Watch:history:184 - Reading watch history index for f97ab8da-5d0c-4e3e-8b25-c58842da78fb
2024-11-05 18:40:01.153 | DEBUG    | changedetectionio.model.Watch:history:184 - Reading watch history index for f97ab8da-5d0c-4e3e-8b25-c58842da78fb
... [about 50 similar lines removed]
2024-11-05 18:40:05.882 | DEBUG    | changedetectionio.model.Watch:history:184 - Reading watch history index for f97ab8da-5d0c-4e3e-8b25-c58842da78fb
2024-11-05 18:40:05.906 | DEBUG    | changedetectionio.model.Watch:history:184 - Reading watch history index for f97ab8da-5d0c-4e3e-8b25-c58842da78fb
2024-11-05 18:40:05.929 | DEBUG    | changedetectionio.model.Watch:history:184 - Reading watch history index for f97ab8da-5d0c-4e3e-8b25-c58842da78fb
2024-11-05 18:40:05.953 | DEBUG    | changedetectionio.model.Watch:history:184 - Reading watch history index for f97ab8da-5d0c-4e3e-8b25-c58842da78fb
2024-11-05 18:40:05.954 | ERROR    | changedetectionio.update_worker:run:479 - Exception reached processing watch UUID: f97ab8da-5d0c-4e3e-8b25-c58842da78fb
2024-11-05 18:40:05.955 | ERROR    | changedetectionio.update_worker:run:480 - BrotliDecompress failed
2024-11-05 18:40:05.957 | DEBUG    | changedetectionio.update_worker:run:580 - Watch f97ab8da-5d0c-4e3e-8b25-c58842da78fb done in 6.86s
2024-11-05 18:40:25.211 | INFO     | changedetectionio.store:sync_to_json:384 - Saving JSON..

Expected behavior

  • Page is successfully parsed and monitored

Installation

  • Ubuntu 24..04 on a VPS
  • Docker v 24.0.7
  • ChangeDetection version 47.04
  • Docker-compose
    • image: ghcr.io/dgtlmoon/changedetection.io
    • image: selenium/standalone-chrome:4

Additional context

  • Happens on webpages that were successful in the past (incl. example)
  • Tried adding dns 8.8.8.8 and 1.1.1.1 in the docker compose, no change in behavior
  • I tried on another Brotli supported page, https://apnews.com/projects/election-results-2024/, for which https://tools.keycdn.com/brotli-test says "Brotli compression is supported.", like https://docs.n8n.io/reference/release-notes/
    • The page check is successful, and the docker logs show:
2024-11-05 18:59:53.670 | DEBUG    | changedetectionio.processors.text_json_diff.processor:run_changedetection:350 - Watch UUID 781ca46a-28ff-4f18-bc71-695f481489a5 content check - Previous MD5: 09f737081a21384cf4d52a45df4e69a9, Fetched MD5 09f737081a21384cf4d52a45df4e69a9
2024-11-05 18:59:53.670 | DEBUG    | changedetectionio.update_worker:run:580 - Watch 781ca46a-28ff-4f18-bc71-695f481489a5 done in 0.15s
  • I tried adding https://docs.n8n.io/ as a Watch, from the same domain as the page generating BrotliDecompress errors, and it is fetched successfully.
  • I tried checking for Brotli in the changedetection container:
sudo docker exec -it 2f61860d3526 /bin/bash
root@changedetection:/app# brotli --version
bash: brotli: command not found
root@changedetection:/app# apt list --installed | grep brotli

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libbrotli1/now 1.0.9-2+b6 amd64 [installed,local]
root@changedetection:/app# pip3 list | grep brotli

[notice] A new release of pip is available: 24.0 -> 24.3.1
[notice] To update, run: pip install --upgrade pip

Screenshot
image

hello, I cant reproduce this, is your disk full or something?

also the package name is Brotli not brotli


root@c3920c7f594a:/app# pip3 list|grep -i brot
Brotli                    1.1.0

hmm I think the snapshots in your datastore are corrupt somehow, clear the history for that watch and it will probably work again

jbd7 commented

Thanks and good tip, clearing the history did the job!