"Page not found" error when using with balena
mlxyz opened this issue · 9 comments
Describe the bug
I am trying to use the container in a balena cloud environment.
This is the excerpt from the docker compose file:
backup:
image: offen/docker-volume-backup:latest
restart: always
volumes:
- paperless_media:/backup/paperless-media:ro
- paperless_data:/backup/paperless-data:ro
- paperless_pgdata:/backup/paperless-pgdata:ro
- caddy_data:/backup/caddy-data:ro
- caddy_config:/backup/caddy-config:ro
privileged: true
labels:
io.balena.features.balena-socket: "1"
networks:
- paperless
- caddy
When running the backup, I get the following error message:
time=2024-02-24T10:49:55.700Z level=ERROR msg="Fatal error running command: page not found" error="main.(*command).runAsCommand: error running script: main.runScript.func4: error running script: main.(*script).stopContainersAndServices: error querying for services: Error response from daemon: page not found"
DOCKER_HOST is set:
~ # printenv | grep DOCKER
DOCKER_HOST=unix:///host/run/balena-engine.sock
To Reproduce
Steps to reproduce the behavior:
- Setup balena cloud device with backup container and consuming container
- Start backup
Expected behavior
Backup running without any issues.
Version (please complete the following information):
- Image Version: v2.38.1
- Docker Version: v20.10.43
- Docker Compose Version (if applicable):
Additional context
Balena-engine:
Client:
Version: v20.10.43
API version: 1.41
Go version: go1.17.13
Git commit: 0be8046a1c191ad789a65d7d8429d899d6bc9fbe
Built: Tue Feb 6 08:42:34 2024
OS/Arch: linux/arm64
Context: default
Experimental: true
Server:
Engine:
Version: v20.10.43
API version: 1.41 (minimum version 1.12)
Go version: go1.17.13
Git commit: 0be8046a1c191ad789a65d7d8429d899d6bc9fbe
Built: Tue Feb 6 08:42:34 2024
OS/Arch: linux/arm64
Experimental: true
containerd:
Version: 1.6.6+unknown
GitCommit:
runc:
Version: unknown
GitCommit:
balena-engine-init:
Version: 0.13.0
GitCommit: 949e6fa-dirty
I don't know anything about Balena, its compatibility with Docker and what concepts it comes with.
Judging from what's going on in the code, there could be two things happening:
- Balena reports it's running in Swarm mode when it's not, making this tool call the
servicesendpoint which does not exist when not running in Swarm mode - Balena is running in Swarm mode, but does not expose a
servicesendpoint, thus you get a 404
To further debug this, could you provide the output of
docker info --format="{{ .Swarm | json }}"docker service ls
Thanks for digging into this.
I can only provide outputs for the corresponding balena-engine commands, as the docker cli is not available on balena-managed devices.
Output of balena-engine info --format="{{ .Swarm | json }}":
{"NodeID":"","NodeAddr":"","LocalNodeState":"","ControlAvailable":false,"Error":"","RemoteManagers":null}
balena-engine service ls is not available. These commands are:
root@central:~# balena-engine --help
Usage: balena-engine [OPTIONS] COMMAND
A self-sufficient runtime for containers
Options:
--config string Location of client config files (default "/home/root/.balena-engine")
-c, --context string Name of the context to use to connect to the daemon (overrides DOCKER_HOST env var and default context
set with "docker context use")
-D, --debug Enable debug mode
-H, --host list Daemon socket(s) to connect to
-l, --log-level string Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info")
--tls Use TLS; implied by --tlsverify
--tlscacert string Trust certs signed only by this CA (default "/home/root/.balena-engine/ca.pem")
--tlscert string Path to TLS certificate file (default "/home/root/.balena-engine/cert.pem")
--tlskey string Path to TLS key file (default "/home/root/.balena-engine/key.pem")
--tlsverify Use TLS and verify the remote
-v, --version Print version information and quit
Management Commands:
builder Manage builds
container Manage containers
image Manage images
manifest Manage Docker image manifests and manifest lists
network Manage networks
system Manage Docker
trust Manage trust on Docker images
volume Manage volumes
Commands:
attach Attach local standard input, output, and error streams to a running container
build Build an image from a Dockerfile
commit Create a new image from a container's changes
cp Copy files/folders between a container and the local filesystem
create Create a new container
diff Inspect changes to files or directories on a container's filesystem
events Get real time events from the server
exec Run a command in a running container
export Export a container's filesystem as a tar archive
history Show the history of an image
images List images
import Import the contents from a tarball to create a filesystem image
info Display system-wide information
inspect Return low-level information on Docker objects
kill Kill one or more running containers
load Load an image from a tar archive or STDIN
login Log in to a Docker registry
logout Log out from a Docker registry
logs Fetch the logs of a container
pause Pause all processes within one or more containers
port List port mappings or a specific mapping for the container
ps List containers
pull Pull an image or a repository from a registry
push Push an image or a repository to a registry
rename Rename a container
restart Restart one or more containers
rm Remove one or more containers
rmi Remove one or more images
run Run a command in a new container
save Save one or more images to a tar archive (streamed to STDOUT by default)
search Search the Docker Hub for images
start Start one or more stopped containers
stats Display a live stream of container(s) resource usage statistics
stop Stop one or more running containers
tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
top Display the running processes of a container
unpause Unpause all processes within one or more containers
update Update configuration of one or more containers
version Show the Docker version information
wait Block until one or more containers stop, then print their exit codes
Run 'balena-engine COMMAND --help' for more information on a command.
To get more help with docker, check out our guides at https://docs.docker.com/go/guides/
Thanks for providing the output, seems the first scenario applies and this check
now fails. This means, the tool thinks your engine is running in Swarm mode, and tries to list services, but obviously fails to do so.While this would be easy to fix by just changing the condition to check for an empty string or even flip the conditions to check against valid swarm states, I am a bit worried about this could become a rabbit hole of sorts, chasing down tiny incompatibilities like this one between Docker and Balena. If I understand the Docker OpenAPI spec correctly, SwarmInfo should be populated always (which Docker Engine also adheres to), but apparently Balena doesn't do it, which leaves me worrying this will also happen elsewhere. Do you have any kind of document about Docker / Balena compatibility "guarantees" or something that I could check in order to get an idea about how big the differences are in areas that this tool is interfacing with? Do you know if there is a way to run "Balena in Docker" so it could be used to run tests against?
All I could find is:
- https://engine-docs.balena.io/#transitioning-from-docker-ce which is only semi helpful (it should work ™️)
- https://github.com/balena-io-experimental/balena-engine-docker which looks like it could be used for testing (unfortunately it's not licensed: balena-io-experimental/balena-engine-docker#2)
@m90 Yes, afaik there is no comprehensive documentation about compatibility between balena engine and docker engine. In this thread, it is noted that balena engine is a moby-compatible container engine.
Your PR looks promising though - brilliant! 🎉
In case your setup allows you to do so, you can try checking out the code from this branch #376 and build an image off that. I do think "it should work ™️"
I'm still struggling to get a non-nausea-inducing test setup working (running Balena-in-Docker in Docker-in-Docker and such 💫). Also having to build the Balena-in-Docker image on each CI run currently adds a whopping extra of 22 minutes for building the image, which is kind of a show stopper. I.e. this might take a little while still, but maybe you can get it working with that self-built image already.
Just tried to build and run it, got a exec /usr/bin/backup: exec format error message - I probably built it for the wrong arch. I'll try again tomorrow!