Not Storing Data?????
Closed this issue · 27 comments
Debian 12 Container on Proxmox
Starts and Runs Fine but,
Grafani wont Store new Password
Prometheus isnt storing Data
Happens with CT Reboot on Docker down
Sounds like docker isn't persisting volumes on your system
So any Idea how i set it?
Please bear in mind ive just mirrored the data from site and ran no customization.
I'll have to review, usually docker will persist volumes with the default install
You may need to look at Docker config for promox, I've read through some forums and there seems to be some extra steps you need to get persistent volumes
If anyone else is experiencing this pls post
just installed on linux mint, docker engine not docker desktop. works fine however no data persistence.
Installed on Ubuntu 22.4.3, Docker Engine 24.0.7, Docker Compose 2.21.0. Previous volumes don't get reused on stack launch. No persitence for any data.
Hi. Thanks for a great tool @plaintextpackets.
I thought I'd contribute another data point on the data persistence issue...
I installed onto Ubuntu 22.04.1 - first time mirror from the git repo and first time docker installation on this machine. Followed the provided instructions in the readme and all worked correctly (except that I had to start it with "sudo docker compose up").
However, the new Grafana password I set, and the collected data did not survive a "docker compose down" and "docker compose up" cycle.
Kind regards
Roger
Grafana login data also did not persist for me and I had to mount (add to the list of volumes "- .netprobe_lite/grafana:/var/lib/grafana") as there we have the grafana.ini file. The same goes for "/data" folder of the redis container and "/prometheus" folder of the prometheus container.
So I think I've figured this out but haven't had time to test it fully:
https://github.com/plaintextpackets/netprobe_lite/tree/data_persist_issue
Basically you need to have folders for both prometheus and grafana data mapped for persistence.
The only catch is that you clone the repo as a user and not root Docker will have an issue accessing those folders. Changing permissions on the whole netprobe folder works, just figuring out an easy method for users to do that on their own.
The latest Docker-Compose file, with more external volumes, does solve this.
But! There's a permissions challenge, in that docker-created folders are owned by root, not my desired user (1000:1000)
I can chmod it so it works, but that's not ideal.
See the "UserID" solution put forward by Docker team, where you externalize the ID the container should run as.
So my only concern with this method:
Is that I don't actually build Prometheus or Grafana, so not sure this would do anything to those containers
Ok so I've tried a new method using Docker volumes instead of bind mounts, and it seems to work and persist data through down / up and also reloads of the host:
https://github.com/plaintextpackets/netprobe_lite/tree/data_persist_issue
I'm going to do some more testing, if anyone else can help by testing this branch that would help
Okay, I think I have it for you...
First, understand that bind mount directories that do not already exist will be created as the Docker user (usually root) unless specified otherwise.
Second, one can control the UserID the Docker container runs as using the USER directive.
I start clean, then do the git clone
I make the grafana and prometheus data directory structures
( mkdir -p netprobe_lite/data/grafana netprobe_lite/data/prometheus )
I indicate in my Docker Compose the UserID I want to run as (UID = 1000 for most)
ALL WORKS FINE!
The containers use the folders I made, they are owned by me, and no permissions issues.
To package this better, you could put those two empty folders in the git repo, creating them automatically during cloning.
All we as users have to do is ensure the USERID is set to whatever user we normally run as.
# NetProbe using shared repository
services:
redis:
restart: unless-stopped
container_name: netprobe-redis
image: "redis:latest"
volumes:
- /home/jherr/docker-containers/netprobe_lite/config/redis/redis.conf:/etc/redis/redis.conf
netprobe:
restart: unless-stopped
container_name: netprobe-probe
image: "plaintextpackets/netprobe:latest"
volumes:
- /home/jherr/docker-containers/netprobe_lite:/netprobe_lite
environment:
MODULE: NETPROBE
presentation:
restart: unless-stopped
container_name: netprobe-presentation
image: "plaintextpackets/netprobe:latest"
volumes:
- /home/jherr/docker-containers/netprobe_lite:/netprobe_lite
environment:
MODULE: PRESENTATION
prometheus:
restart: unless-stopped
container_name: netprobe-prometheus
image: "prom/prometheus"
volumes:
- /home/jherr/docker-containers/netprobe_lite/config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
- /home/jherr/docker-containers/netprobe_lite/data/prometheus:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
user: "1000"
grafana:
restart: unless-stopped
image: grafana/grafana-enterprise
container_name: netprobe-grafana
volumes:
- /home/jherr/docker-containers/netprobe_lite/config/grafana/datasources/automatic.yml:/etc/grafana/provisioning/datasources/automatic.yml
- /home/jherr/docker-containers/netprobe_lite/config/grafana/dashboards/main.yml:/etc/grafana/provisioning/dashboards/main.yml
- /home/jherr/docker-containers/netprobe_lite/config/grafana/dashboards/netprobe.json:/var/lib/grafana/dashboards/netprobe.json
- /home/jherr/docker-containers/netprobe_lite/data/grafana:/var/lib/grafana
ports:
- '3001:3000'
user: "1000"
Ok so I've tried a new method using Docker volumes instead of bind mounts, and it seems to work and persist data through down / up and also reloads of the host:
https://github.com/plaintextpackets/netprobe_lite/tree/data_persist_issue
I'm going to do some more testing, if anyone else can help by testing this branch that would help
This branch has fixed my qnap pull that was previously not storing any data.
That only works, I believe, because that whole folder structure (the volume) is then created by Docker when firing up the Compose file. Everything is owned by Root, which is what some of the vendor containers are running as. I'm in over my head to be able to recommend any solutions for you. All I can say is this approach seems to be going in the wrong direction... I don't have these constraints with other Docker deployments. (Except WikiJS. Same issue there, but with PostgreSQL as the container in question) I don't know how they solve it, nor do I expect you have to figure it out better than you have. I appreciate all you've done. I would prefer a container that can run as my user ID, making files as my user ID. I can keep testing, but I won't be using the Volume approach when running daily in my home lab, since it doesn't fit with my Docker data backup strategy or Portainer approach.
…
On Mon, Apr 29, 2024 at 4:13 PM plaintextpackets @.> wrote: Ok so I've tried a new method using Docker volumes instead of bind mounts, and it seems to work and persist data through down / up and also reloads of the host: https://github.com/plaintextpackets/netprobe_lite/tree/data_persist_issue I'm going to do some more testing, if anyone else can help by testing this branch that would help — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB32XY7NBSWRIJ43WKYVYX3Y72ZXPAVCNFSM6AAAAABG3B46Y2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBTGY3TINJWGU . You are receiving this because you commented.Message ID: @.>
-- "What good is it for a man to gain the whole world, yet forfeit his soul?" - Mark 8:36 http://christianbookshelf.org/spurgeon/spurgeons_sermons_volume_2_1856/profit_and_loss.htm
So the tradeoff I think here is ease of installation and usability vs the ability to directly interact with / backup your data.
With Docker volumes you don't have to set permissions on folders, create folders or do anything except for clone and docker compose. Docker will automatically set permissions on that volume such that the user running the container can RW on it. In the case of grafana and prometheus it's actually not root, I believe it's 'nobody' and some other custom user. But the downside is to backup your data you need to either manually mount the volume and bind, or extract otherwise.
Doing a bind mount allows you full access to the raw data from prometheus and grafana to do with what you want, but you have to do more steps in the setup, plus you'd need instructions for Linux vs Mac vs Windows as they have different permissions.
For now I think I will stick to Docker volumes (if it works) but reference how you've done your compose config in case people want to go the bind mount route.
@ctv99 @suzunov @Roger0011 @ZundersHD would you mind testing:
https://github.com/plaintextpackets/netprobe_lite/tree/data_persist_issue
@plaintextpackets, did you see the "win/win" I mentioned above? It's still just a github pull, with a setting in the Docker Compose file.
But go with it, if you think the "volumes" approach works better for your needs. And for me, I'll stick with the UserID approach, which also works great!
But either way, thanks for all your work!
@ctv99 @suzunov @Roger0011 @ZundersHD would you mind testing:
https://github.com/plaintextpackets/netprobe_lite/tree/data_persist_issue
Yes, it works now but similar to @Jeppedy I would like to backup data regularly so my preferred solution would be with mounted folders and setting user IDs in docker compose so as the folders are writeable.
@plaintextpackets , awesome work by the way. Forgot to mention in my earlier post.
I think what I'll do is publish this version and add in a section to the Readme explaining how to use bind mounts if needed. That gives users both options.
What I'm finding is a lot of the new users aren't as tech savvy, which is totally fine as that was the intended audience. So I'm spending a lot of time helping with those kinds of questions. My fear is that making the install more complicated may make that worse.
I think that is a good compromise for now
@ctv99 @suzunov @Roger0011 @ZundersHD would you mind testing:
https://github.com/plaintextpackets/netprobe_lite/tree/data_persist_issue
I don't mind it the Volume of this docker Stack vanish one day, so this option is exactly what i needed.
As i only use it to add a monitoring option in my homelab dashboard to avoid the frequent question "why is the internet so slow" this is the perfect tool. Thanks @plaintextpackets .
I updated a new readme here with instructions to use bind mounts:
https://github.com/plaintextpackets/netprobe_lite/blob/data_persist_issue/README.md
@Jeppedy gave you credit for helping with these instructions, feel free to suggest improvements. Hoping to publish later today.
Confirmed Working with Volumes
Ok this release is now out:
https://github.com/plaintextpackets/netprobe_lite/releases/tag/v1.2.0
lol, Thx was just installing on 12 other servers and branch wasnt working.
Ok this release is now out:
https://github.com/plaintextpackets/netprobe_lite/releases/tag/v1.2.0
Working fine, retained data in the volumes created from the branch release.