Volume Disconnection after deploying grafana-backup-tool
Closed this issue · 2 comments
Hello there!
I'm encountering a persistent volume disconnection issue after running the grafana-backup-tool within a Docker container deployed using Ansible. Restarting the container doesn't resolve the problem, confirming it's not a transient issue.
Steps to Reproduce:
- Set up a Debian server with Docker.
- Deploy Grafana container with persistent volumes mounted using an Ansible playbook.
- Deploy backup tool with Ansible playbook (using VENV)
Grafana Ansible playbook
hosts: master
become: true
tasks:
- name: Ensure Grafana Data Directory Exists
file:
path: "{{ general_cont_path }}/grafana/data"
state: directory
mode: "0755"
- name: Ensure Grafana backups Directory Exists
file:
path: "{{ general_cont_path }}/grafana/backups"
state: directory
mode: "0755"
- name: Ensure Grafana vend_dir directory Exists
file:
path: "{{ venv_dir }}"
state: directory
mode: "0755"
- name: Create Docker Compose File for Grafana
copy:
content: |
version: '3.8'
services:
grafana:
container_name: grafana
image: grafana/grafana:latest
ports:
- "3000:3000"
volumes:
- /etc/localtime:/etc/localtime:ro
- grafana_data:/var/lib/grafana
- grafana_venv_dir:/opt/grafana_venv
- grafana_backup:/var/lib/grafana/backups # Corrected line
restart: unless-stopped
networks:
- home_automation_network
volumes:
grafana_data:
name: grafana_data
driver: local
driver_opts:
type: none
device: "{{ general_cont_path }}/grafana/data"
o: bind
grafana_venv_dir:
name: grafana_venv_dir
driver: local
driver_opts:
type: none
device: "{{ venv_dir }}"
o: bind
grafana_backup:
name: grafana_backup
driver: local
driver_opts:
type: none
device: "{{ general_cont_path }}/grafana/backups"
o: bind
networks:
home_automation_network:
external: true
dest: "{{ composes_path }}/grafana-compose.yml"
mode: "0644"
- name: Deploy Grafana Container Using Docker Compose
community.general.docker_compose:
project_src: "{{ composes_path }}"
files:
- grafana-compose.yml
state: present
pull: yes
restarted: yes
- All global vars are correct,The volumes are beeing created and verified with Inspect command.
Script Ansible playbook
hosts: master
become: true
vars:
backup_days: "1,2,3" # Monday, Tuesday, Wednesday
backup_hour: 4
backup_minute: 0
backup_retention_days: 14
tasks:
- name: Ensure rsync is installed
apt:
name: rsync
state: present
become: true
# Use docker_container module instead of docker exec commands
- name: Create virtual environment within Grafana container
docker_container:
name: grafana
state: started
command: bash -c "python3 -m venv {{ venv_dir }}"
env:
VIRTUAL_ENV: "{{ venv_dir }}"
- name: Install grafana-backup-tool within virtual environment
docker_container:
name: grafana
state: started
command: bash -c "source {{ venv_dir }}/bin/activate && pip install grafana-backup-tool"
env:
VIRTUAL_ENV: "{{ venv_dir }}"
- name: Backup Grafana configuration on specific days
cron:
name: "Grafana Backup - Multiple Days"
job: "grafana-backup-tool --config /var/lib/grafana/grafana.ini --token {{ grafana_token }} --backup /var/lib/grafana/backups{{ lookup('pipe', 'date +%Y-%m-%d-%H-%M-%S') }}.tar.gz"
hour: "{{ backup_hour }}"
minute: "{{ backup_minute }}"
weekday: "{{ backup_days }}" # Using the variable directly
user: root
- name: Execute backup tool within container
docker_container:
name: grafana
state: started
command: /bin/bash -c "source {{ venv_dir }}/bin/activate && grafana-backup-tool --config /var/lib/grafana/grafana.ini --token {{ grafana_token }} --backup /var/lib/grafana/backups{{ lookup('pipe', 'date +%Y-%m-%d-%H-%M-%S') }}.tar.gz"
env:
VIRTUAL_ENV: "{{ venv_dir }}"
- name: List backup files in the container
ansible.builtin.shell: |
docker exec grafana ls -lah /var/lib/grafana/backups
register: backup_files
ignore_errors: yes
- name: Show backup files
ansible.builtin.debug:
msg: "{{ backup_files.stdout_lines }}"
- name: Set proper ownership and permissions on the backup directory
file:
path: "{{ grafana_backup_directory }}"
owner: root
group: root
mode: "0640"
recurse: yes
become: true
- name: Copy backup from container to host
command: rsync --archive --compress --delay-updates "{{ general_cont_path }}/grafana/backups" /home/sktech/backups/grafana
become: yes
- name: Clean up old backups (optional)
find:
paths: "{{ grafana_backup_directory }}/archive"
patterns: "*.tar.gz"
age: "{{ backup_retention_days | default(30) }}"
# Make sure to replace placeholders like `venv_dir`, `grafana_backup_directory`, and `grafana_token` with your actual values.
# Adjust the backup schedule, retention days, and other settings as needed.
# Consider using Ansible Vault to securely store sensitive information like the `grafana_token`.
# Key Improvements:
#
# - Uses `docker_container` module for more structured and reliable container interaction.
# - Consolidated virtual environment creation and activation into a single step.
# - Clarified variable usage and provided notes for placeholder replacement.
# - Emphasized security considerations in the comments.
#
# Remember to adapt this script to your specific environment and security requirements. If you have any further questions, feel free to ask!
Expected Behavior:
The grafana-backup-tool should run successfully (whichs runs) within the container but without impacting volume persistence. Volumes should remain attached even after container restarts.After this backup is finished no files are created anywhere which is expected Cause of the volume disconection when the script tryed to backup i assume.
Actual Behavior:
Volumes become disconnected after running the script and persist even after various container restarts, indicating a deeper issue.
Troubleshooting:
- I've isolated the problem to the script section executing the grafana-backup-tool.
docker_container:
name: grafana
state: started
command: /bin/bash -c "source {{ venv_dir }}/bin/activate && grafana-backup-tool --config /var/lib/grafana/grafana.ini --token {{ grafana_token }} --backup /var/lib/grafana/backups{{ lookup('pipe', 'date +%Y-%m-%d-%H-%M-%S') }}.tar.gz"
env:
VIRTUAL_ENV: "{{ venv_dir }}"
-
Commenting out this section prevents volume disconnection, confirming the tool's involvement.
-
I haven't found similar issues reported online or in the tool's documentation.
-
I've tested volume persistence across various container restart methods (restart, stop/start, exit/start).
-
The issue is narrowed down to the specific code block provided earlier.
Request:
I'm seeking assistance in pinpointing the root cause of the volume disconnection and finding a solution to ensure persistent backups within the container. Any insights, suggestions, or workarounds related to the grafana-backup-tool or potential volume manipulation within the container would be greatly appreciated.
This is not an issue in any way specific to grafana-backup-tool. My guess is you are misunderstanding the proper use of the docker_container ansible module, which is not something we are able to help with, other than point you to the upstream documentation for that module.
It seems I caused some confusion about my setup. While the issue I'm facing might not be directly related to the grafana-backup-tool, it appears backups aren't working at all,the fact that i was lucky with the ansible playbook to pinpoint the issue can help you for sure, regardless of whether I use manual methods or the tool with an Ansible playbook. I've tried various approaches and installations, but the data remains unbacked up and the volumes keep breaking.