mariadb-operator/mariadb-operator

[Bug] Prefixed s3 backups do not get cleaned up

Closed this issue · 2 comments

markus-brln commented

Documentation

Describe the bug

Hi, we're encountering the issue that when setting a prefix in the s3 storage backup config, the backups do get created properly in a sub-path in the s3 bucket, but the cleanup according to the maxRetention is not working.

Expected behaviour

Cleanup according to the maxRetention, resulting in deletion of older backups. (As soon as I remove prefix, it works fine.)

Steps to reproduce the bug

  1. Run mariadb-operator on your cluster(we're running Helm chart version 0.25.0 of the operator)
  2. Create a MariaDB instance
  3. Create a Backup using s3 as storage
  4. Set maxRetention: 1h so you can see the cleanup not working as soon as possible (I think that's the minimum)

Debug information

Logs of mariadb-operator container in the backup pod (see "no old backups were found"):

{"level":"info","ts":1713342902.1655796,"msg":"starting backup"}
{"level":"info","ts":1713342902.1657372,"msg":"configuring S3 backup storage"}
{"level":"info","ts":1713342902.165813,"msg":"reading target file","path":"/backup/0-backup-target.txt"}
{"level":"info","ts":1713342902.16585,"msg":"obtained target backup","file":"backup.2024-04-17T08:35:00Z.sql"}
{"level":"info","ts":1713342902.1658537,"msg":"pushing target backup","file":"backup.2024-04-17T08:35:00Z.sql"}
{"level":"info","ts":1713342902.2018082,"msg":"cleaning up old backups"}
{"level":"info","ts":1713342902.2018452,"msg":"no old backups were found"}

I looked a bit through the code (this my the first time ever looking at Go...) and I think the problem might be somewhere close to this line. I find it weird that it's passing a minio.ListObjectsOptions even though the implementation of ListObjects only wants a prefix there. Maybe I'm completely wrong, but maybe it helps to get started with investigating :)

Environment details:

  • Kubernetes version: 1.27
  • Kubernetes distribution: microk8s
  • mariadb-operator version: 0.25.0
  • Install method: helm
  • Install flavor: not sure [minimal, recommended, or custom]

Additional context

Backup CRD 
apiVersion: mariadb.mmontes.io/v1alpha1
kind: Backup
metadata:
  name: mariadb-backup
  namespace: my-namespace
spec:
  args:
    - '--single-transaction'
    - '--add-drop-database'
    - '--all-databases'
  backoffLimit: 5
  logLevel: info
  mariaDbRef:
    name: mariadb
    waitForIt: true
  maxRetention: 1h
  resources:
    limits:
      cpu: 2
      memory: 2Gi
    requests:
      cpu: 100m
      memory: 100Mi
  restartPolicy: OnFailure
  schedule:
    cron: '* * * * *'
    suspend: false
  storage:
    s3:
      accessKeyIdSecretKeyRef:
        key: my-backups-access-key
        name: my-backups-s3-credentials
      bucket: my-backups
      endpoint: my.s3.endpoint.com
      prefix: my-prefix/
      region: ''
      secretAccessKeySecretKeyRef:
        key: my-backups-secret-key
        name: backups-s3-credentials
mmontes11 commented

Hey there @markus-brln ! Thanks for reporting.

Indeed we have an issue with S3 prefixes. It has been solved in #554, and it will be released in v0.0.28.

The code you are looking at in main already has the issue solved, but it hasn't yet been released yet. We are close to release v0.0.28 but we still miss the release e2e tests. Will double check this use case!

Thanks! Closing this! Stay tuned for v0.0.28

markus-brln commented

Hey @mmontes11! Thanks for the super fast response and for the tip looking on the correct branch!

Keep up the good work 👍

  • 👍1