minio/minio-go

bitrot healing not working as expected

dzhang28 opened this issue · 3 comments

dzhang28 commented

Hi MinIO team,

I setup a MINIO cluster with bitrot scan enabled (MinIO console -> Settings -> MinIO Configuration -> Heal -> Bitrot Scan on).
I uploaded one file and I can saw it was distributed among EC set.
Then I changed some bytes of one sharded file on disk ( part.1 file).
But after one night, I still saw the file was not healed.

My bitrot configuration is as follow:

art@minio:$ mc admin config get minio/ scanner
scanner speed=default delay=1 max_wait=1s cycle=5s
art@minio:$ mc admin config get minio/ heal
heal bitrotscan=on max_sleep=250ms max_io=3 drive_workers=6

BTW, the manual way of healing can work normally (mc admin heal -r --scan deep ...).
But I still think this is not acceptable.
Why the bitrot healing could not work properly in background if it is enabled?
Thanks.

My MinIO version is:
minio version RELEASE.2023-10-07T15-07-38Z (commit-id=c27d0583d4d496981140eff822e2abb34a6f0b60)
Runtime: go1.21.1 linux/arm64
License: GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html
Copyright: 2015-2023 MinIO, Inc.

harshavardhana commented

Bitrot healing happens in scanner cycles, so if the scanner has not visited your object it won't heal it.

We also automatically heal when we find corrupted content. You simply have to read the object for the healing to get triggered or you can run manually like the way you did if you want to run healing under your control.

dzhang28 commented

Bitrot healing happens in scanner cycles, so if the scanner has not visited your object it won't heal it.

We also automatically heal when we find corrupted content. You simply have to read the object for the healing to get triggered or you can run manually like the way you did if you want to run healing under your control.

I have several points needing further clarify:

  1. I already enabled the scanner (scanner speed=default delay=1 max_wait=1s cycle=5s). So why scanner did not visit my object then? It sounds a bug of scanner then.
  2. How to let MINIO automatically find corrupted content and fix it?
  3. I also tried reading the object by previewing it through console. Still it's not healed.
  4. I also tried reading the object by downloading it through console/mc. Still it's not healed.
klauspost commented

MinIO will only rarely actually select an object for a heal scan. Under usual circumstances there can be months between a given object being selected.

MinIO will only detect corruption if the part of the file is actually read. There are various selection criteria, prioritizing local disks first, data shards second and then only reading what it needs to. So if you happened to corrupt a parity shard it will rarely be used.

If this is so horrible, sign up for a subscription and we can go through the details of your case.