Copying large files (with shard on) fails
Closed this issue · 1 comments
Description of problem:
We upgraded gluster from 7 to 8 today, when copying large files (with the shard option enabled), it fails. By fail I mean it copies, but ls -lah shows it is 64MB in size rather than 7gb. It works intermittently apparently (e.g sometimes copying it will copy the whole thing). Most of the time it just copies 64MB of it rather then the whole lot. This seems like a big bug with gluster 8 (not sure if this happened with gluster 7).
The exact command to reproduce the issue:
root@test2:/mnt/mediawiki-static/private/dumps/speleowiki# ls -lah
total 72M
drwxr-xr-x 2 www-data www-data 4.0K Sep 9 16:36 .
drwxr-xr-x 268 www-data www-data 4.0K Sep 7 22:25 ..
-rw-r--r-- 1 www-data www-data 64M Sep 9 16:14 speleowiki_image_79312e0b88e02ed5785e.zip
root@test2:/mnt/mediawiki-static/private/dumps/speleowiki# ls -lah speleowiki_image_79312e0b88e02ed5785e.zip
-rw-r--r-- 1 www-data www-data 7.0G Sep 9 16:14 speleowiki_image_79312e0b88e02ed5785e.zip
(seems like the above is either cached or somehow. Since doing cp speleowiki_image_79312e0b88e02ed5785e.zip /srv,
got it to change it to 64M, though the file in /srv was 64MB too)
root@test2:/mnt/mediawiki-static/private/dumps/speleowiki# ls -lah speleowiki_image_79312e0b88e02ed5785e.zip
-rw-r--r-- 1 www-data www-data 64M Sep 9 16:14 speleowiki_image_79312e0b88e02ed5785e.zip
Expected results:
I expect that ls -lah (file or dir) show that the file size is 7gb for speleowiki_image_79312e0b88e02ed5785e.zip and that copying it will copy the whole thing.
Additional info:
- The output of the gluster volume info command:
Volume Name: mvol
Type: Distribute
Volume ID: 39259ae4-3eaf-48a2-a7a9-275cfc80204f
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: gluster1.miraheze.org:/srv/mvol
Brick2: gluster2.miraheze.org:/srv/mvol
Options Reconfigured:
features.read-only: off
performance.parallel-readdir: on
performance.io-cache: off
performance.read-ahead: off
cluster.use-compound-fops: on
performance.nl-cache: off
performance.write-behind-window-size: 1MB
network.ping-timeout: 200
storage.health-check-interval: 0
cluster.lookup-optimize: on
network.inode-lru-limit: 5000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
performance.readdir-ahead: on
client.ssl: on
server.ssl: on
server.event-threads: 7
client.event-threads: 7
features.shard: on
ssl.certificate-depth: 2
performance.client-io-threads: on
ssl.ca-list: /etc/ssl/certs/Sectigo.crt
ssl.own-cert: /etc/ssl/certs/wildcard.miraheze.org.crt
ssl.private-key: /etc/ssl/private/wildcard.miraheze.org.key
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.cache-size: 112MB
cluster.rebal-throttle: normal
diagnostics.brick-log-level: INFO
performance.write-behind: on
network.compression: off
performance.open-behind: on
- The operating system / glusterfs version:
Debian 10.5 / 8.1