Unexpected short file read

[bigdogdan2]

Getting errors trying to do a read test:
Is there an error in the command?
elbencho version 2.1-6, RHEL 8.2, Running on 2 nodes with shared filesystem.

[root@w1 a]# elbencho --hosts w1,w2 --threads 32 --size 4G --block 128K --write --nolive --mkdir --rand --direct --iodepth 16 /mnt/scratch/data/
OPERATION RESULT TYPE FIRST DONE LAST DONE
========= ================ ========== =========
MKDIRS Elapsed ms : 2 11
Dirs/s : 10925 ### 5553
Dirs total : 32 64

WRITE Elapsed ms : 1210 7353
Files/s : 0 8
IOPS : 404015 285176
Throughput MiB/s : 50501 35647
Total MiB : 61153 262144
Files total : 1 64

[root@w1 a]#

[root@w1 a]# elbencho --hosts w1,w2 --threads 32 --size 4G --block 128K --read --nolive --mkdir --rand --direct --iodepth 16 /mnt/scratch/data/
OPERATION RESULT TYPE FIRST DONE LAST DONE
========= ================ ========== =========
MKDIRS Elapsed ms : 2 5
Dirs/s : 12825 12158
Dirs total : 32 64

=== [ HOST: w1:1611 (Rank: 0) ] ===
ERROR: Unexpected short file read. Path: /mnt/scratch/data//r26/d0/r26-f0; Bytes read: 44957696; Expected read: 4294967296
ERROR: Unexpected short file read. Path: /mnt/scratch/data//r16/d0/r16-f0; Bytes read: 43253760; Expected read: 4294967296
=== [ END ] ===
=== [ HOST: w2:1611 (Rank: 1) ] ===
ERROR: Unexpected short file read. Path: /mnt/scratch/data//r41/d0/r41-f0; Bytes read: 68550656; Expected read: 4294967296
ERROR: Unexpected short file read. Path: /mnt/scratch/data//r59/d0/r59-f0; Bytes read: 77594624; Expected read: 4294967296
=== [ END ] ===
[root@w1 a]#

2nd question. How to create a dataset and then every read will use the same dataset?

[breuner]

Hi @bigdogdan2 !
The short read comes from the random write phase. For elencho, random means that the write offsets are really random, so there is no guarantee that the last block of a file gets written. Thus, after a random write phase, a file might end up with a smaller size than what you specified as --size.
But no worries, there are ways to address this:

• The easiest one: Add the --trunctosize parameter to the write phase. This will ensure that the files end up with the full size that you specified (thus no short reads afterwards), although some of the blocks might not have been written, so some reads afterwards might be served from sparse areas of the file.
• My recommendation: Keep doing your random write as you did before, then afterwards run another write without the --rand parameter to ensure that all file blocks get written (if you want to make it harder for the storage system, you can also add --backward parameter to start writes at the end of the file and proceed in decreasing offset order) and then afterwards do the random reads as you did before.

Regarding your 2nd question:
I'm not sure if I understand it correctly, because you don't specify the -F parameter, so the files will remain in place after the end of the read phase and thus they are available for further read phases.
But maybe you are referring to creating a set of shared files, which can be used with a different number of threads afterwards? If so, I do it usually like this to create a set of 256 shared large files for bandwidth and IOPS testing. Since this is always using the same 256 files, it can be used for bandwidth and IOPS testing with different number of threads and hosts.

# Create dataset and test streaming write throughput:
# (If the storage system supports compression, then I usually add "--blockvarpct 100"
# to write phases to generate non-reducible data.)

$ elbencho /mnt/mystorage/file{1..256} -w --direct -b 4m -s 12g -t 48 --hostsfile myhosts 

# Test streaming read throughput:

$ elbencho /mnt/mystorage/file{1..256} -r --direct -b 1m --iodepth 4 -s 12g -t 48 --hostsfile myhosts

# Test random 4KB read IOPS (time-limited to 60 seconds, just because 
# waiting longer should typically not change the result):

$ elbencho /mnt/mystorage/file{1..256} -r --direct -b 4k --iodepth 16 -s 12g -t 48 --hostsfile myhosts --rand --timelimit 60

# And finally random 4KB write IOPS. (Again time-limited to 60 seconds.
# If the storage system does any kind of caching despite the "--direct" flag, e.g. 
# through a hardware RAID controller, then a longer run might change the result.)

$ elbencho /mnt/mystorage/file{1..256} -w --direct -b 4k --iodepth 16 -s 12g -t 48 --hostsfile myhosts --rand --timelimit 60

I hope this helps. Please close this issue if your question has been answered or otherwise of course please feel free to let me know if you have any further questions.

[bigdogdan2]

Thanks, worked!