Snakemake pipeline to test file system performance. This tests file creation, network download, and disk throughput speeds.
Modify config.yaml
based on the desired parameters.
- Edit
numfiles
to change number of files to create. - Edit
filesize
to change the size of a single file to create. This will be copied to test disk throughput speeds. - Edit
download
to the source of file to be downloaded. This is to test network download speed.
All files and raw benchmarks from Snakemake can be found in results
. The calculated speeds are reported in results/benchmarks/speeds.log
.
Below are testing results of the pipeline with FlowDeploy on different AWS file system configurations.
Operation | EBS | Instance Store | FSx | ObjectiveFS | S3Fs |
---|---|---|---|---|---|
Files created per second | 24878 | 31850 | 418 | 582 | 8 |
Network download (MB/s) | 24.44 | 30.31 | 33.04 | 48.42 | 33.47 |
Disk throughput (MB/s) | 114.0 | 555.6 | 862.3 | 215.6 | 45.13 |