Erasure coding (EC) is a method of data protection in which data is broken into fragments, expanded and encoded with redundant data pieces and stored across a set of different locations or storage media.
Data encoding/decoding is very CPU intensive and can be a major overhead when using Erasure coding.
This is my benchmark test on Mellanox ConnectX-5 EN adapter cards' ECO(Erasure Coding Offloading) capabilities.
The code in this repository is modified from the sample test code in Mellaxnox's EC repository(https://github.com/Mellanox/EC.git).
- Mellanox ConnectX-4 or ConnectX-4 Lx.
- MLNX_OFED_LINUX-3.3-1.0.0.0 (or later).
- Firmware - v12.16.1006 for ConnectX-4 (or later), v14.16.1006 for ConnectX-4 Lx (or later).
- Jerasure library v2.0. (https://github.com/tsuraan/Jerasure).
-
install Jerasure & GF-Complete using the following parameters:
./configure --prefix=/usr/ --libdir=/usr/lib64/
-
It is highly recommended to install ConnectX庐-4 on PCIe3.0 x16, and ConnectX庐-4 Lx on PCIe3.0 x8 slot for better performance.
-
Use block size aligned to 64 bytes to avoid copying to remainder to internal buffers.
Test code resides in tests
folder.
Build
- cd tests
- make
ibv_ec_capability_test
Checking EC offload capabilities for IB devices.
Usage
./ibv_ec_capability_test
Usage = ./ec_capability_test <device_name>
Available devices : <List of available IB devices>
ibv_ec_encoder
Perform encode operations on input files using Erasure Coding NIC Offload library, Erasure Coding Offload Experimental Verbs API and Jerasure library.
Galois field GF(2^w) must be equal to GF(2^4).
The test will encode the input file three times (all the results should be equal):
- using Erasure Coding NIC Offload library - the results will be written into .encode.code.eco
- using Experimental Verbs API - the results will be written into .encode.code.verbs
- using Jerasure Library - the results will be written into .encode.code.sw
Usage
Usage:
./ibv_ec_encoder start EC encoder
Options:
-i, --ib-dev=<dev> use IB device <dev> (default first device found)
-k, --data_blocks=<blocks> Number of data blocks
-m, --code_blocks=<blocks> Number of code blocks
-w, --gf=<gf> Galois field GF(2^w)
-D, --datafile=<name> Name of input file to encode
-s, --frame_size=<size> size of EC frame
-d, --debug print debug messages
-v, --verbose add verbosity
-h, --help display this output
ibv_ec_encoder_mem
Perform encode operations on data blocks generated in memory using Erasure Coding Offload Experimental Verbs API and Jerasure library.
Galois field GF(2^w) must be equal to GF(2^4).
The test will encode the data blocks generated in memory with multiple threads and output the elapsed time(wall time) for encode operations.
- The size of the "file" to be encoded should be specified in command line arguments.
- The total number of encode operations E = file_size / frame_size. All threads share a single counter E, each time a thread removes a batch of work from E until E reaches zero.
- Since Erasure Coding NIC offload library is not thread safe, we can only choose between Experimental Verbs API and Jerasure Library.
Usage
Usage:
./ibv_ec_encoder_mem start EC encoder
Options:
-i, --ib-dev=<dev> use IB device <dev> (default first device found)
-k, --data_blocks=<blocks> Number of data blocks
-m, --code_blocks=<blocks> Number of code blocks
-w, --gf=<gf> Galois field GF(2^w)
-F, --file_size=<size> size of file in bytes
-s, --frame_size=<size> size of EC frame
-t, --thread_number=<number> number of threads used
-V, --verbs use verbs api
-S, --software_ec use software EC(Jerasure library)
-L, --eco_lib use mellanox ECO library(mlx_lib)
-d, --debug print debug messages
-v, --verbose add verbosity
-h, --help display this output
ibv_ec_decoder
Perform decode operations on input files using Erasure Coding NIC Offload library and Erasure Coding Offload
Experimental Verbs API.
Galois field GF(2^w) must be equal to GF(2^4).
The test will decode the input file two times (The code result should be equal to the input code file, data results should be equal to the input file):
- using Erasure Coding NIC Offload library - the data results will be written into .decode.data.eco
the code results will be written into .decode.code.eco - using Experimental Verbs API - the data results will be written into .decode.data.verbs
Usage
Usage:
./ibv_ec_decoder start EC decoder
Options:
-i, --ib-dev=<dev> use IB device <dev> (default first device found)
-k, --data_blocks=<blocks> Number of data blocks
-m, --code_blocks=<blocks> Number of code blocks
-w, --gf=<gf> Galois field GF(2^w)
-D, --datafile=<name> Name of input data file
-C, --codefile=<name> Name of input code file
-E, --erasures=<erasures> Comma separated failed blocks
-s, --frame_size=<size> size of EC frame
-d, --debug print debug messages
-v, --verbose add verbosity
-h, --help display this output
Benchmark scripts resides in bench
folder.
best_block_size Determine the best block size for ECO and Jerasure library. Use program ibv_ec_encoder_mem. The encoder is single-threaded. For ECO, we use sync verbs.
max_throughput Determine the maximum throughput for ECO and Jerasure library. Use program ibv_ec_encoder_mem. The encoder is multi-threaded. For ECO, we use sync verbs.