/xrs

Fast and Efficient Data Reconstruction in Erasure-code

Primary LanguageGoMIT LicenseMIT

X-Reed-Solomon

GoDoc MIT licensed Build Status Go Report Card

Introduction:

  1. X-Reed-Solomon Erasure Code engine in pure Go.
  2. Fast and Efficient Data Reconstruction in Erasure-code
  3. Saving about 30% I/O in reconstruction
  4. Based on papers: <A “Hitchhiker’s” Guide to Fast and Efficient Data Reconstruction in Erasure-coded Data Centers> & <A Piggybacking Design Framework for Read-and Download-efficient Distributed Storage Codes>

Installation

To get the package use the standard:

go get github.com/templexxx/xrs

Documentation

See the associated GoDoc

Specification

GOARCH

  1. All arch are supported
  2. Go1.11(for AVX512)

Performance

And we must know the benchmark test is quite different with encoding/decoding in practice.

Because in benchmark test loops, the CPU Cache will help a lot. In practice, we must reuse the memory to make the performance become as good as the benchmark test.

Example of performance on my MacBook Pro (Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz) DataCnt = 10; ParityCnt = 4

Encoding:

Vector size AVX512 (MB/S) AVX2 (MB/S)
4KB -- 8632
64KB -- 7978
1MB -- 5967

Links & Deps