filecoin-project/lotus

Lotus-bench results thread (v20 params)

magik6k opened this issue · 27 comments

This issue is a place to put lotus-bench results for v20 params.

To best help us, run four tests:

Start by installing build dependencies from https://docs.lotu.sh/en+getting-started

git clone https://github.com/filecoin-project/lotus.git
cd lotus
make build bench

./bench --sector-size=1073741824
./bench --sector-size=1073741824 --no-gpu

# Only run these with > 64GiB of ram, recommended 128G
./bench --sector-size=34359738368
./bench --sector-size=34359738368 --no-gpu

Additionally, please tell us what CPU, GPU, and memory (including speed) you have in your setup.

Previous (v19) thread - #694

s1eke commented

CPU 9700k @4.90 GHz
GPU 1*2080 super
MEM 64GB (4x16GiB) DDR4 2666MHz


./bench --sector-size=1073741824

seal: addPiece: 6.863115265s (149 MiB/s)
seal: preCommit: 11m41.269313291s (1.46 MiB/s)
seal: commit: 13m8.823703216s (1.3 MiB/s)
seal: verify: 59.606013ms
unseal: 2m28.643021527s  (6.89 MiB/s)
generate candidates: 1.169117ms (855 GiB/s)
compute epost proof (cold): 14.162488298s
compute epost proof (hot): 11.548752742s
verify epost proof (cold): 16.515839ms
verify epost proof (hot): 6.34118ms

./bench --sector-size=1073741824 --no-gpu


seal: addPiece: 6.718153897s (152 MiB/s)
seal: preCommit: 11m34.258260928s (1.47 MiB/s)
seal: commit: 12m45.335208165s (1.34 MiB/s)
seal: verify: 57.145659ms
unseal: 2m26.040538518s  (7.01 MiB/s)
generate candidates: 2.504357ms (399 GiB/s)
compute epost proof (cold): 13.926699007s
compute epost proof (hot): 11.279428586s
verify epost proof (cold): 16.014568ms
verify epost proof (hot): 6.056955ms

CPU E5-2609 @ 2.40GHz
GPU GTX 680
MEM 32G (8 x 4GB) 1600MHz DDR3


./bench --sector-size=1073741824 --no-gpu

seal: addPiece: 23.301141108s (43.9 MiB/s)
seal: preCommit: 35m12.188289545s (496 KiB/s)
seal: commit: 35m7.261326445s (498 KiB/s)
seal: verify: 163.350239ms
unseal: 7m57.191085251s  (2.15 MiB/s)
generate candidates: 13.063559ms (76.5 GiB/s)
compute epost proof (cold): 37.324017752s
compute epost proof (hot): 30.293600465s
verify epost proof (cold): 44.748206ms
verify epost proof (hot): 16.236796ms

./bench --sector-size=1073741824

seal: addPiece: 24.090850412s (42.5 MiB/s)
seal: preCommit: 35m30.573062283s (492 KiB/s)
seal: commit: 30m14.141953904s (578 KiB/s)
seal: verify: 161.515732ms
unseal: 8m2.056245813s  (2.12 MiB/s)
generate candidates: 13.770987ms (72.6 GiB/s)
compute epost proof (cold): 38.348139298s
compute epost proof (hot): 25.801995146s
verify epost proof (cold): 48.463121ms
verify epost proof (hot): 16.184275ms

CPU xeon w-3175x @ 3.1Ghz
GPU Titan Rtx
Mem 16gb (2x8gb) @ 2133mhz DDR4

1GB, no GPU:

results (1073741824) (no gpu)
seal: addPiece: 8.2119413s (125 MiB/s)
seal: preCommit: 4m49.823511879s (3.53 MiB/s)
seal: commit: 9m26.336711307s (1.81 MiB/s)
seal: verify: 64.862152ms
unseal: 2m33.998776232s  (6.65 MiB/s)
generate candidates: 5.543861ms (180 GiB/s)
compute epost proof (cold): 10.568530719s
compute epost proof (hot): 6.943092937s
verify epost proof (cold): 39.186687ms
verify epost proof (hot): 6.78493ms

1GB with GPU

results (1073741824)
seal: addPiece: 8.328374331s (123 MiB/s)
seal: preCommit: 4m49.169407083s (3.54 MiB/s)
seal: commit: 6m55.558959772s (2.46 MiB/s)
seal: verify: 65.734771ms
unseal: 2m33.787052688s  (6.66 MiB/s)
generate candidates: 6.78989ms (147 GiB/s)
compute epost proof (cold): 9.602402437s
compute epost proof (hot): 5.863074044s
verify epost proof (cold): 22.706226ms
verify epost proof (hot): 6.849377ms

CPU xeon w-3175x @ 3.1Ghz
GPU Titan Rtx
Mem 16gb (2x8gb) @ 2133mhz DDR4

1GB, no GPU:

results (1073741824) (no gpu)
seal: addPiece: 8.2119413s (125 MiB/s)
seal: preCommit: 4m49.823511879s (3.53 MiB/s)
seal: commit: 9m26.336711307s (1.81 MiB/s)
seal: verify: 64.862152ms
unseal: 2m33.998776232s  (6.65 MiB/s)
generate candidates: 5.543861ms (180 GiB/s)
compute epost proof (cold): 10.568530719s
compute epost proof (hot): 6.943092937s
verify epost proof (cold): 39.186687ms
verify epost proof (hot): 6.78493ms

1GB with GPU

results (1073741824)
seal: addPiece: 8.328374331s (123 MiB/s)
seal: preCommit: 4m49.169407083s (3.54 MiB/s)
seal: commit: 6m55.558959772s (2.46 MiB/s)
seal: verify: 65.734771ms
unseal: 2m33.787052688s  (6.66 MiB/s)
generate candidates: 6.78989ms (147 GiB/s)
compute epost proof (cold): 9.602402437s
compute epost proof (hot): 5.863074044s
verify epost proof (cold): 22.706226ms
verify epost proof (hot): 6.849377ms

just 16G memory? so fast!

akme commented

CPU: 2 x Intel Xeon Gold 6126 @ 2.60GHz (48 cores)
RAM: 192GB RAM (12 x 16GB) DDR4 @ 2666MHz
GPU: Nvidia Tesla v100

./bench --sector-size=1073741824

2019-12-15T09:07:54.015Z	WARN	lotus-bench	lotus-bench/main.go:305	separate epost calls returned different proof values (this might be bad)
results (1073741824)
seal: addPiece: 22.781058369s (44.9 MiB/s)
seal: preCommit: 7m21.578003912s (2.32 MiB/s)
seal: commit: 10m57.640316606s (1.56 MiB/s)
seal: verify: 81.324454ms
unseal: 3m10.068874071s  (5.39 MiB/s)
generate candidates: 5.474838ms (183 GiB/s)
compute epost proof (cold): 12.795126398s
compute epost proof (hot): 7.502692283s
verify epost proof (cold): 29.891154ms
verify epost proof (hot): 8.067997ms

./bench --sector-size=1073741824 --no-gpu

2019-12-15T09:32:43.180Z        WARN    lotus-bench     lotus-bench/main.go:305 separate epost calls returned different proof values (this might be bad)
results (1073741824)
seal: addPiece: 30.370811928s (33.7 MiB/s)
seal: preCommit: 7m14.489077341s (2.36 MiB/s)
seal: commit: 12m23.281041774s (1.38 MiB/s)
seal: verify: 105.81063ms
unseal: 3m9.135767391s  (5.41 MiB/s)
generate candidates: 3.912892ms (256 GiB/s)
compute epost proof (cold): 15.218138796s
compute epost proof (hot): 9.59722685s
verify epost proof (cold): 30.609406ms
verify epost proof (hot): 8.576134ms

CPU: Ryzen 3700X
RAM: 64GB (4*16) DDR4 @ 2400Mhz
GPU: GTX1060 5G

root@root:~/ipfs/lotus# ./bench --sector-size=1073741824

results (1073741824)
seal: addPiece: 1m54.057384892s (8.98 MiB/s)
seal: preCommit: 11m48.383594364s (1.45 MiB/s)
seal: commit: 9m37.627019314s (1.77 MiB/s)
seal: verify: 62.816496ms
unseal: 2m28.871014837s  (6.88 MiB/s)
generate candidates: 1.509646ms (662 GiB/s)
compute epost proof (cold): 12.303857015s
compute epost proof (hot): 8.75769072s
verify epost proof (cold): 17.687187ms
verify epost proof (hot): 6.54885ms

root@root:~/ipfs/lotus# ./bench --sector-size=1073741824 --no-gpu

results (1073741824)
seal: addPiece: 1m39.716693258s (10.3 MiB/s)
seal: preCommit: 12m34.600479162s (1.36 MiB/s)
seal: commit: 11m48.372805577s (1.45 MiB/s)
seal: verify: 62.215504ms
unseal: 2m28.482918448s  (6.9 MiB/s)
generate candidates: 1.503021ms (665 GiB/s)
compute epost proof (cold): 12.53609654s
compute epost proof (hot): 10.017784469s
verify epost proof (cold): 17.458841ms
verify epost proof (hot): 6.501055ms

CPU xeon w-3175x @ 3.1Ghz
GPU Titan Rtx
Mem 128gb (8x16gb) @ 3200mhz DDR4

32GB sectors, with GPU

results (34359738368)
seal: addPiece: 5m34.118494113s (98.1 MiB/s)
seal: preCommit: 1h30m6.347843229s (6.06 MiB/s)
seal: commit: 53m19.732747851s (10.2 MiB/s)
seal: verify: 65.70126ms
unseal: 21m10.052851555s  (25.8 MiB/s)
generate candidates: 119.31557ms (268 GiB/s)
compute epost proof (cold): 11.566749879s
compute epost proof (hot): 6.391276204s
verify epost proof (cold): 28.867292ms
verify epost proof (hot): 7.035924ms

CPU AMD Ryzen 7 1700
GPU Radeon RX Vega 64 8GB
Mem 64GB (4X16) @3200mhx DDR4

./lotus-bench --sector-size=1073741824

results (1073741824)
seal: addPiece: 9.988058315s (103 MiB/s)
seal: preCommit: 13m35.923484516s (1.26 MiB/s)
seal: commit: 17m17.882961141s (1.01e+03 KiB/s)
seal: verify: 81.937809ms
unseal: 3m17.836944181s  (5.18 MiB/s)
generate candidates: 16.540585ms (60.5 GiB/s)
compute epost proof (cold): 18.059096352s
compute epost proof (hot): 14.905890068s
verify epost proof (cold): 28.283672ms
verify epost proof (hot): 9.744995ms

./lotus-bench --sector-size=1073741824 --no-gpu

results (1073741824)
seal: addPiece: 9.931773866s (103 MiB/s)
seal: preCommit: 13m35.653940855s (1.26 MiB/s)
seal: commit: 17m16.853389163s (1.01e+03 KiB/s)
seal: verify: 82.326422ms
unseal: 3m18.05062861s  (5.17 MiB/s)
generate candidates: 1.951008ms (513 GiB/s)
compute epost proof (cold): 18.012725436s
compute epost proof (hot): 14.876289777s
verify epost proof (cold): 25.070873ms
verify epost proof (hot): 9.262604ms

CPU xeon w-3175x @ 3.5Ghz
GPU Titan Rtx
Mem 128gb (8x16gb) @ 3600mhz DDR4

1G sectors, with GPU

results (1073741824)
seal: addPiece: 7.601784717s (135 MiB/s)
seal: preCommit: 4m38.645238791s (3.67 MiB/s)
seal: commit: 6m15.288139572s (2.73 MiB/s)
seal: verify: 61.490132ms
unseal: 2m25.476712775s  (7.04 MiB/s)
generate candidates: 1.481038ms (675 GiB/s)
compute epost proof (cold): 8.976461917s
compute epost proof (hot): 5.603638937s
verify epost proof (cold): 21.835998ms
verify epost proof (hot): 6.377835ms

CPU: Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz x 2

RAM: 256G ( 8 x 32G ) @ 2400MHz DDR4

GPU: GeForce RTX 2080ti

1g sector, with GPU

results (1073741824)
seal: addPiece: 25.464896099s (40.2 MiB/s)
seal: preCommit: 9m4.172052199s (1.88 MiB/s)
seal: commit: 9m56.831035991s (1.72 MiB/s)
seal: verify: 103.895145ms
unseal: 3m22.115522237s  (5.07 MiB/s)
generate candidates: 4.368273ms (229 GiB/s)
compute epost proof (cold): 14.420113214s
compute epost proof (hot): 8.460359031s
verify epost proof (cold): 37.81341ms
verify epost proof (hot): 10.93241ms

1g sector, no GPU

results (1073741824)
seal: addPiece: 39.661693239s (25.8 MiB/s)
seal: preCommit: 8m53.534806081s (1.92 MiB/s)
seal: commit: 14m43.495411259s (1.16 MiB/s)
seal: verify: 104.238912ms
unseal: 3m21.570040592s  (5.08 MiB/s)
generate candidates: 4.255491ms (235 GiB/s)
compute epost proof (cold): 17.775027168s
compute epost proof (hot): 11.589833683s
verify epost proof (cold): 32.902141ms
verify epost proof (hot): 11.000928ms

32g sector, with GPU

results (34359738368)
seal: addPiece: 15m19.295074145s (35.6 MiB/s)
seal: preCommit: 3h25m26.737883477s (2.66 MiB/s)
seal: commit: 1h17m29.088493622s (7.05 MiB/s)
seal: verify: 105.770193ms
unseal: 37m0.270668647s  (14.8 MiB/s)
generate candidates: 780.042454ms (41 GiB/s)
compute epost proof (cold): 55.697355676s
compute epost proof (hot): 9.056273919s
verify epost proof (cold): 107.767871ms
verify epost proof (hot): 11.129075ms

32g sector, no GPU

results (34359738368)
seal: addPiece: 16m2.533049462s (34 MiB/s)
seal: preCommit: 3h56m0.259543877s (2.31 MiB/s)
seal: commit: 3h32m10.143880419s (2.57 MiB/s)
seal: verify: 105.959822ms
unseal: 37m50.105183539s  (14.4 MiB/s)
generate candidates: 637.541031ms (50.2 GiB/s)
compute epost proof (cold): 49.151256819s
compute epost proof (hot): 12.10967765s
verify epost proof (cold): 176.924859ms
verify epost proof (hot): 11.012562ms

CPU: Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz x 2

RAM: 64G ( 4 x 16G ) @ 2400MHz DDR4

GPU: GeForce GTX 1080ti

1g sector, with GPU

results (1073741824)
seal: addPiece: 45.473024947s (22.5 MiB/s)
seal: preCommit: 8m57.856122648s (1.9 MiB/s)
seal: commit: 15m31.526385695s (1.1 MiB/s)
seal: verify: 103.407751ms
unseal: 3m23.510257404s  (5.03 MiB/s)
generate candidates: 4.268336ms (234 GiB/s)
compute epost proof (cold): 20.086428311s
compute epost proof (hot): 13.754659692s
verify epost proof (cold): 30.739592ms
verify epost proof (hot): 10.930319ms

1g sector, no GPU

results (1073741824)
seal: addPiece: 43.8081646s (23.4 MiB/s)
seal: preCommit: 8m57.758378734s (1.9 MiB/s)
seal: commit: 14m38.674287784s (1.17 MiB/s)
seal: verify: 103.524802ms
unseal: 3m25.452192666s  (4.98 MiB/s)
generate candidates: 4.241595ms (236 GiB/s)
compute epost proof (cold): 17.688321421s
compute epost proof (hot): 12.215557561s
verify epost proof (cold): 30.500858ms
verify epost proof (hot): 10.777101ms

CPU: Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz x 2

RAM: 64G ( 4 x 16G ) @ 2400MHz DDR4

GPU: GeForce GTX 1080

1g sector, with GPU

results (1073741824)
seal: addPiece: 3m17.349204737s (5.19 MiB/s)
seal: preCommit: 9m43.802510382s (1.75 MiB/s)
seal: commit: 19m35.8992115s (892 KiB/s)
seal: verify: 115.356288ms
unseal: 3m52.413417314s  (4.41 MiB/s)
generate candidates: 4.441589ms (225 GiB/s)
compute epost proof (cold): 23.495351551s
compute epost proof (hot): 17.159111779s
verify epost proof (cold): 45.09672ms
verify epost proof (hot): 14.235944ms

1g sector, no GPU

results (1073741824)
seal: addPiece: 2m2.863704569s (8.33 MiB/s)
seal: preCommit: 9m52.662955601s (1.73 MiB/s)
seal: commit: 16m23.789922187s (1.04 MiB/s)
seal: verify: 137.224909ms
unseal: 3m52.227311726s  (4.41 MiB/s)
generate candidates: 6.980509ms (143 GiB/s)
compute epost proof (cold): 20.119303527s
compute epost proof (hot): 13.084617774s
verify epost proof (cold): 36.00739ms
verify epost proof (hot): 11.305107ms

CPU AMD EPYC 7551 32-Core Processor @ 1195 MHz - 2x (128 threads)
GPU n/a
MEM 512GB - (16x32GB) DDR4 2667 MHz

./bench --sector-size=1073741824 --no-gpu

seal: addPiece: 53.580885456s (19.1 MiB/s)
seal: preCommit: 6m33.647319689s (2.6 MiB/s)
seal: commit: 14m3.17960682s (1.21 MiB/s)
seal: verify: 101.505876ms
unseal: 3m41.81955434s  (4.62 MiB/s)
generate candidates: 4.589143ms (218 GiB/s)
compute epost proof (cold): 18.358869363s
compute epost proof (hot): 13.524472521s
verify epost proof (cold): 31.224749ms
verify epost proof (hot): 11.356651ms

./bench --sector-size=34359738368 --no-gpu

seal: addPiece: 11m5.055681568s (49.3 MiB/s)
seal: preCommit: 1h28m6.819472562s (6.2 MiB/s)
seal: commit: 2h30m32.996527693s (3.63 MiB/s)
seal: verify: 103.90661ms
unseal: 18m26.750538891s  (29.6 MiB/s)
generate candidates: 4.59161ms (6.81 TiB/s)
compute epost proof (cold): 16.803712004s
compute epost proof (hot): 12.070086278s
verify epost proof (cold): 35.66904ms
verify epost proof (hot): 11.361722ms

@pezinek is that two threadrippers? If so, i would suggest only using one by pinning the benchmark process to one CPU with numactl

CPU: Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz x 2

RAM: 128G ( 8 x 16G ) @ 2133MHz DDR4

GPU: No GPU

numactl --cpunodebind=0 --membind=0 ./bench --sector-size=1073741824 --no-gpu --skip-unseal
numactl --cpunodebind=1 --membind=1 ./bench --sector-size=1073741824 --no-gpu --skip-unseal


numa result:
node 0:
seal: addPiece: 25.649542179s (39.9 MiB/s)
seal: preCommit: 14m12.35822088s (1.2 MiB/s)
seal: commit: 34m3.064612326s (513 KiB/s)
seal: verify: 103.828221ms
unseal: 4.901µs (199 TiB/s)
generate candidates: 2.337277ms (428 GiB/s)
compute epost proof (cold): 1m59.468007739s
compute epost proof (hot): 31.936415202s
verify epost proof (cold): 29.980145ms
verify epost proof (hot): 11.002503ms

node 1:
seal: addPiece: 32.840111138s (31.2 MiB/s)
seal: preCommit: 14m9.756395073s (1.21 MiB/s)
seal: commit: 35m30.486453997s (492 KiB/s)
seal: verify: 105.927692ms
unseal: 4.632µs (211 TiB/s)
generate candidates: 269.401794ms (3.71 GiB/s)
compute epost proof (cold): 31.006845797s
compute epost proof (hot): 31.896077006s
verify epost proof (cold): 31.50822ms
verify epost proof (hot): 10.755815ms

It looks slower than running two in parallel(without Numa), especially the commit phase. @whyrusleeping

CPU: AMD Ryzen Threadripper 3970X 32-Core Processor
Ram: 256GB DDR4 @ 2133 Mhz
Gpu: 1 x RTX 2080 TI

1GiB

GPU

results (1073741824)
seal: addPiece: 6.823021762s (150 MiB/s)
seal: preCommit: 4m18.980537164s (3.95 MiB/s)
seal: commit: 8m38.081979865s (1.98 MiB/s)
seal: verify: 61.414694ms
unseal: 2m20.16371753s  (7.31 MiB/s)
generate candidates: 2.029217ms (493 GiB/s)
compute epost proof (cold): 15.645444254s
compute epost proof (hot): 13.012640201s
verify epost proof (cold): 33.773214ms
verify epost proof (hot): 12.619712ms

No GPU

results (1073741824)
seal: addPiece: 6.937398835s (148 MiB/s)
seal: preCommit: 4m19.506192167s (3.95 MiB/s)
seal: commit: 8m38.47087506s (1.98 MiB/s)
seal: verify: 62.428054ms
unseal: 2m21.358568527s  (7.24 MiB/s)
generate candidates: 1.643263ms (609 GiB/s)
compute epost proof (cold): 9.190877765s
compute epost proof (hot): 6.738571618s
verify epost proof (cold): 17.796773ms
verify epost proof (hot): 6.651537ms

@dignifiedquire install GPU drivers, its not getting used.

CPU: AMD Ryzen Threadripper 3970X 32-Core Processor
Ram: 256GB DDR4 @ 2666 Mhz
Gpu: 2 x RTX 2080 TI

1GiB

GPU

results (1073741824)
seal: addPiece: 6.578052293s (156 MiB/s)
seal: preCommit: 4m19.985479632s (3.94 MiB/s)
seal: commit: 7m0.673755805s (2.43 MiB/s)
seal: verify: 61.642301ms
unseal: 2m3.54665118s  (8.29 MiB/s)
generate candidates: 8.511564ms (117 GiB/s)
compute epost proof (cold): 16.733294052s
compute epost proof (hot): 14.211801921s
verify epost proof (cold): 36.632919ms
verify epost proof (hot): 12.602867ms

CPU: AMD Ryzen 7 3700X 8-Core Processor (16 SMT)
RAM: 128GB DDR4 2133 MHz
GPU: 2x 2080ti

1GiB

GPU

results (1073741824)
seal: addPiece: 7.811869877s (131 MiB/s)
seal: preCommit: 9m26.210096594s (1.81 MiB/s)
seal: commit: 9m4.809578778s (1.88 MiB/s)
seal: verify: 62.521161ms
unseal: 2m26.347872593s  (7 MiB/s)
generate candidates: 9.392964ms (106 GiB/s)
compute epost proof (cold): 10.539019407s
compute epost proof (hot): 8.165584308s
verify epost proof (cold): 23.307615ms
verify epost proof (hot): 6.554879ms

NO GPU

results (1073741824)
seal: addPiece: 7.725134252s (133 MiB/s)
seal: preCommit: 9m31.847032247s (1.79 MiB/s)
seal: commit: 12m8.256964017s (1.41 MiB/s)
seal: verify: 61.497467ms
unseal: 2m26.294993501s  (7 MiB/s)
generate candidates: 1.506113ms (664 GiB/s)
compute epost proof (cold): 12.574304771s
compute epost proof (hot): 10.280571625s
verify epost proof (cold): 20.841289ms
verify epost proof (hot): 6.543938ms

CPU: AMD Ryzen Threadripper 3970X 32-Core Processor
Ram: 64GB DDR4 @ 3200 Mhz
Gpu: RTX 2080 TI

results (1073741824)
seal: addPiece: 6.955012985s (147 MiB/s)
seal: preCommit: 4m12.266940818s (4.06 MiB/s)
seal: commit: 8m26.575607473s (2.02 MiB/s)
seal: verify: 62.873951ms
unseal: 4.899µs  (199 TiB/s)
generate candidates: 1.575725ms (635 GiB/s)
compute epost proof (cold): 8.870124455s
compute epost proof (hot): 6.281588751s
verify epost proof (cold): 35.326687ms
verify epost proof (hot): 13.379037ms

CPU: AMD Ryzen 9 3950X 16-Core Processor
RAM: 128 GB DDR4 @ 3200 Mhz
GPU: RTX 2080

results (34359738368)
seal: addPiece: 3m50.841548985s (142 MiB/s)
seal: preCommit: 2h18m48.137170849s (3.93 MiB/s)
seal: commit: 1h20m48.549095246s (6.76 MiB/s)
seal: verify: 65.490349ms
unseal: 7.515229ms  (4.16 TiB/s)
generate candidates: 9.143537ms (3.42 TiB/s)
compute epost proof (cold): 10.498943658s
compute epost proof (hot): 7.812942964s
verify epost proof (cold): 29.922794ms
verify epost proof (hot): 6.229383ms
results (1073741824)
seal: addPiece: 7.250010742s (141 MiB/s)
seal: preCommit: 5m48.676041669s (2.94 MiB/s)
seal: commit: 7m40.285516095s (2.22 MiB/s)
seal: verify: 61.338126ms
unseal: 4.92µs  (198 TiB/s)
generate candidates: 1.744295ms (573 GiB/s)
compute epost proof (cold): 8.844967724s
compute epost proof (hot): 6.574530215s
verify epost proof (cold): 25.08133ms
verify epost proof (hot): 6.455958ms

CPU: AMD Ryzen Threadripper 2950X 16-Core Processor
RAM: 128 GB DDR4 @ 2133 Mhz
GPU: RTX 2080

1G

results (1073741824)
seal: addPiece: 6.671809642s (153 MiB/s)
seal: preCommit: 8m6.541107303s (2.1 MiB/s)
seal: commit: 10m34.485954891s (1.61 MiB/s)
seal: verify: 70.213845ms
unseal: 2m37.62206002s  (6.5 MiB/s)
generate candidates: 1.713369ms (584 GiB/s)
compute epost proof (cold): 12.639233087s
compute epost proof (hot): 9.791814312s
verify epost proof (cold): 20.727762ms
verify epost proof (hot): 7.516395ms

32G

results (34359738368)
seal: addPiece: 4m40.269704978s (117 MiB/s)
seal: preCommit: 3h41m48.069340531s (2.46 MiB/s)
seal: commit: 2h17m39.759877739s (3.97 MiB/s)
seal: verify: 73.335866ms
generate candidates: 554.492137ms (57.7 GiB/s)
compute epost proof (cold): 17.861240968s
compute epost proof (hot): 10.236637844s
verify epost proof (cold): 34.047689ms
verify epost proof (hot): 8.012858ms

CPU: AMD Ryzen 5 2600X Six-Core Processor
RAM: 32GB DDR4 @ 2933 MHz
GPU: 2x 2080ti

GPU

results (1073741824)
seal: addPiece: 7.514011542s (136 MiB/s)
seal: preCommit: 15m18.181263904s (1.12 MiB/s)
seal: commit: 7m50.960180414s (2.17 MiB/s)
seal: verify: 73.051438ms
unseal: 3m35.133161183s  (4.76 MiB/s)
generate candidates: 4.155487ms (241 GiB/s)
compute epost proof (cold): 10.314847702s
compute epost proof (hot): 7.289115758s
verify epost proof (cold): 22.625963ms
verify epost proof (hot): 7.573291ms

No GPU

results (1073741824)
seal: addPiece: 7.198063681s (142 MiB/s)
seal: preCommit: 14m50.346825136s (1.15 MiB/s)
seal: commit: 17m0.003157163s (1 MiB/s)
seal: verify: 73.096029ms
unseal: 3m36.752855791s  (4.72 MiB/s)
generate candidates: 3.54977ms (282 GiB/s)
compute epost proof (cold): 17.785490635s
compute epost proof (hot): 14.773526336s
verify epost proof (cold): 22.365886ms
verify epost proof (hot): 7.653069ms

CPU: AMD Ryzen 7 3700X 8-Core Processor (16 SMT)
RAM: 128GB DDR4 2133 MHz
GPU: 2x 2080ti (Without PCI Risers, 6m vs 9m)

1GiB

GPU On PCI Express x16 Slots

results (1073741824)
seal: addPiece: 7.525911728s (136 MiB/s)
seal: preCommit: 9m27.349850516s (1.8 MiB/s)
seal: commit: 6m11.510801458s (2.76 MiB/s)
seal: verify: 62.326975ms
unseal: 2m26.932411408s  (6.97 MiB/s)
generate candidates: 1.521648ms (657 GiB/s)
compute epost proof (cold): 8.25150168s
compute epost proof (hot): 6.009122583s
verify epost proof (cold): 21.822715ms
verify epost proof (hot): 6.45406ms

GPU With PCI Riser Cables

results (1073741824)
seal: addPiece: 7.811869877s (131 MiB/s)
seal: preCommit: 9m26.210096594s (1.81 MiB/s)
seal: commit: 9m4.809578778s (1.88 MiB/s)
seal: verify: 62.521161ms
unseal: 2m26.347872593s  (7 MiB/s)
generate candidates: 9.392964ms (106 GiB/s)
compute epost proof (cold): 10.539019407s
compute epost proof (hot): 8.165584308s
verify epost proof (cold): 23.307615ms
verify epost proof (hot): 6.554879ms

CPU: AMD Ryzen Threadripper 3970X 32-Core Processor
Ram: 128GB DDR4 @ 3466 Mhz
Gpu: RTX 2080 TI

GPU

results (1073741824)
seal: addPiece: 6.737909694s (152 MiB/s)
seal: preCommit: 4m7.293631777s (4.14 MiB/s)
seal: commit: 9m21.978126984s (1.82 MiB/s)
seal: verify: 62.352255ms
unseal: 2m16.968646557s  (7.48 MiB/s)
generate candidates: 1.377884ms (726 GiB/s)
compute epost proof (cold): 12.303818863s
compute epost proof (hot): 10.29473069s
verify epost proof (cold): 33.301153ms
verify epost proof (hot): 12.614496ms

No GPU

results (1073741824)
seal: addPiece: 6.727191053s (152 MiB/s)
seal: preCommit: 4m7.932703143s (4.13 MiB/s)
seal: commit: 7m38.773732773s (2.23 MiB/s)
seal: verify: 63.033975ms
unseal: 2m17.485833709s  (7.45 MiB/s)
generate candidates: 1.379607ms (725 GiB/s)
compute epost proof (cold): 8.588115108s
compute epost proof (hot): 6.046917088s
verify epost proof (cold): 34.640765ms
verify epost proof (hot): 12.605079ms

2 Parallel NO GPU runs

results (1073741824)
seal: addPiece: 6.601149475s (155 MiB/s)
seal: preCommit: 6m8.115017964s (2.78 MiB/s)
seal: commit: 14m13.847142298s (1.2 MiB/s)
seal: verify: 62.564774ms
unseal: 2m23.953918776s  (7.11 MiB/s)
generate candidates: 1.422849ms (703 GiB/s)
compute epost proof (cold): 9.501887513s
compute epost proof (hot): 7.058616056s
verify epost proof (cold): 28.35669ms
verify epost proof (hot): 10.630277m

results (1073741824)
seal: addPiece: 6.564463734s (156 MiB/s)
seal: preCommit: 6m10.229348737s (2.77 MiB/s)
seal: commit: 14m59.978782438s (1.14 MiB/s)
seal: verify: 63.772699ms
unseal: 2m20.615873381s  (7.28 MiB/s)
generate candidates: 1.393053ms (718 GiB/s)
compute epost proof (cold): 8.533980543s
compute epost proof (hot): 6.074835486s
verify epost proof (cold): 35.723315ms
verify epost proof (hot): 12.582498ms
AKZX commented

8 Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
32GB DDR4 2400
GTX 1060

./bench --sector-size=1073741824

results (1073741824)
seal: addPiece: 16.729809291s (61.2 MiB/s)
seal: preCommit: 21m4.41726964s (829 KiB/s)
seal: commit: 18m47.907206678s (930 KiB/s)
seal: verify: 91.997195ms
unseal: 4m6.601177228s  (4.15 MiB/s)
generate candidates: 14.097849ms (70.9 GiB/s)
compute epost proof (cold): 23.197453013s
compute epost proof (hot): 18.245150752s
verify epost proof (cold): 19.533122ms
verify epost proof (hot): 7.444096ms

./bench --sector-size=1073741824 --no-gpu

> results (1073741824)
> seal: addPiece: 16.31489131s (62.8 MiB/s)
> seal: preCommit: 21m11.753253189s (825 KiB/s)
> seal: commit: 23m59.814988852s (728 KiB/s)
> seal: verify: 70.531649ms
> unseal: 4m5.850127382s  (4.17 MiB/s)
> generate candidates: 15.467043ms (64.7 GiB/s)
> compute epost proof (cold): 26.442362739s
> compute epost proof (hot): 21.335751157s
> verify epost proof (cold): 20.686405ms
> verify epost proof (hot): 7.488105ms

v25 are now a thing.