Lotus-bench results thread (v24 params)

Question

Lotus-bench results thread (v24 params)

ytjoe opened this issue 4 years ago · 21 comments

This issue is a place to put lotus-bench results for v24 params. (testnet/3)

# Pull testnet/3 for compilation
FFI_BUILD_FROM_SOURCE=1 make clean all bench

# Maximize cache 
export FIL_PROOFS_MAXIMIZE_CACHING=1

# Run 32g sector test
./bench --sector-size=34359738368
./bench --sector-size=34359738368 --no-gpu

Additionally, please tell us what CPU, GPU, and memory (including speed) you have in your setup.

Answer 1 · 2020-03-31T04:39:45.000Z

# 3700x
# no gpu
# memory 128G 
results (v24) (34359738368)
seal: addPiece: 6m29.658286168s (84.1 MiB/s)
seal: preCommit phase 1: 3h51m36.3388762s (2.36 MiB/s)
seal: preCommit phase 2: 4h3m46.195089422s (2.24 MiB/s)
seal: commit phase 1: 3.329454884s (9.61 GiB/s)
seal: commit phase 2: 1h35m36.694015932s (5.71 MiB/s)
seal: verify: 64.685677ms
unseal: 5.300121ms  (5.9 TiB/s)
generate candidates: 642.129991ms (49.8 GiB/s)
compute epost proof (cold): 10.123346465s
compute epost proof (hot): 9.211648893s
verify epost proof (cold): 41.600541ms
verify epost proof (hot): 15.11789ms

Answer 2 · 2020-03-31T05:53:23.000Z

@ytQiao preCommit phase 1 How can it be completed in such a short time. My test is completed in 23 hours. Is there any way?

Answer 3 · 2020-03-31T05:58:54.000Z

@ytQiao preCommit phase 1 How can it be completed in such a short time. My test is completed in 23 hours. Is there any way?

Compile on bench test machine, use AMD processor, memory requirement is greater than 128G, swap partition is recommended to be greater than 128G (specific can be tested later), add parameters during compilation and bench running

Answer 4 · 2020-03-31T06:04:26.000Z

Pull testnet/3 for compilation

FFI_BUILD_FROM_SOURCE=1 make clean all bench

Maximize cache

export FIL_PROOFS_MAXIMIZE_CACHING=1

Do you add these two parameters when compiling and running? Or something else?

Answer 5 · 2020-03-31T06:06:37.000Z

Pull testnet/3 for compilation

FFI_BUILD_FROM_SOURCE=1 make clean all bench

Maximize cache

export FIL_PROOFS_MAXIMIZE_CACHING=1

Do you add these two parameters when compiling and running? Or something else?

Yes. There's nothing else.

Answer 6 · 2020-03-31T06:09:26.000Z

OK, I retested after compiling, thank you for your answer

Answer 7 · 2020-04-01T02:36:58.000Z

# 3700x
# no gpu
# memory 128G 
results (v24) (34359738368)
seal: addPiece: 6m29.658286168s (84.1 MiB/s)
seal: preCommit phase 1: 3h51m36.3388762s (2.36 MiB/s)
seal: preCommit phase 2: 4h3m46.195089422s (2.24 MiB/s)
seal: commit phase 1: 3.329454884s (9.61 GiB/s)
seal: commit phase 2: 1h35m36.694015932s (5.71 MiB/s)
seal: verify: 64.685677ms
unseal: 5.300121ms  (5.9 TiB/s)
generate candidates: 642.129991ms (49.8 GiB/s)
compute epost proof (cold): 10.123346465s
compute epost proof (hot): 9.211648893s
verify epost proof (cold): 41.600541ms
verify epost proof (hot): 15.11789ms

results (v24) (34359738368)
seal: addPiece: 9m35.637211091s (56.9 MiB/s)
seal: preCommit phase 1: 4h52m16.234933759s (1.87 MiB/s)
seal: preCommit phase 2: 3h42m59.305512531s (2.45 MiB/s)
seal: commit phase 1: 1m50.349284613s (297 MiB/s)
seal: commit phase 2: 2h50m1.595939973s (3.21 MiB/s)
seal: verify: 298.305989ms
unseal: 418.540821ms  (76.5 GiB/s)
generate candidates: 1.525869374s (21 GiB/s)
compute epost proof (cold): 9.030044981s
compute epost proof (hot): 8.965704682s
verify epost proof (cold): 55.595438ms
verify epost proof (hot): 19.265659ms

Thank you for your proposal, the efficiency has been greatly improved, but the difference between the same methods is 1 hour. Is there any other factors that will affect it? Is there anything that can be improved?

Answer 8 · 2020-04-01T02:41:56.000Z

WARN: sha-ni not available, falling back

It reported the WRAN ，Will it have any effect？

Answer 9 · 2020-04-01T02:47:32.000Z

@Tylertest8 Can I show you your configuration information？

Answer 10 · 2020-04-01T02:50:59.000Z

@s1eke More information is needed, but I think you need to compile on the tested machine.

Answer 11 · 2020-04-01T02:52:24.000Z

@ytQiao

AMD Ryzen 3970X + RAM 128G + SWAP 300G + HDD + NOGPU

Answer 12 · 2020-04-01T02:58:27.000Z

@Tylertest8 I used some nvme as storage, which may be one of the reasons，you can try it

Answer 13 · 2020-04-01T03:00:41.000Z

Test data source: magik6k

# TR 3970x + 2x 2080ti
results (v24) (34359738368)
seal: addPiece: 6m8.798820562s (88.9 MiB/s)
seal: preCommit phase 1: 3h59m13.609729554s (2.28 MiB/s)
seal: preCommit phase 2: 52m3.442064626s (10.5 MiB/s)
seal: commit phase 1: 7.536231307s (4.25 GiB/s)
seal: commit phase 2: 37m25.869552159s (14.6 MiB/s)
seal: verify: 57.648867ms
generate candidates: 573.01274ms (55.8 GiB/s)
compute epost proof (cold): 15.398034616s
compute epost proof (hot): 14.742154327s
verify epost proof (cold): 39.170784ms
verify epost proof (hot): 16.905623ms

Answer 14 · 2020-04-01T03:38:10.000Z

@ytQiao Ok thank you i need to keep trying

Answer 15 · 2020-04-01T06:04:48.000Z

specifications of my computer：

CPU：Intel Xeon E5-2683 v4  @ 3.000GHz * 2
RAM：32G * 24
GPU：NVIDIA Tesla T4

This is my order of operations：

# FFI_BUILD_FROM_SOURCE=1 make clean all bench
# export FIL_PROOFS_MAXIMIZE_CACHING=1
# export BELLMAN_CUSTOM_GPU="Tesla T4:2560"
# ./bench --storage-dir=/lotus/tmp --sector-size=34359738368

and output log:

2020-03-31T23:39:50.561-0400    INFO    lotus-bench     lotus-bench/main.go:213 Writing piece into sector...
2020-04-01T00:33:39.561-0400    INFO    lotus-bench     lotus-bench/main.go:227 Running replication(1)...
WARN: sha-ni not available, falling back

@ytQiao

Answer 16 · 2020-04-02T17:15:42.000Z

@s1eke Only amd processors have Sha instruction set, which Intel does not have

Answer 17 · 2020-04-03T02:54:28.000Z

CPU：Intel Xeon E5-2683 v4 @ 3.000GHz * 2
RAM：32G * 24
GPU：NVIDIA Tesla T4

results (v24) (34359738368)
seal: addPiece: 54m59.980201531s (9.93 MiB/s)
seal: preCommit phase 1: 35h40m44.350375592s (261 KiB/s)
seal: preCommit phase 2: 2h26m25.74103585s (3.73 MiB/s)
seal: commit phase 1: 670.996849ms (47.7 GiB/s)
seal: commit phase 2: 2h7m21.05272074s (4.29 MiB/s)
seal: verify: 86.39903ms
unseal: 3.210476ms  (9.73 TiB/s)
generate candidates: 846.492758ms (37.8 GiB/s)
compute epost proof (cold): 12.787342364s
compute epost proof (hot): 11.96141447s
verify epost proof (cold): 49.16003ms
verify epost proof (hot): 30.577227ms

This is too slow😂

Answer 18 · 2020-04-03T03:04:40.000Z

@s1eke Adding parameters FIL_PROOFS_MAXIMIZE_CACHING = 1 can also increase speed,but the performance improvement is not as big as AMD. I think.

Answer 19 · 2020-04-03T03:10:20.000Z

It's already increase speed😂

Answer 20 · 2020-04-06T16:49:08.000Z

AMD Ryzen 7 3700X 8-Core Processor + 128G RAM + Some nvme

results (v24) (34359738368)
seal: addPiece: 6m18.579275883s (86.6 MiB/s)
seal: preCommit phase 1: 4h8m30.565015104s (2.2 MiB/s) 
seal: preCommit phase 2: 3h5m10.332082143s (2.95 MiB/s)
seal: commit phase 1: 5.557466928s (5.76 GiB/s)

AMD Ryzen 5 3600X 6-Core Processor + 128G RAM + Some nvme

results (v24) (34359738368)
seal: addPiece: 6m18.215139138s (86.6 MiB/s)
seal: preCommit phase 1: 4h10m31.226008064s (2.18 MiB/s)
seal: preCommit phase 2: 4h9m33.291920304s (2.19 MiB/s)
seal: commit phase 1: 1.123879236s (28.5 GiB/s)

TR 3970x + 128G RAM + 2x 2080ti + Some nvme

results (v24) (34359738368)
seal: addPiece: 6m8.798820562s (88.9 MiB/s)
seal: preCommit phase 1: 3h59m13.609729554s (2.28 MiB/s)
seal: preCommit phase 2: 52m3.442064626s (10.5 MiB/s)
seal: commit phase 1: 7.536231307s (4.25 GiB/s)
seal: commit phase 2: 37m25.869552159s (14.6 MiB/s)
seal: verify: 57.648867ms
generate candidates: 573.01274ms (55.8 GiB/s)
compute epost proof (cold): 15.398034616s
compute epost proof (hot): 14.742154327s
verify epost proof (cold): 39.170784ms
verify epost proof (hot): 16.905623ms

Answer 21 · 2020-04-22T16:48:12.000Z

v25 params are now out in testnet/3.

You can view and submit benchmarks to https://filecoin-benchmarks.on.fleek.co/