ethereum/tests

Benchmark modexp precompiled contract in different clients

Closed this issue · 24 comments

pirapira commented

There is an opinion that the gas cost formula of modexp precompiled contract can be improved. To find out the reasonal gas cost formula we need to benchmark modexp contract of different clients up to blockgaslimit.

timxor commented

@pirapira i can take that, looks fun. will work on it tonight

timxor commented

@pirapira what option do you think would be better? Go it solo or join forces....maybe solo because then it goes horizontal not vertical :)

pirapira commented

That's what I thought. The second option will be the quickest.

timxor commented

agreed. on it.

holiman commented

I have a geth-benchmark implementation for modexp at https://github.com/holiman/go-ethereum/tree/benchmark_metro , the only thing I'm missing now are some proper values to use in testing. And I guess the other clients ought to use the same. Have either of you located any good inputs? I've found a few from the EIP examples, but I don't think they're very representative.

  • ❤️1
timxor commented

This is awesome thanks for sharing, can we make a generator for testing all ranges or what do you think the best way to test it? Are there already tests in the test repo that we can use? @holiman

holiman commented

I've added some inputs to my branch the other day

gumb0 commented

@tcsiwula Hi, is there any update from you on this?

timxor commented

Hey @gumb0, I should have the parity one done by friday this week. I was interviewing for jobs the last two weeks. Is the c++ client still in active development? cc: @pirapira

pirapira commented

@tcsiwula I sent you an email.

winsvega commented

to test benchmark we have to run a scenario with calling a function/opcode on different range of input parameters and eventually build up a graph "exec time / input parameters value"
I think making a static contract in state test is not a best option for determining the reasonable gas cost formula

  • 👍1
timxor commented

Hello everyone, here is an update from me. Let me know where I should focus my attention/efforts next. =)

My most recent (August 9th) parity benchmark results for {bn_128_add, bn_128_mul, ...}.

Here are the steps to recreate and run it locally on your computer (linux/ubuntu).

cc: @pirapira @holiman @arkpar @Souptacular

  • 👍1
pirapira commented

On my machine, I get 6 times different numbers for ecadd from
go-ethereum

BenchmarkPrecompiledBn256Add/chfast1-Gas=0-4                       20000             90297 ns/op

and Parity

test bn_128_add          ... bench:      14,527 ns/iter (+/- 413)

(will edit with links to steps)

  • 😕1
pirapira commented

The original purpose of this issue has been accomplished, as modexp gas formula has been determined.

Since we are here, maybe we can open issues for remaining problems and paste links here.

  • 👍1
timxor commented

@pirapira wonderful, yeah that sounds good to me. What other precompile benchmarks would be useful to have that would give us a complete and accurate picture?

pirapira commented

@tcsiwula since ecadd and ecmul are supposed to be of constant gas costs, it would be great if we can come up with some numbers already.

gcolvin commented

@tcsiwula just pointed me at this issue. I've been working with @s-matthew-english on improving the test suite at https://github.com/ethereum/cpp-ethereum/tree/develop/test/unittests/performance so that more opcodes and precompiles and more clients can be easily added to the suite. The more the merrier. The advantage of this suite is that it will work with any client with a command-line program that can interpret EVM binaries.

winsvega commented

@pirapira status?

pirapira commented

No idea.

winsvega commented

I thought we came to the final gas price for this contract. @holiman

holiman commented

Yes, for all precompiles. I think we can close this ticket.. (?)

pirapira commented

OK.