per-run untimed setup
minrk opened this issue · 7 comments
In pyzmq, I want to test the Python overhead of thing like recv_multipart, and I don't want to time the corresponding send/poll. This means I need to have setup per run, not just creating the available namespace once. The timeit template provides exactly this (currently just specified as pass), but it's not exposed to the Benchmark object.
Just kidding - turns out timeit's setup is also run once per timer bundle, so it doesn't help. I still need this, or some approximation of it, because I need the number of sends to match the number of recvs, but I also need to not time the sends.
Maybe need to write a Cython timer class that makes system time calls, the main issue I see is that if the operation that you're timing takes < 5 microseconds that having code like:
setup_call()
s = time.time()
bench_call()
elapsed = time.time() - s
is going to lead to a lot of overhead. Should probably look at https://bitbucket.org/robertkern/line_profiler/src for some inspiration =)
I'm fine making the runs bigger (e.g. recv_multipart 100 times), but the important part is that I can't make the number of sends and recvs match up, so it's not possible to test one side of anything.
yeah, I think it would be good to have fast micro-benchmarks with possible setup. Cython would probably be the right place to have the least timing overhead-- can call eval on a compiled code object, right?
Also feel free to hack on this code as much as you like :)
I've figured out that I can fake it because if I specify ncalls, I know that the timed part should be called 3*ncalls times (I think). But this means that I can never allow timeit to make its informed decision about how many times the test should be run.
What I really want is:
for i in range(ncalls):
setup_call()
tic()
for i in range(ncalls):
bench_call()
toc()Not setup inside the loop. I just want the numbers to match up without having to specify ncalls and timeit.repeat inside all of my setup code.