rackerlabs/mimic

Python 3 slowness

Closed this issue · 2 comments

ksheedlo commented

Mimic seems to run a lot slower under Python 3 than Python 2, at least when running tox tests.

Here's a tox run using -e py34:

Ran 542 tests in 10.378s

PASSED (successes=542)

And using -e py27:

Ran 542 tests in 5.885s

PASSED (successes=542)

Questions I have about this:

  1. Is this expected when modernizing a code base for Python 3? i.e., is it a Mimic-specific issue or not?
  2. Is the slowness coming from Mimic's test suite or production code, or both? This may require us to follow up with some perf analysis.
glyph commented

Mimic seems to run a lot slower under Python 3 than Python 2, at least when running tox tests.

Tests are not a benchmark and should not be treated as one. So I think the correct resolution for this is to ignore it.

Is this expected when modernizing a code base for Python 3? i.e., is it a Mimic-specific issue or not?

It is almost certainly not a mimic-specific issue. The bytes object is missing many high-level features in python 3 which allowed for certain CPython-specific micro-optimizations within Twisted, and Mimic is probably feeling their loss. Plus, Unicode strings are (sometimes, this is complicated) 4x as large in memory and therefore take longer to copy and manipulate than bytestrings.

Is the slowness coming from Mimic's test suite or production code, or both? This may require us to follow up with some perf analysis.

If we want to set performance targets, then we should create a benchmark suite, not measure test run time. But I suspect that this would be a waste of time, since Mimic's closest competitor for most of its functionality is on the order of 1000x slower :-). However, having more automated test coverage is never a bad thing, so I definitely would not object to a performance suite.

Closing this ticket though, because test suite benchmark performance is not a goal in itself. If this is a bug, it's a bug in CPython.

glyph commented

Also worth noting: PyPy is certainly the fastest way to run mimic (and the only platform we should really invest effort in performance testing) but also has the slowest test suite by far :)