michaelmcandrew/civicrm-buildkit-docker

Tests running slow

Closed this issue · 4 comments

michaelmcandrew commented

@totten - I am opening up this issue to discuss slow running tests on buildkit (see #15 (comment) for the origin)

time env CIVICRM_UF=UnitTests phpunit4 tests/phpunit/CRM/Core/AllTests.php on my machine (Intel(R) Core(TM) i7-5500U CPU @ 2.40GHz dual core with 8GB ram) is 14m4.434s based on a sample size of one.

I do not recall getting it much below this even with bare metal on this machine. Not sure what that means but regardless, it would be great to speed things up.

michaelmcandrew commented

The civicrm image is based on php:7.0-apache-jessie. The php cli on this image uses the same php config as apache, which is PHP defaults apart from these lines https://github.com/michaelmcandrew/civicrm-buildkit-docker/blob/master/civicrm/buildkit.ini

One of which sets the memory limit to 1024M.

However, running phpunit with time env CIVICRM_UF=UnitTests phpunit4 -d memory_limit=-1 tests/phpunit/CRM/Core/AllTests.php had no effect on execution time

In other words, still a mystery.

totten commented

  1. @michaelmcandrew , were you running mysql with a ramdisk (tmpfs) or hdd/ssd (persistent volume)?

  2. I forgot to mention the hardware in the previous benchmarks. Reproducing below with hardware info.

  • OS X 10.12, Docker 17.12, civicrm-buildkit-docker, PHP 7.0, MySQL 5.7
    • Physical Hardware: i7-4870HQ; 16GB RAM; PCIe SSD
    • Virtual Hardware: 4 cores; 2gb (4gb)
    • Execution time: 19m44.911s (19m36.352s)
  • OS X 10.12, Nix 2.0, bknix, PHP 5.6, MySQL 5.7
    • Physical Hardware: i7-4870HQ; 16GB RAM; PCIe SSD
    • Virtual Hardware: Not applicable
    • Execution time: 5m35s

  1. FWIW, just to get a ballpark to compare performance in different hardware, we could pull up the single-threaded scores fromcpubenchmark.net: i7-5500U gets 1546 and i7-4870HQ gets 2039 (~30% faster). If the software configurations were identical, we'd expect the 4870HQ to run ~30% faster (~10m). But the software stacks are different, and it's actually ~40% slower (~19m).

  2. IIRC, the biggest time sink in CiviUnitTestCase is resetting mysqld. Although I setup docker-compose.yml to put mysqld in tmpfs, and although I checked that this went live (via docker ... exec and mount/df), this kind of performance discrepancy still feels like a slow mysqld. A smart thing might be to try some synthetic benchmark on each of (a) php processes and (b) mysql i/o -- see if one or the other has a clear performance issue.

michaelmcandrew commented

re 1. nope
re 4: yeah, does anything spring to mind? was also wondering if some xdebug profiling might be of use to point to a particular function that was slow.

we are using the default mysql docker image with out of the box settings.

/googles 'mysql docker performance'

michaelmcandrew commented

did some testing docker vs bare metal on comparable machines. test speed seems within range.