A few simple but efficient performance test suites for Apache OpenWhisk. Determines the maximum throughput and end-user latency of the system.
- A standard OpenWhisk system is deployed. Note that the edge nginx router and API Gateway are left out currently. As a consequence, the tests talk directly to the controller.
- Limits are set to 999999 each, for the test's load that means: No throttling at all.
- The deployment uses a docker setup as proposed by the OpenWhisk development team:
overlaydriver and HTTP API enabled via a UNIX port.
The load is driven by either the beautiful loadtest module or the blazingly fast wrk.
The machine provided by Travis has ~2 CPU cores (likely shared through virtualization) and 7.5GB memory.
The latency test determines the end-to-end latency a user experiences when doing a blocking invocation. The action used is a noop so the numbers returned are plain overhead of the OpenWhisk system.
- 1 HTTP request at a time (concurrency: 1)
- 10.000 samples with a single user
- noop action
Note: The throughput number has a 100% correlation with the latency in this case. This test does not serve to determine the maximum throughput of the system.
The throughput test determines the maximum throughput a user can get out of the system while using a single action. The action used again is a noop, so the numbers are plain OpenWhisk overhead. Note that the throughput does not directly correlate to end-to-end latency here, as the system does more processing in the background as it shows to the user in a blocking invocation.
- 4 HTTP requests at a time (concurrency: 4) (basically CPU cores * 2 to exploit some buffering)
- 10.000 samples with a single user
- noop action
To run the suites against your own system (or any other OpenWhisk deployment really), simply use the repsective script in /tests script. Their parameters are defined inline in the respective file.