Investigate newer Siege versions
fredemmott opened this issue · 8 comments
fredemmott commented
At first glance, 3.1.0 has no issues - hopefully the blacklist can be loosened to 3.0.*
need to make sure that for major applications, RPS, byte size, and response code ratios are similar (or have an explanation)
fredemmott commented
refs #48
centminmod commented
would love to see the results of such an investigation .. siege 3.1.4 is the latest version now
reedy commented
Ubuntu 12.04, hhvm 3.6.6
siege 2.70
reedy@oss-perf:~/oss-performance$ hhvm perf.php --mediawiki --hhvm=`which hhvm` --no-proxygen --i-am-not-benchmarking
**************************************************
** 2016-01-11 14:37:25 UTC
** Configuration: MediaWikiTarget on /usr/bin/hhvm
**************************
** 2016-01-11 14:37:25 UTC
** Installing framework
**************************
** 2016-01-11 14:37:34 UTC
** There is no setUpTest
**************************
** 2016-01-11 14:37:34 UTC
** Starting Nginx
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
**************************
** 2016-01-11 14:37:34 UTC
** Starting PHP Engine
running hphp...
all files saved in /tmp/hhvm-nginxmhxFtc ...
running hphp took 1'11" (71278461 us) wall time
mapping self...
mapping self took 0'00" (3070 us) wall time
loading static content...
loaded file cache from /tmp/hhvm-nginxmhxFtc/static.content
loading static content took 0'00" (9675 us) wall time
**************************
** 2016-01-11 14:38:48 UTC
** Running sanity check
*******************************************
** 2016-01-11 14:38:51 UTC
** Starting Siege for single request warmup
** SIEGE 2.70
** Preparing 1 concurrent users for battle.
The server is now under siege...
done.
Transactions: 300 hits
Availability: 100.00 %
Elapsed time: 29.09 secs
Data transferred: 53.87 MB
Response time: 0.10 secs
Transaction rate: 10.31 trans/sec
Throughput: 1.85 MB/sec
Concurrency: 1.00
Successful transactions: 285
Failed transactions: 0
Longest transaction: 7.73
Shortest transaction: 0.02
******************************************
** 2016-01-11 14:39:20 UTC
** Starting Siege for multi request warmup
** SIEGE 2.70
** Preparing 200 concurrent users for battle.
The server is now under siege...
Lifting the server siege.. done.
Transactions: 2094 hits
Availability: 100.00 %
Elapsed time: 59.80 secs
Data transferred: 442.87 MB
Response time: 5.52 secs
Transaction rate: 35.02 trans/sec
Throughput: 7.41 MB/sec
Concurrency: 193.43
Successful transactions: 1898
Failed transactions: 0
Longest transaction: 9.77
Shortest transaction: 0.36
****************************
** 2016-01-11 14:40:20 UTC
** Clearing nginx access.log
*******************************
** 2016-01-11 14:40:20 UTC
** Starting Siege for benchmark
** SIEGE 2.70
** Preparing 200 concurrent users for battle.
The server is now under siege...
Lifting the server siege.. done.
Transactions: 2451 hits
Availability: 100.00 %
Elapsed time: 59.86 secs
Data transferred: 453.24 MB
Response time: 4.68 secs
Transaction rate: 40.95 trans/sec
Throughput: 7.57 MB/sec
Concurrency: 191.68
Successful transactions: 2251
Failed transactions: 0
Longest transaction: 14.34
Shortest transaction: 1.48
**************************
** 2016-01-11 14:41:22 UTC
** Collecting results
**************************
** 2016-01-11 14:41:25 UTC
** Collecting TC/PCRE data
***************************
** 2016-01-11 14:41:25 UTC
** There is no tearDownTest
{
"Combined": {
"Siege requests": 2451,
"Siege wall sec": 4.68,
"Siege RPS": 40.95,
"Siege successful requests": 2251,
"Siege failed requests": 0,
"Nginx hits": 2651,
"Nginx avg bytes": 179834.88042248,
"Nginx avg time": 4.0732470765749,
"Nginx P50 time": 3.978,
"Nginx P90 time": 5.922,
"Nginx P95 time": 6.609,
"Nginx P99 time": 7.927,
"Nginx 200": 2261,
"Nginx 499": 190,
"Nginx 404": 200,
"canonical": 0
}
}
siege 3.1.4
reedy@oss-perf:~/oss-performance$ hhvm perf.php --mediawiki --hhvm=`which hhvm` --no-proxygen --i-am-not-benchmarking --skip-version-checks
**************************************************
** 2016-01-11 14:53:48 UTC
** Configuration: MediaWikiTarget on /usr/bin/hhvm
**************************
** 2016-01-11 14:53:48 UTC
** Installing framework
**************************
** 2016-01-11 14:53:56 UTC
** There is no setUpTest
**************************
** 2016-01-11 14:53:56 UTC
** Starting Nginx
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
**************************
** 2016-01-11 14:53:56 UTC
** Starting PHP Engine
running hphp...
all files saved in /tmp/hhvm-nginxBstzPr ...
running hphp took 1'02" (62088676 us) wall time
mapping self...
mapping self took 0'00" (2477 us) wall time
loading static content...
loaded file cache from /tmp/hhvm-nginxBstzPr/static.content
loading static content took 0'00" (8268 us) wall time
**************************
** 2016-01-11 14:55:01 UTC
** Running sanity check
*******************************************
** 2016-01-11 14:55:03 UTC
** Starting Siege for single request warmup
** SIEGE 3.1.4
** Preparing 1 concurrent users for battle.
The server is now under siege...
done.
Transactions: 300 hits
Availability: 100.00 %
Elapsed time: 25.00 secs
Data transferred: 53.96 MB
Response time: 0.08 secs
Transaction rate: 12.00 trans/sec
Throughput: 2.16 MB/sec
Concurrency: 1.00
Successful transactions: 285
Failed transactions: 0
Longest transaction: 6.55
Shortest transaction: 0.01
******************************************
** 2016-01-11 14:55:28 UTC
** Starting Siege for multi request warmup
** SIEGE 3.1.4
** Preparing 200 concurrent users for battle.
The server is now under siege...
Lifting the server siege.. done.
Transactions: 2423 hits
Availability: 100.00 %
Elapsed time: 60.05 secs
Data transferred: 450.61 MB
Response time: 4.78 secs
Transaction rate: 40.35 trans/sec
Throughput: 7.50 MB/sec
Concurrency: 193.05
Successful transactions: 2224
Failed transactions: 0
Longest transaction: 13.00
Shortest transaction: 0.32
****************************
** 2016-01-11 14:56:29 UTC
** Clearing nginx access.log
*******************************
** 2016-01-11 14:56:29 UTC
** Starting Siege for benchmark
** SIEGE 3.1.4
** Preparing 200 concurrent users for battle.
The server is now under siege...
Lifting the server siege.. done.
Transactions: 2351 hits
Availability: 100.00 %
Elapsed time: 59.14 secs
Data transferred: 448.55 MB
Response time: 4.85 secs
Transaction rate: 39.75 trans/sec
Throughput: 7.58 MB/sec
Concurrency: 192.85
Successful transactions: 2151
Failed transactions: 0
Longest transaction: 14.63
Shortest transaction: 2.10
**************************
** 2016-01-11 14:57:30 UTC
** Collecting results
**************************
** 2016-01-11 14:57:33 UTC
** Collecting TC/PCRE data
***************************
** 2016-01-11 14:57:33 UTC
** There is no tearDownTest
{
"Combined": {
"Siege requests": 2351,
"Siege wall sec": 4.85,
"Siege RPS": 39.75,
"Siege successful requests": 2151,
"Siege failed requests": 0,
"Nginx hits": 2551,
"Nginx avg bytes": 184646.40846727,
"Nginx avg time": 4.1381387691102,
"Nginx P50 time": 4.062,
"Nginx P90 time": 5.578,
"Nginx P95 time": 6.198,
"Nginx P99 time": 7.993,
"Nginx 200": 2153,
"Nginx 499": 198,
"Nginx 404": 200,
"canonical": 0
}
}
reedy commented
| Result | 2.70 | 3.1.4 |
|---|---|---|
| Siege requests | 2451 | 2351 |
| Siege wall sec | 4.68 | 4.85 |
| Siege RPS | 40.95 | 39.75 |
| Siege successful requests | 2251 | 2151 |
| Siege failed requests | 0 | 0 |
| Nginx hits | 2651 | 2551 |
| Nginx avg bytes | 179834.88042248 | 184646.40846727 |
| Nginx avg time | 4.0732470765749 | 4.1381387691102 |
| Nginx P50 time | 3.978 | 4.062 |
| Nginx P90 time | 5.922 | 5.578 |
| Nginx P95 time | 6.609 | 6.198 |
| Nginx P99 time | 7.927 | 7.993 |
| Nginx 200 | 2261 | 2153 |
| Nginx 499 | 190 | 198 |
| Nginx 404 | 200 | 200 |
| canonical | 0 | 0 |
fredemmott commented
It looks promising, but I'll need to check all of the supported frameworks
mofarrell commented
The update to the blacklist is in master already.
reedy commented
Um... Are you sure? https://github.com/hhvm/oss-performance/blame/master/base/Siege.php#L31
The patch was to blacklist 3.0 versions, not just 3 versions
Isn't this still going to complain for 3.1, unless you pass --skip-version-checks ?
reedy commented
See also JoeDog/siege#15 (comment)