incorrect benchmark ?
jjmr opened this issue · 20 comments
Hi,
I have ran the benchmarks in my local machine with nodejs 0.12.2, node_redis 0.12.1 and the master branch of ioredis and I have got the following results:
$ npm run bench
> ioredis@1.2.0 bench /tmp/ioredis
> matcha benchmarks/*.js
child_process: customFds option is deprecated, use stdio instead.
simple set
109,742 op/s » ioredis
105,823 op/s » node_redis
simple get
88,683 op/s » ioredis
102,495 op/s » node_redis
simple get with pipeline
10,600 op/s » ioredis
13,012 op/s » node_redis
lrange 100
68,533 op/s » ioredis
72,331 op/s » node_redis
Suites: 4
Benches: 8
Elapsed: 53,973.10 ms
I have ran the benchmark with and without hiredis having similar performance.
Those results show better or equal performance between node_redis and ioredis. However, in the documentation, the performance of ioredis is quite better than node_redis. Any idea?
I've run the benchmark in my computers and our servers, and the results are similar. However there are many factors that can impact the results, so it's hard to say why yours are different. I just submit a commit to print system info as well when running benchmark, and here's the result on my laptop(MacBook Pro, Retina, 15-inch, Late 2013):
ioredis git:master ❯ npm run bench ✹
> ioredis@1.2.0 bench /Users/luin/opensource/ioredis
> matcha benchmarks/*.js
child_process: customFds option is deprecated, use stdio instead.
==========================
ioredis: 1.2.0
node_redis: 0.12.1
CPU: 8
OS: darwin x64
==========================
simple set
75,846 op/s » ioredis
43,185 op/s » node_redis
simple get
75,463 op/s » ioredis
41,240 op/s » node_redis
simple get with pipeline
12,329 op/s » ioredis
4,146 op/s » node_redis
lrange 100
58,273 op/s » ioredis
45,782 op/s » node_redis
Suites: 4
Benches: 8
Elapsed: 61,455.94 ms
Could you pull the latest commit and run the benchmark again?
I'm also seeing great results:
nodejs: 0.10.25
ioredis: 1.2.0
os: ubuntu x86_64 (Vagrant)
> ioredis@1.2.0 bench /vagrant/
> matcha benchmarks/*.js
simple set
104,896 op/s » ioredis
55,791 op/s » node_redis
simple get
95,655 op/s » ioredis
54,011 op/s » node_redis
simple get with pipeline
19,481 op/s » ioredis
7,032 op/s » node_redis
lrange 100
80,833 op/s » ioredis
46,986 op/s » node_redis
Suites: 4
Benches: 8
Elapsed: 57,291.53 ms
My becnhmark updated:
$ npm run bench
> ioredis@1.2.0 bench /tmp/ioredis
> matcha benchmarks/*.js
child_process: customFds option is deprecated, use stdio instead.
==========================
ioredis: 1.2.0
node_redis: 0.12.1
CPU: 8
OS: linux x64
==========================
simple set
102,374 op/s » ioredis
100,401 op/s » node_redis
simple get
69,507 op/s » ioredis
90,876 op/s » node_redis
simple get with pipeline
10,670 op/s » ioredis
11,413 op/s » node_redis
lrange 100
57,448 op/s » ioredis
67,558 op/s » node_redis
Suites: 4
Benches: 8
Elapsed: 54,293.49 ms
My OS is Linux Mint.
If I run the same benchmark more than once I can see different values and in general ioredis and node_redis have similar performance in each suite (more or less)
Which redis-server version are you using? Is Redis running on the same machine as the client?
redis server is in the same machine and the version is 2.8.4
Node.js v0.12.2, early 2009 MacBook Pro, redis-server 3.0.0 running on the same machine:
macbook:ioredis luigi$ npm run bench
> ioredis@1.2.0 bench /Users/luigi/Desktop/repos/ioredis
> matcha benchmarks/*.js
child_process: customFds option is deprecated, use stdio instead.
==========================
ioredis: 1.2.0
node_redis: 0.12.1
CPU: 2
OS: darwin x64
==========================
simple set
23,289 op/s » ioredis
27,762 op/s » node_redis
simple get
22,474 op/s » ioredis
26,978 op/s » node_redis
simple get with pipeline
4,312 op/s » ioredis
2,758 op/s » node_redis
lrange 100
18,529 op/s » ioredis
22,116 op/s » node_redis
Suites: 4
Benches: 8
Elapsed: 67,757.50 msVitalys-MacBook-Pro:mservice-smtp vitaly$ redis-cli -v
redis-cli 2.8.19
Vitalys-MacBook-Pro:ioredis vitaly$ npm run bench
> ioredis@1.2.0 bench /Users/vitaly/projects/ioredis
> matcha benchmarks/*.js
==========================
ioredis: 1.2.0
node_redis: 0.12.1
CPU: 8
OS: darwin x64
==========================
simple set
86,129 op/s » ioredis
75,235 op/s » node_redis
simple get
82,437 op/s » ioredis
74,842 op/s » node_redis
simple get with pipeline
15,923 op/s » ioredis
8,383 op/s » node_redis
lrange 100
61,766 op/s » ioredis
54,674 op/s » node_redis
Suites: 4
Benches: 8
Elapsed: 56,919.29 ms
Vitalys-MacBook-Pro:ioredis vitaly$ node -v
v0.10.38
I added a link to this issue to the benchmark section in README, and will leave this issue open util we know more about the reason of the difference of results.
At a guess, ioredis has better performance when the server is able to handle higher concurrency.
redis-cli 2.8.17
debian jessie
> ioredis@1.5.4 bench /home/tests/ioredis
> matcha benchmarks/*.js
child_process: customFds option is deprecated, use stdio instead.
==========================
ioredis: 1.5.4
node_redis: 0.12.1
CPU: 8
OS: linux x64
==========================
simple set
133,532 op/s » ioredis
156,371 op/s » node_redis
simple get
133,728 op/s » ioredis
135,891 op/s » node_redis
simple get with pipeline
17,511 op/s » ioredis
15,210 op/s » node_redis
lrange 100
90,610 op/s » ioredis
106,528 op/s » node_redis
Suites: 4
Benches: 8
Elapsed: 53,750.60 ms
Updated for mac / iojs 2.2.1. Redis 3.0.1
Vitalys-MacBook-Pro:ioredis vitaly$ npm run bench
> ioredis@1.5.4 bench /Users/vitaly/projects/ioredis
> matcha benchmarks/*.js
child_process: customFds option is deprecated, use stdio instead.
==========================
ioredis: 1.5.4
node_redis: 0.12.1
CPU: 8
OS: darwin x64
==========================
simple set
113,225 op/s » ioredis
88,521 op/s » node_redis
simple get
98,778 op/s » ioredis
80,953 op/s » node_redis
simple get with pipeline
14,946 op/s » ioredis
7,907 op/s » node_redis
lrange 100
86,316 op/s » ioredis
81,688 op/s » node_redis
Suites: 4
Benches: 8
Elapsed: 58,897.17 ms
Vitalys-MacBook-Pro:ioredis vitaly$ node -v
v2.2.1
你好，我本地机器是win7系统，使用npm install ioredis命令下载ioredis到了本地，我本地有redis-server,版本为2.6.12,请问怎样跑这个ioredis/benchmarks/single_node.js文件？直接在ioredis/benchmarks/目录下输入 node single_node.js会报错：
ReferenceError: suite is not defined
at Object. (C:\Users\Administrator\node_modules\ioredis\benchmark
s\single_node.js:30:1)
at Module._compile (module.js:460:26)
at Object.Module._extensions..js (module.js:478:10)
at Module.load (module.js:355:32)
at Function.Module._load (module.js:310:12)
at Function.Module.runMain (module.js:501:10)
at startup (node.js:129:16)
at node.js:814:3
npm run bench. 文档里有写 https://github.com/luin/ioredis#benchmark
输入npm run bench后回车，依然不成功，控制打印如下：
ioredis@1.7.5 bench C:\Users\Administrator\node_modules\ioredis
matcha benchmarks/*.js
'matcha' 不是内部或外部命令，也不是可运行的程序
或批处理文件。
npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs
\node_modules\npm\bin\npm-cli.js" "run" "bench"
npm ERR! node v0.12.0
npm ERR! npm v2.5.1
npm ERR! code ELIFECYCLE
npm ERR! ioredis@1.7.5 bench: matcha benchmarks/*.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the ioredis@1.7.5 bench script 'matcha benchmarks/.js'.
npm ERR! This is most likely a problem with the ioredis package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! matcha benchmarks/.js
npm ERR! You can get their info via:
npm ERR! npm owner ls ioredis
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! C:\Users\Administrator\node_modules\ioredis\benchmarks\npm-debug.lo
g
然后我npm i matcha后，再次输入npm run bench后回车，还是不成功，控制打印如下：
ioredis@1.7.5 bench C:\Users\Administrator\node_modules\ioredis
matcha benchmarks/*.js
child_process: customFds option is deprecated, use stdio instead.
module.js:338
throw err;
^
Error: Cannot find module 'C:\Users\Administrator\node_modules\ioredis\benchmark
s*.js'
at Function.Module._resolveFilename (module.js:336:15)
at Function.require.resolve (module.js:388:19)
at C:\Users\Administrator\node_modules\ioredis\node_modules\matcha\bin_matc
ha:80:18
at Array.map (native)
at Object.runSuite (C:\Users\Administrator\node_modules\ioredis\node_modules
\matcha\bin_matcha:79:17)
at Program.parse (C:\Users\Administrator\node_modules\ioredis\node_modules\m
atcha\node_modules\electron\lib\electron\program.js:262:27)
at Object. (C:\Users\Administrator\node_modules\ioredis\node_modu
les\matcha\bin_matcha:36:9)
at Module._compile (module.js:460:26)
at Object.Module._extensions..js (module.js:478:10)
at Module.load (module.js:355:32)
npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs
\node_modules\npm\bin\npm-cli.js" "run" "bench"
npm ERR! node v0.12.0
npm ERR! npm v2.5.1
npm ERR! code ELIFECYCLE
npm ERR! ioredis@1.7.5 bench: matcha benchmarks/*.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the ioredis@1.7.5 bench script 'matcha benchmarks/.js'.
npm ERR! This is most likely a problem with the ioredis package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! matcha benchmarks/.js
npm ERR! You can get their info via:
npm ERR! npm owner ls ioredis
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! C:\Users\Administrator\node_modules\ioredis\benchmarks\npm-debug.lo
g
这是肿么了
npm run 时会设置系统环境变量保证 devdeps 里面的依赖会作为全局依赖。Windows 会有问题。可以直接用 npm i -g matcha
然后再次执行npm run bench
这是为什么呢~~~~(>_<)~~~~
直接执行 matcha single_node.js 吧
非常感谢，已经ok了，接下来我去centOS跑跑看看结果
node 4.2.1:
version: pull-request #184
12:09:59 nos:~/Projects/Github/ioredis 13 (redis-dev-dep)
~> npm run bench
> ioredis@1.10.0 bench /Users/nos/Projects/Github/ioredis
> matcha benchmarks/*.js
(node) child_process: options.customFds option is deprecated. Use options.stdio instead.
==========================
ioredis: 1.10.0
node_redis: 2.2.5
CPU: 8
OS: darwin x64
==========================
simple set
111,917 op/s » ioredis
87,705 op/s » node_redis
simple get
107,242 op/s » ioredis
86,911 op/s » node_redis
simple get with pipeline
9,105 op/s » ioredis
8,371 op/s » node_redis
lrange 100
10,572 op/s » ioredis
8,917 op/s » node_redis
publish
108,210 op/s » ioredis
87,479 op/s » node_redis
subscribe
48,086 op/s » ioredis
46,994 op/s » node_redis
Suites: 6
Benches: 12
Elapsed: 89,436.12 ms
Here are results with Mid 2014 MBPro 2.5 GHz i7, 16GB RAM, NVIDIA GeForce GT 750M 2GB. Running against Redis 3.0.0 on the same machine. Any special configs needed for Redis to optimize it for ioredis?
> matcha benchmarks/*.js
(node) child_process: options.customFds option is deprecated. Use options.stdio instead.
==========================
ioredis: 1.14.0
node_redis: 0.12.1
CPU: 8
OS: darwin x64
node version: v5.3.0
current commit: 581524d
==========================
simple set
34,464 op/s » ioredis
49,481 op/s » node_redis
simple get
33,857 op/s » ioredis
40,399 op/s » node_redis
simple set
36,601 op/s » ioredis
52,078 op/s » node_redis
simple get
41,298 op/s » ioredis
51,527 op/s » node_redis
simple get with pipeline
2,769 op/s » ioredis
9,503 op/s » node_redis
lrange 100
34,479 op/s » ioredis
65,115 op/s » node_redis
publish
40,704 op/s » ioredis
49,343 op/s » node_redis
subscribe
30,397 op/s » ioredis
40,692 op/s » node_redis
Suites: 6
Benches: 12
Elapsed: 96,849.92 ms
I'm running Redis via Docker (something very close to docker run -d --name=redis -p 127.0.0.1:6379:6379 redis which automagically pulls the stock Redis image and then starts it listening locally). I'm posting because my results are much more in favor of ioredis than node_redis.
Redis server v=3.0.7
==========================
ioredis: 1.15.0
node_redis: 2.4.2
CPU: 8
OS: linux x64
node version: v4.3.0
current commit: 13ac1c4
==========================
simple set
99,635 op/s » ioredis
55,776 op/s » node_redis
simple get
86,711 op/s » ioredis
53,505 op/s » node_redis
simple get with pipeline
8,082 op/s » ioredis
7,081 op/s » node_redis
lrange 100
14,140 op/s » ioredis
9,686 op/s » node_redis
publish
92,252 op/s » ioredis
56,567 op/s » node_redis
subscribe
41,925 op/s » ioredis
33,334 op/s » node_redis
Suites: 6
Benches: 12
Elapsed: 92,940.75 ms