/pika

Pika is a nosql compatible with redis, it is developed by Qihoo's DBA and infrastructure team

Primary LanguageC++MIT LicenseMIT

Pika

Build Status

Introduction中文

Pika is a persistent huge storage service , compatible with the vast majority of redis interfaces (details), including string, hash, list, zset, set and management interfaces. With the huge amount of data stored, redis may suffer for a capacity bottleneck, and pika was born for solving it. Except huge storage capacity, pika also support master-slave mode by slaveof command, including full and partial synchronization. You can also use pika together with twemproxy or codis(pika has supported data migration in codis,thanks left2right and fancy-rabbit) for distributed Redis solution

UserList

Qihoo 360game Weibo Garena
Apus Ffan Meituan XES
HX XL GWD DYD
YM XM XL YM
MM VIP LK KS

More

Feature

  • huge storage capacity
  • compatible with redis interface, you can migrate to pika easily
  • support master-slave mode (slaveof)
  • various management interfaces

For developer

Releases

The User can download the binary release from releases or compile the source release.

Dependencies

  • snappy - a library for fast data compression
  • glog - google log library

Upgrade your gcc to version at least 4.8 to get C++11 support.

Supported platforms

  • linux - Centos 5&6

  • linux - Ubuntu

If it comes to some missing libs, install them according to the prompts and retry it.

Compile

Upgrade your gcc to version at least 4.8 to get C++11 support.

Get the source code

git clone https://github.com/Qihoo360/pika.git

Then compile pika, all submodules will be updated automatically.

make

Usage

./output/bin/pika -c ./conf/pika.conf

Performance (provided by deep011)

test environment

CPU module:Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz

CPU threads:56

MEMORY:256G

DISK:3T flash

NETWORK:10GBase-T/Full * 2

OS:centos 6.6

benchmark tools

vire-benchmark

Test 1

Purpose

With different number of pika worker threads, we test pika's max QPS.

Condition

pika db_size : 800G

value : 128bytes

Threads are not bound to CPU cores.

Result

Description : Horizontal axis is the number of worker threads; Vertical axis is the QPS. The value size is 128bytes. set3/get7 means 30% set and 70% get.

1

Conclusion

The best pika worker threads number is 20-24.

Test 2

Purpose

With the optimal worker threads number, we test pika's round-trip time.

Condition

pika db_size:800G

value:128bytes

Result

====== GET ======
  10000000 requests completed in 23.10 seconds
  200 parallel clients
  3 bytes payload
  keep alive: 1
99.89% <= 1 milliseconds
100.00% <= 2 milliseconds
100.00% <= 3 milliseconds
100.00% <= 5 milliseconds
100.00% <= 6 milliseconds
100.00% <= 7 milliseconds
100.00% <= 7 milliseconds
432862.97 requests per second
====== SET ======
  10000000 requests completed in 36.15 seconds
  200 parallel clients
  3 bytes payload
  keep alive: 1
91.97% <= 1 milliseconds
99.98% <= 2 milliseconds
99.98% <= 3 milliseconds
99.98% <= 4 milliseconds
99.98% <= 5 milliseconds
99.98% <= 6 milliseconds
99.98% <= 7 milliseconds
99.98% <= 9 milliseconds
99.98% <= 10 milliseconds
99.98% <= 11 milliseconds
99.98% <= 12 milliseconds
99.98% <= 13 milliseconds
99.98% <= 16 milliseconds
99.98% <= 18 milliseconds
99.99% <= 19 milliseconds
99.99% <= 23 milliseconds
99.99% <= 24 milliseconds
99.99% <= 25 milliseconds
99.99% <= 27 milliseconds
99.99% <= 28 milliseconds
99.99% <= 34 milliseconds
99.99% <= 37 milliseconds
99.99% <= 39 milliseconds
99.99% <= 40 milliseconds
99.99% <= 46 milliseconds
99.99% <= 48 milliseconds
99.99% <= 49 milliseconds
99.99% <= 50 milliseconds
99.99% <= 51 milliseconds
99.99% <= 52 milliseconds
99.99% <= 61 milliseconds
99.99% <= 63 milliseconds
99.99% <= 72 milliseconds
99.99% <= 73 milliseconds
99.99% <= 74 milliseconds
99.99% <= 76 milliseconds
99.99% <= 83 milliseconds
99.99% <= 84 milliseconds
99.99% <= 88 milliseconds
99.99% <= 89 milliseconds
99.99% <= 133 milliseconds
99.99% <= 134 milliseconds
99.99% <= 146 milliseconds
99.99% <= 147 milliseconds
100.00% <= 203 milliseconds
100.00% <= 204 milliseconds
100.00% <= 208 milliseconds
100.00% <= 217 milliseconds
100.00% <= 218 milliseconds
100.00% <= 219 milliseconds
100.00% <= 220 milliseconds
100.00% <= 229 milliseconds
100.00% <= 229 milliseconds
276617.50 requests per second

Conclusion

Both the 99.9% get/set RTT are below 2ms.

Test 3

Purpose

With the optimal worker threads number, we test the max qps of different commands.

Condition

pika worker threads:20

key count:10000

field count:100(except list)

value:128bytes

commands execute times:10000000(except lrange)

Result

PING_INLINE: 548606.50 requests per second
PING_BULK: 544573.31 requests per second
SET: 231830.31 requests per second
GET: 512163.91 requests per second
INCR: 230861.56 requests per second
MSET (10 keys): 94991.12 requests per second
LPUSH: 196093.81 requests per second
RPUSH: 195186.69 requests per second
LPOP: 131156.14 requests per second
RPOP: 152292.77 requests per second
LPUSH (needed to benchmark LRANGE): 196734.20 requests per second
LRANGE_10 (first 10 elements): 334448.16 requests per second
LRANGE_100 (first 100 elements): 50705.12 requests per second
LRANGE_300 (first 300 elements): 16745.16 requests per second
LRANGE_450 (first 450 elements): 6787.94 requests per second
LRANGE_600 (first 600 elements): 3170.38 requests per second
SADD: 160885.52 requests per second
SPOP: 128920.80 requests per second
HSET: 180209.41 requests per second
HINCRBY: 153364.81 requests per second
HINCRBYFLOAT: 141095.47 requests per second
HGET: 506791.00 requests per second
HMSET (10 fields): 27777.31 requests per second
HMGET (10 fields): 38998.52 requests per second
HGETALL: 109059.58 requests per second
ZADD: 120583.62 requests per second
ZREM: 161689.33 requests per second
PFADD: 6153.47 requests per second
PFCOUNT: 28312.57 requests per second
PFADD (needed to benchmark PFMERGE): 6166.37 requests per second
PFMERGE: 6007.09 requests per second

Test 4

Purpose

Compare the pika max qps with the redis.

Condition

pika worker threads:20

key count:10000

field count:100(except list)

value:128bytes

commands execute times:10000000(except lrange)

Redis version:3.2.0

Result

1

Documents

  1. Wiki

Contact Us

Mail: g-infra@360.cn

QQ group: 294254078

For more information about Pika, Atlas and some other technology please pay attention to our Hulk platform official account

2