libmc is a memcached client library for Python without any other dependencies in runtime. It's mainly written in C++ and Cython. libmc can be considered as a drop in replacement for libmemcached and python-libmemcached.
libmc is developing and maintaining by Douban Inc. Currently, It is working in production environment, powering all web traffics in douban.com. Realtime benchmark result is available on travis.
For users:
pip install libmc
Usage:
import libmc
mc = libmc.Client(['localhost:11211', 'localhost:11212'])
mc.set('foo', 'bar')
assert mc.get('foo') == 'bar'
Under the hood, libmc consists of 2 parts: an internal fully-functional
memcached client implementation in C++ and a Cython wrapper around that
implementation. Dynamic memory allocation and memory-copy are slow, so
we tried our best to avoid them. The set_multi
command is not
natively supported by the memcached
protocol.
Some techniques are applied to make set_multi
command extremely fast
in libmc (compared to some other similiar libraries).
import libmc
from libmc import (
MC_HASH_MD5, MC_POLL_TIMEOUT, MC_CONNECT_TIMEOUT, MC_RETRY_TIMEOUT
)
mc = libmc.Client(
['localhost:11211', 'localhost:11212'], do_split=True,
comp_threshold=0, noreply=False, prefix=None,
hash_fn=MC_HASH_MD5, failover=False
)
mc.config(MC_POLL_TIMEOUT, 100) # 100 ms
mc.config(MC_CONNECT_TIMEOUT, 300) # 300 ms
mc.config(MC_RETRY_TIMEOUT, 5) # 5 s
do_split
: Memcached server will refuse to store value if size >= 1MB, ifdo_split
is enabled, large value (< 10 MB) will be splitted into several blocks. If the value is too large (>= 10 MB), it will not be stored. default:True
comp_threshold
: All kinds of values will be encoded into string buffer. Ifbuffer length > comp_threshold > 0
, it will be compressed using zlib. Ifcomp_threshold = 0
, string buffer will never be compressed using zlib. default:0
noreply
: Whether to enable memcached'snoreply
behaviour. default:False
prefix
: The key prefix. default:''
hash_fn
: hashing function for keys. possible values:MC_HASH_MD5
MC_HASH_FNV1_32
MC_HASH_FNV1A_32
MC_HASH_CRC_32
default:
MC_HASH_MD5
NOTE: fnv1_32, fnv1a_32, crc_32 implementations in libmc are per each spec, but they're not compatible with corresponding implementions in libmemcached.
failover
: Whether to failover to next server when current server is not available. default:False
MC_POLL_TIMEOUT
Timeout parameter used during set/get procedure. (default:300
ms)MC_CONNECT_TIMEOUT
Timeout parameter used when connecting to memcached server on initial phase. (default:100
ms)MC_RETRY_TIMEOUT
When a server is not available dur to server-end error. libmc will try to establish the broken connection in everyMC_RETRY_TIMEOUT
s until the connection is back to live.(default:5
s)
NOTE: The hashing algorithm for host mapping on continuum is always md5.
Feel free to send a Pull Request. For feature requests or any questions, please open an Issue.
For SECURITY DISCLOSURE, please disclose the information responsibly by sending an email to security@douban.com directly instead of creating a GitHub issue.
No. But if you like, you can write a wrapper for PHP based on the C++ implementation.
No. Only Memcached ASCII protocol is supported currently.
Before libmc, we're using python-libmemcached, which is a python extention for libmemcached. libmemcached is quite weird and buggy. After nearly one decade, there're still some unsolved bugs.
libmc is a single-threaded memcached client. If you initialize a libmc
client in one thread but reuse that in another thread, a Python
Exception ThreadUnsafe
will raise in Python.
Yes, with the help of greenify,
libmc is friendly to gevent. Read tests/shabby/gevent_issue.py
for
details.
- Thanks to @fahrenheit2539 and the llvm project for the standalone. SmallVector implementation.
- Thanks to @miloyip for the high performance i64toa implementation.
- Thanks to Ivan Novikov for the research in THE NEW PAGE OF INJECTIONS BOOK: MEMCACHED INJECTIONS.
- Thanks to the PolarSSL project for the md5 implementation.
- Thanks to @lericson for the benchmark script in pylibmc.
- Thanks to the libmemcached project and some other projects possibly not mentioned here.
Copyright (c) 2014-2015, Douban Inc. All rights reserved.
Licensed under a BSD license: https://github.com/douban/libmc/blob/master/LICENSE.txt