Enhanced Nginx Memcached Module

Goals

This module is based on the standard Nginx Memcached module, with some additonal features:

Send custom http headers, like Content-Type, Last-Modified. Http headers are stored in memcached, with your body data.
Hash keys to use large keys (> 250 chars, memcached limit)
Store data into memcached, via HTTP request to nginx
Delete data from memcached, via HTTP request to nginx
Flush memcached, via HTTP request to nginx
Get memcached'stats, via HTTP request to nginx
Manage key namespaces, for partial memcached flush
Reply 304 Not Modified for request with If-Modified-Since headers and content with Last-Modified in cache
Reply 304 Not Modified for request with If-None-Match headers and content with ETag in cache
Set custom HTTP code to send redirect

You can find some explanations qbout why this module has been created in this blog post.

Note: base module configuration is identical to the standard Nginx Memcached module.

How to use it

Clone the code:

git clone git://github.com/bpaquet/ngx_http_enhanced_memcached_module.git

Compile Nginx with option in ./configure, as static or dynamic module

--add-module=/my/path/to/my/clone/ngx_http_enhanced_memcached_module
--add-dynamic-module=/my/path/to/my/clone/ngx_http_enhanced_memcached_module

Rebuild Nginx, and enjoy !

You can find configuration example in tests.

This module is tested with Nginx 1.2.x, 1.4.x, 1.6.x, 1.8.x, 1.10.x, 1.11.x and is used in production at fasterize.

Base config

This module has the same base configuration than the standard Nginx Memcached module.

All commands and variables are prepfixed by enhanced.

Custom HTTP Headers

Instead of inserting raw data in memcached, put something like that:

EXTRACT_HEADERS
Content-Type: text/xml

<toto></toto>

Ehanced memcached module will set the header Content-Type to the specified value text-xml instead of the default one. The HTTP body will only contains <toto></toto>.

Before the body, line delimiters have to be \r\n, like in HTTP.

Another example with special chars and two headers:

EXTRACT_HEADERS\r\n
Content-Type: text/html\r\n
Cache-Control:max-age=21600\r\n
\r\n
<html><body>toto</body></html>

You can add multiple headers if you need. If you don't start with EXTRACT_HEADERS, enhanced memcached module will only output the content in the HTTP body.

No modification of nginx config is needed.

Status code

If you want to send a custom status code, (not a 200), just add the header X-Nginx-Status in custom headers. The ehanced memcached module will set the HTTP return code accordingly, and remove this header.

Example, to send a redirect 302:

EXTRACT_HEADERS\r\n
Location: http://www.google.com\r\n
X-Nginx-Status: 302\r\n
\r\n

Hash keys

Memcached keys are limited to 250 chars. To use largest keys, just add in config :

enhanced_memcached_hash_keys_with_md5 on;

The enhanced memcached module will hash keys with md5 algorithm before inserting into memcached, and before getting data from memcached.

Store data into memcached

Add a location in nginx config like that:

location / {
  set $enhanced_memcached_key "$request_uri";
  enhanced_memcached_allow_put on;
  enhanced_memcached_pass memcached_upstream;
}

And send a PUT HTTP request into nginx, with body containing what you want to store in memcached, under the key $enhanced_memcached_key. The set memcached command is used.

Response is a HTTP code 200, with body containing the string STORED.

Note : You can also send get request to this location, data will be extracted from memcached, like in a standard memcached location.

Expiration time

Expire time in memcached is set by default to 0. To set another value, add following line to config :

set $enhanced_memcached_expire 2;

set $enhanced_memcached_expire $http_memcached_expire;

The first one will set a fixed expire value (2 seconds).

The second one will take the expire value to set in memcached from HTTP header Memcached-Expire.

Use the `add` memcached command

If you want to use the add memcached command, add following line in config :

set $enhanced_memcached_use_add 1;

set $enhanced_memcached_use_add $http_memcached_use_add;

The first one will always force the use of add memcached command.

The second one will use the add memcached command only if the HTTP header Memcached-Use-Add is present.

If you send an add command on an existing key, memcached will respond NOT_STORED, and the nginx module will issue a HTTP code 409.

Delete data in memcached

To delete entries in memcached, add a location in nginx config :

location / {
  set $enhanced_memcached_key "$request_uri";
  enhanced_memcached_allow_delete on;
  enhanced_memcached_pass memcached_upstream;
}

And send a DELETE HTTP request to this location.

Response is a HTTP code 200, with body containing the string DELETED, or HTTP code 404, with body NOT_FOUND if the key does not exist in memcached.

Note : It can be used with enhanced_memcached_allow_put in the same location

Flush memcached

To completely flush memcached, add a location in nginx config :

location /flush {
  enhanced_memcached_flush on;
  enhanced_memcached_pass memcached_upstream;
}

And send a GET HTTP request on uri /flush.

Response is a HTTP code 200, with body containing the string OK.

Stats memcached

To get memcached stats, add a location in nginx config :

location /stats {
  enhanced_memcached_stats on;
  enhanced_memcached_pass memcached_upstream;
}

And send a GET HTTP request on uri /stats.

Response is a HTTP code 200, with body containing all stats returned by memcached.

Key namespaces

This feature is an implementation of namespaces : see the memcached documentation for more details.

You can set the namespace to use with a location by adding :

set $enhanced_memcached_key_namespace "$host";

The enhanced memached module will use the HTTP host as namespace for the current location.

You can flush a namespace (in reality, it only increment the key prefix) with a location

location /flush_ns_to {
  set $enhanced_memcached_key "$request_uri";
  set $enhanced_memcached_key_namespace "$host";
  enhanced_memcached_flush_namespace on;
  enhanced_memcached_pass memcached_upstream;
}

304 Not Modified

For request with HTTP Header If-Modified-Since, and associated resource in memcached with HTTP Header Last-Modified, the module will send a 304 Not Modified if resource has not been modified, and if Nginx configuration allows this behaviour.

For request with HTTP Header If-None-Match, and associated resource in memcached with HTTP Header ETag, the module will send a 304 Not Modified if resource has not been modified.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

fasterize/ngx_http_enhanced_memcached_module