/nchan

Fast, horizontally scalable, multiprocess pub/sub queuing server and proxy for HTTP, long-polling, Websockets and EventSource (SSE), powered by Nginx.

Primary LanguageCOtherNOASSERTION

NCHAN

https://nchan.slact.net

Nchan is a scalable, flexible pub/sub server for the modern web, built as a module for the Nginx web server. It can be configured as a standalone server, or as a shim between your application and tens, thousands, or millions of live subscribers. It can buffer messages in memory, on-disk, or via Redis. All connections are handled asynchronously and distributed among any number of worker processes. It can also scale to many nginx server instances with Redis.

Messages are published to channels with HTTP POST requests or websockets, and subscribed also through websockets, long-polling, EventSource (SSE), old-fashioned interval polling, and more. Each subscriber can listen to up to 255 channels per connection, and can be optionally authenticated via a custom application url. An events meta channel is also available for debugging.

For use in a web browser, you can try the NchanSubscriber.js wrapper library. It supports Long-Polling, EventSource, and resumable Websockets -- or you can build your own.

Status and History

The latest Nchan release is v0.99.16 (June 10, 2016) (changelog). This is a beta release. There may be some bugs but Nchan is already stable and well-tested.

The first iteration of Nchan was written in 2009-2010 as the Nginx HTTP Push Module, and was vastly refactored into its present state in 2014-2016. The present release is in the testing phase. The core features and old functionality are thoroughly tested and stable. Some of the new functionality, specifically channel events is still experimental and may be a bit buggy.

Please help make the entire codebase ready for production use! Report any quirks, bugs, leaks, crashes, or larvae you find.

Upgrade from Nginx HTTP Push Module

Although Nchan is backwards-compatible with all Push Module configuration directives, some of the more unusual and rarely used settings have been disabled and will be ignored (with a warning). See the upgrade page for a detailed list of changes and improvements, as well as a full list of incompatibilities.

Does it scale?

benchmarking internal subscriber response times

Yes it does. Like Nginx, Nchan can easily handle as much traffic as you can throw at it. I've tried to benchmark it, but my benchmarking tools are much slower than Nchan. The data I've gathered is on how long Nchan itself takes to respond to every subscriber after publishing a message -- this excludes TCP handshake times and internal HTTP request parsing. Basically, it measures how Nchan scales assuming all other components are already tuned for scalability. The graphed data are averages of 5 runs with 50-byte messages.

With a well-tuned OS and network stack on commodity server hardware, expect to handle upwards of 300K concurrent subscribers per second at minimal CPU load. Nchan can also be scaled out to multiple Nginx instances using the Redis storage engine.

Currently, Nchan's performance is limited by available memory bandwidth. This can be improved significantly in future versions with fewer allocations and the use of contiguous memory pools. Please consider supporting Nchan to speed up the work of memory cache optimization.

Download

Packages

From Source

Grab the latest copy of Nginx from nginx.org. Grab the latest Nchan source from github. Follow the instructions for building Nginx, except during the configure stage, add

./configure --add-module=path/to/nchan ...

If you're using Nginx > 1.9.11, you can build Nchan as a dynamic module with --add-dynamic-module=path/to/nchan

Run make, make install, and enjoy. (Caution, contents may be hot.)

Conceptual Overview

The basic unit of most pub/sub solutions is the messaging channel. Nchan is no different. Publishers send messages to channels with a certain channel id, and subscribers subscribed to those channels receive them. Some number of messages may be buffered for a time in a channel's message buffer before they are deleted. Pretty simple, right?

Well... the trouble is that nginx configuration does not deal with channels, publishers, and subscribers. Rather, it has several sections for incoming requests to match against server and location sections. Nchan configuration directives map servers and locations onto channel publishing and subscribing endpoints:

#very basic nchan config
worker_processes 5;

http {  
  server {
    listen       80;
    
    location = /sub {
      nchan_subscriber;
      nchan_channel_id foobar;
    }
    
    location = /pub {
      nchan_publisher;
      nchan_channel_id foobar;
    }
  }
}

The above maps requests to the URI /sub onto the channel foobar's subscriber endpoint , and similarly /pub onto channel foobar's publisher endpoint.

Publisher Endpoints

Publisher endpoints are Nginx config locations with the nchan_publisher directive.

Messages can be published to a channel by sending HTTP POST requests with the message contents to the publisher endpoint locations. You can also publish messages through a Websocket connection to the same location.

Publishing Messages

Requests and websocket messages are responded to with information about the channel at time of message publication. Here's an example from publishing with curl:

>  curl --request POST --data "test message" http://127.0.0.1:80/pub

 queued messages: 5
 last requested: 18 sec. ago
 active subscribers: 0
 last message id: 1450755280:0

The response can be in plaintext (as above), JSON, or XML, based on the request's Accept header:

> curl --request POST --data "test message" -H "Accept: text/json" http://127.0.0.2:80/pub

 {"messages": 6, "requested": 55, "subscribers": 0, "last_message_id": "1450755317:0" }

Websocket publishers also receive the same responses when publishing, with the encoding determined by the Accept header present during the handshake.

The response code for an HTTP request is 202 Accepted if no subscribers are present at time of publication, or 201 Created if at least 1 subscriber was present.

Metadata can be added to a message when using an HTTP POST request for publishing. A Content-Type header will be associated as the message's content type (and output to Long-Poll, Interval-Poll, and multipart/mixed subscribers). A X-EventSource-Event header can also be used to associate an EventSource event: line value with a message.

Other Publisher Endpoint Actions

HTTP GET requests return channel information without publishing a message. The response code is 200 if the channel exists, and 404 otherwise:

> curl --request POST --data "test message" http://127.0.0.2:80/pub
  ...

> curl -v --request GET -H "Accept: text/json" http://127.0.0.2:80/pub

 {"messages": 1, "requested": 7, "subscribers": 0, "last_message_id": "1450755421:0" }

HTTP DELETE requests delete a channel and end all subscriber connections. Like the GET requests, this returns a 200 status response with channel info if the channel existed, and a 404 otherwise.

Subscriber Endpoint

Subscriber endpoints are Nginx config locations with the nchan_subscriber directive.

Nchan supports several different kinds of subscribers for receiving messages: Websocket, EventSource (Server Sent Events), Long-Poll, Interval-Poll. HTTP chunked transfer, and HTTP multipart/mixed.

  • Long-Polling

    The tried-and-true server-push method supported by every browser out there.
    Initiated by sending an HTTP GET request to a channel subscriber endpoint.
    The long-polling subscriber walks through a channel's message queue via the built-in cache mechanism of HTTP clients, namely with the "Last-Modified" and "Etag" headers. Explicitly, to receive the next message for given a long-poll subscriber response, send a request with the "If-Modified-Since" header set to the previous response's "Last-Modified" header, and "If-None-Match" likewise set to the previous response's "Etag" header.
    Sending a request without a "If-Modified-Since" or "If-None-Match" headers returns the oldest message in a channel's message queue, or waits until the next published message, depending on the value of the nchan_subscriber_first_message config directive.
    A message's associated content type, if present, will be sent to this subscriber with the Content-Type header.

  • Interval-Polling

    Works just like long-polling, except if the requested message is not yet available, immediately responds with a 304 Not Modified. There is no way to differentiate between long-poll and interval-poll subscriber requests, so long-polling must be disabled for a subscriber location if you wish to use interval-polling.

  • Websocket

    Bidirectional communication for web browsers. Part of the HTML5 spec. Nchan supports the latest protocol version 13 (RFC 6455).
    Initiated by sending a websocket handshake to the desired subscriber endpoint location.
    If the websocket connection is closed by the server, the close frame will contain the HTTP response code and status line describing the reason for closing the connection. Server-initiated keep-alive pings can be configured with the nchan_websocket_ping_interval config directive. Websocket extensions are not yet supported.
    Messages published through a websocket connection can be forwarded to an upstream application with the nchan_publisher_upstream_request config directive.
    Websocket subscribers can use the custom ws+meta.nchan subprotocol to receive message metadata with messages, making websocket connections resumable. Messages received with this subprotocol are of the form

    id: message_id
    content-type: message_content_type
    \n
    message_data
    

    The content-type: line may be omitted.

  • EventSource

    Also known as Server-Sent Events or SSE, it predates Websockets in the HTML5 spec, and is a very simple protocol.
    Initiated by sending an HTTP GET request to a channel subscriber endpoint with the "Accept: text/event-stream" header.
    Each message data: segment will be prefaced by the message id: .
    To resume a closed EventSource connection from the last-received message, one should start the connection with the "Last-Event-ID" header set to the last message's id.
    Unfortunately, browsers don't support setting this header for an EventSource object, so by default the last message id is set either from the "Last-Event-Id" header or the last_event_id url query string argument.
    This behavior can be configured via the nchan_subscriber_last_message_id config.
    A message's associated event type, if present, will be sent to this subscriber with the event: line.

  • The multipart/mixed MIMEtype was conceived for emails, but hey, why not use it for HTTP? It's easy to parse and includes metadata with each message.
    Initiated by including an Accept: multipart/mixed header.
    The response headers and the unused "preamble" portion of the response body are sent right away, with the boundary string generated randomly for each subscriber. Each subsequent message will be sent as one part of the multipart message, and will include the message time and tag (Last-Modified and Etag) as well as the optional Content-Type headers.
    Each message is terminated with the next multipart message's boundary without a trailing newline. While this conforms to the multipart spec, it is unusual as multipart messages are defined as starting, rather than ending with a boundary.
    A message's associated content type, if present, will be sent to this subscriber with the Content-Type header.

  • HTTP Raw Stream

    A simple subscription method similar to the streaming subscriber of the Nginx HTTP Push Stream Module. Messages are appended to the response body, separated by a newline or configurable by nchan_subscriber_http_raw_stream_separator.

  • This subscription method uses the chunked Transfer-Encoding to receive messages.
    Initiated by explicitly including chunked in the TE header:
    TE: chunked (or TE: chunked;q=?? where the qval > 0)
    The response headers are sent right away, and each message will be sent as an individual chunk. Note that because a zero-length chunk terminates the transfer, zero-length messages will not be sent to the subscriber.
    Unlike the other subscriber types, the chunked subscriber cannot be used with http/2 because it dissallows chunked encoding.

PubSub Endpoint

PubSub endpoints are Nginx config locations with the nchan_pubsub directive.

A combination of publisher and subscriber endpoints, this location treats all HTTP GET requests as subscribers, and all HTTP POST as publishers. One simple use case is an echo server:

  location = /pubsub {
    nchan_pubsub;
    nchan_channel_id foobar;
  }

A more applicable setup may set different publisher and subscriber channel ids:

  location = /pubsub {
    nchan_pubsub;
    nchan_publisher_channel_id foo;
    nchan_subscriber_channel_id bar;
  }

Here, subscribers will listen for messages on channel foo, and publishers will publish messages to channel bar. This can be useful when setting up websocket proxying between web clients and your application.

The Channel ID

So far the examples have used static channel ids, which is not very useful in practice. It can be set to any nginx variable, such as a querystring argument, a header value, or a part of the location url:

  location = /sub_by_ip {
    #channel id is the subscriber's IP address
    nchan_subscriber;
    nchan_channel_id $remote_addr;
  }
  
  location /sub_by_querystring {
    #channel id is the query string parameter chanid
    # GET /sub/sub_by_querystring?foo=bar&chanid=baz will have the channel id set to 'baz'
    nchan_subscriber;
    nchan_channel_id $arg_chanid;
  }

  location ~ /sub/(\w+)$ {
    #channel id is the word after /sub/
    # GET /sub/foobar_baz will have the channel id set to 'foobar_baz'
    # I hope you know your regular expressions...
    nchan_subscriber;
    nchan_channel_id $1; #first capture of the location match
  }

Channel Multiplexing

Any subscriber location can be an endpoint for up to 255 channels. Messages published to all the specified channels will be delivered in-order to the subscriber. There are two ways to enable multiplexing:

Up to 7 channel ids can be specified for the nchan_channel_id or nchan_channel_subscriber_id config directive:

  location ~ /multisub/(\w+)/(\w+)$ {
    nchan_subscriber;
    nchan_channel_id "$1" "$2" "common_channel";
    #GET /multisub/foo/bar will be subscribed to:
    # channels 'foo', 'bar', and 'common_channel',
    #and will receive messages from all of the above.
  }

For more than 7 channels, nchan_channel_id_split_delimiter can be used to split the nchan_channel_id or nchan_channel_subscriber_id into up to 255 individual channel ids:

  location ~ /multisub-split/(.*)$ {
    nchan_subscriber;
    nchan_channel_id "$1";
    nchan_channel_id_split_delimiter ",";
    #GET /multisub-split/foo,bar,baz,a will be subscribed to:
    # channels 'foo', 'bar', 'baz', and 'a'
    #and will receive messages from all of the above.
  }

DELETE requests on any channel are forwarded to relevant multi-channel subscribers, and their connections are terminated.

Publishing to multiple channels with a single request is also possible, with similar configuration:

  location ~ /multipub/(\w+)/(\w+)$ {
    nchan_publisher;
    nchan_channel_id "$1" "$2" "another_channel";
  }

Configuration Directives

  • nchan_channel_id
    arguments: 1 - 7
    default: (none)
    context: server, location, if

    Channel id for a publisher or subscriber location. Can have up to 4 values to subscribe to up to 4 channels.

  • nchan_channel_id_split_delimiter
    arguments: 1
    default: (none)
    context: server, location, if

    Split the channel id into several ids for multiplexing using the delimiter string provided.

  • nchan_eventsource_event
    arguments: 1
    default: (none)
    context: server, location, if

    Set the EventSource event: line to this value. When used in a publisher location, overrides the published message's X-EventSource-Event header and associates the message with the given value. When used in a subscriber location, overrides all messages' associated event: string with the given value.

  • nchan_longpoll_multipart_response [ off | on | raw ]
    arguments: 1
    default: off
    context: server, location, if

    when set to 'on', enable sending multiple messages in a single longpoll response, separated using the multipart/mixed content-type scheme. If there is only one available message in response to a long-poll request, it is sent unmodified. This is useful for high-latency long-polling connections as a way to minimize round-trips to the server. When set to 'raw', sends multiple messages using the http-raw-stream message separator.

  • nchan_publisher [ http | websocket ]
    arguments: 0 - 2
    default: http websocket
    context: server, location, if
    legacy name: push_publisher

    Defines a server or location as a publisher endpoint. Requests to a publisher location are treated as messages to be sent to subscribers. See the protocol documentation for a detailed description.

  • nchan_publisher_channel_id
    arguments: 1 - 7
    default: (none)
    context: server, location, if

    Channel id for publisher location.

  • nchan_publisher_upstream_request <url>
    arguments: 1
    context: server, location, if

    Send POST request to internal location (which may proxy to an upstream server) with published message in the request body. Useful for bridging websocket publishers with HTTP applications, or for transforming message via upstream application before publishing to a channel.
    The upstream response code determine how publishing will proceed. A 200 OK will publish the message from the upstream response's body. A 304 Not Modified will publish the message as it was received from the publisher. A 204 No Content will result in the message not being published.

  • nchan_pubsub [ http | websocket | eventsource | longpoll | intervalpoll | chunked | multipart-mixed | http-raw-stream ]
    arguments: 0 - 6
    default: http websocket eventsource longpoll chunked multipart-mixed
    context: server, location, if

    Defines a server or location as a pubsub endpoint. For long-polling, GETs subscribe. and POSTs publish. For Websockets, publishing data on a connection does not yield a channel metadata response. Without additional configuration, this turns a location into an echo server.

  • nchan_subscriber [ websocket | eventsource | longpoll | intervalpoll | chunked | multipart-mixed | http-raw-stream ]
    arguments: 0 - 5
    default: websocket eventsource longpoll chunked multipart-mixed
    context: server, location, if
    legacy name: push_subscriber

    Defines a server or location as a channel subscriber endpoint. This location represents a subscriber's interface to a channel's message queue. The queue is traversed automatically, starting at the position defined by the nchan_subscriber_first_message setting.
    The value is a list of permitted subscriber types.

  • nchan_subscriber_channel_id
    arguments: 1 - 7
    default: (none)
    context: server, location, if

    Channel id for subscriber location. Can have up to 4 values to subscribe to up to 4 channels.

  • nchan_subscriber_compound_etag_message_id
    arguments: 1
    default: off
    context: server, location, if

    Override the default behavior of using both Last-Modified and Etag headers for the message id.
    Enabling this option packs the entire message id into the Etag header, and discards
    Last-Modified and If-Modified-Since headers.

  • nchan_subscriber_first_message [ oldest | newest ]
    arguments: 1
    default: oldest
    context: server, location, if

    Controls the first message received by a new subscriber. 'oldest' returns the oldest available message in a channel's message queue, 'newest' waits until a message arrives.

  • nchan_subscriber_http_raw_stream_separator <string>
    arguments: 1
    default:
    context: server, location, if

    Message separator string for the http-raw-stream subscriber. Automatically terminated with a newline character.

  • nchan_subscriber_last_message_id
    arguments: 1 - 5
    default: $http_last_event_id $arg_last_event_id
    context: server, location, if

    If If-Modified-Since and If-None-Match headers are absent, set the message id to the first non-empty of these values. Used primarily as a workaround for the inability to set the first Last-Message-Id of a web browser's EventSource object.

  • nchan_subscriber_message_id_custom_etag_header
    arguments: 1
    default: (none)
    context: server, location, if

    Use a custom header instead of the Etag header for message ID in subscriber responses. This setting is a hack, useful when behind a caching proxy such as Cloudflare that under some conditions (like using gzip encoding) swallow the Etag header.

  • nchan_subscriber_timeout <number> (seconds)
    arguments: 1
    default: 0 (none)
    context: http, server, location, if
    legacy name: push_subscriber_timeout

    Maximum time a subscriber may wait for a message before being disconnected. If you don't want a subscriber's connection to timeout, set this to 0. When possible, the subscriber will get a response with a 408 Request Timeout status; otherwise the subscriber will simply be disconnected.

  • nchan_websocket_ping_interval <number> (seconds)
    arguments: 1
    default: 0 (none)
    context: server, location, if

    Interval for sending websocket ping frames. Disabled by default.

  • nchan_access_control_allow_origin <string>
    arguments: 1
    default: *
    context: http, server, location

    Set the Cross-Origin Resource Sharing (CORS) Access-Control-Allow-Origin header to this value. If the publisher or subscriber request's Origin header does not match this value, respond with a 403 Forbidden.

  • nchan_authorize_request <url>
    arguments: 1
    context: server, location, if

    Send GET request to internal location (which may proxy to an upstream server) for authorization of a publisher or subscriber request. A 200 response authorizes the request, a 403 response forbids it.

  • nchan_max_reserved_memory <size>
    arguments: 1
    default: 32M
    context: http
    legacy name: push_max_reserved_memory

    The size of the shared memory chunk this module will use for message queuing and buffering.

  • nchan_message_buffer_length <number>
    arguments: 1
    default: 10
    context: http, server, location
    legacy names: push_max_message_buffer_length, push_message_buffer_length

    Publisher configuration setting the maximum number of messages to store per channel. A channel's message buffer will retain a maximum of this many most recent messages.

  • nchan_message_timeout <time>
    arguments: 1
    default: 1h
    context: http, server, location
    legacy name: push_message_timeout

    Publisher configuration setting the length of time a message may be queued before it is considered expired. If you do not want messages to expire, set this to 0. Applicable only if a nchan_publisher is present in this or a child context.

  • nchan_redis_ping_interval
    arguments: 1
    default: 4m
    context: http

    Send a keepalive command to redis to keep the Nchan redis clients from disconnecting. Set to 0 to disable.

  • nchan_redis_url
    arguments: 1
    default: 127.0.0.1:6379
    context: http

    The path to a redis server, of the form 'redis://:password@hostname:6379/0'. Shorthand of the form 'host:port' or just 'host' is also accepted.

  • nchan_store_messages [ on | off ]
    arguments: 1
    default: on
    context: http, server, location, if
    legacy name: push_store_messages

    Publisher configuration. "off" is equivalent to setting nchan_channel_buffer_length 0

  • nchan_use_redis [ on | off ]
    arguments: 1
    default: off
    context: http, server, location

    Use redis for message storage at this location.

  • nchan_channel_group <string>
    arguments: 1
    default: (none)
    context: server, location, if
    legacy name: push_channel_group

    Because settings are bound to locations and not individual channels, it is useful to be able to have channels that can be reached only from some locations and never others. That's where this setting comes in. Think of it as a prefix string for the channel id.

  • nchan_subscribe_existing_channels_only [ on | off ]
    arguments: 1
    default: off
    context: http, server, location
    legacy name: push_authorized_channels_only

    Whether or not a subscriber may create a channel by sending a request to a push_subscriber location. If set to on, a publisher must send a POST or PUT request before a subscriber can request messages on the channel. Otherwise, all subscriber requests to nonexistent channels will get a 403 Forbidden response.

  • nchan_channel_event_string <string>
    arguments: 1
    default: $nchan_channel_event $nchan_channel_id
    context: server, location, if

    Contents of channel event message

  • nchan_channel_events_channel_id
    arguments: 1
    context: server, location, if

    Channel id where nchan_channel_id's events should be sent. Events like subscriber enqueue/dequeue, publishing messages, etc. Useful for application debugging. The channel event message is configurable via nchan_channel_event_string. The channel group for events is hardcoded to 'meta'.

  • nchan_max_channel_id_length <number>
    arguments: 1
    default: 512
    context: http, server, location
    legacy name: push_max_channel_id_length

    Maximum permissible channel id length (number of characters). Longer ids will be truncated.

  • nchan_max_channel_subscribers <number>
    arguments: 1
    default: 0 (unlimited)
    context: http, server, location
    legacy name: push_max_channel_subscribers

    Maximum concurrent subscribers.

  • nchan_channel_timeout
    arguments: 1
    context: http, server, location
    legacy name: push_channel_timeout

    Amount of time an empty channel hangs around. Don't mess with this setting unless you know what you are doing!

  • nchan_storage_engine [ memory | redis ]
    arguments: 1
    default: memory
    context: http, server, location

    Development directive to completely replace default storage engine. Don't use unless you are an Nchan developer.

Contribute

Please support this project with a donation to keep me warm through the winter. I accept bitcoin at 15dLBzRS4HLRwCCVjx4emYkxXcyAPmGxM3 . Other donation methods can be found at https://nchan.slact.net