multistream-select

Friendly protocol multiplexing. It enables a multicodec to be negotiated between two entities.

Motivation
Implementations
Maintainers
Contribute
License

Motivation

Some protocols have sub-protocols or protocol-suites. Often, these sub protocols are optional extensions. Selecting which protocol to use -- or even knowing what is available to chose from -- is not simple.

What if there was a protocol that allowed mounting or nesting other protocols, and made it easy to select which protocol to use?

At the operating system level, protocol selection is usually accomplished using numbered ports, with some out-of-band agreement on the mapping of port numbers to protocols. For example, HTTP requests will use TCP port 80 unless otherwise specified.

multistream-select brings this concept into the "protocol level", which gives you the flexibility to evolve a mapping of human-readable protocol names to the semantics you need, which may change over time.

This is especially useful in environments where connections to arbitrary OS ports is difficult or impossible, for example because one or more parties is behind a NAT. In such cases, a single connection with a stream multiplexer will allow you to open many independent communication channels, but you'll need some mechanism for deciding what protocol to use for each channel. If you support multiple stream multiplexers, you can even use multistream-select to decide which one to use in the first place.

Protocol

The actual protocol is very simple. It is a multistream protocol itself, with a multistream header. And it has a set of other protocols available to be used by the remote side. The remote side must enter:

> <multistream-header-for-multistream>
> <multistream-header-for-whatever-protocol-that-we-want-to-speak>

for example:

> /multistream/1.0.0
> /ipfs/kad/1.0.0

The <multistream-header-for-multistream> ensures a protocol selection is happening.
The <multistream-header-for-whatever-protocol-is-then-selected> hopefully describes a valid protocol listed. Otherwise we return a na("not available") error:

na\n

# in hex (note the varint prefix = 3)
# 0x036e610a

for example:

 # open connection + send multistream headers, inc for a protocol not available
> /multistream/1.0.0
> /some-protocol-that-is-not-available

# open connection + signal protocol not available.
< /multistream/1.0.0
< na

# send a selection of a valid protocol + upgrade the conn and send traffic
> /ipfs/kad/1.0.0
> <kad-dht-traffic>
> ...

# receive a selection of the protocol + sent traffic
< /ipfs/kad/1.0.0
< <kad-dht-traffic>
< ...

Note 1: Every multistream message is a "length-prefixed-message", which means that every message is preprended by a varint that describes the size of the message.

Note 2: Every multistream message is appended by a \n character, this character is included in the byte count that is accounted by the prepended varint.

Note 3: Although multistream-select uses a "call and response" pattern to propose and confirm protocol selection, it is possible for both sides to send multistream headers at the same time. This is especially likely for the initial <multistream-header-for-multistream>, which both sides may optimistically send as soon as the connection is opened, without waiting for the other side to send it first.

Listing

Implementations MAY support "listing" of the available protocols. A list message is simply:

ls\n

# in hex (note the varint prefix = 3)
0x036c730a

So a remote side asking for a protocol listing would look like this:

# request
<multistream-header-for-multistream-select>
ls\n

# response
<varint-total-response-size-in-bytes>
<varint-protocol-length><multicodec-of-available-protocol>\n
<varint-protocol-length><multicodec-of-available-protocol>\n
<varint-protocol-length><multicodec-of-available-protocol>\n
# ...more protocols
\n

For example

# send request
> /multistream/1.0.0
> ls

# get response
< /multistream/1.0.0
< /ipfs/kad/0.2.3
< /ipfs/kad/1.0.0
< /ipfs/bitswap/0.4.3
< /ipfs/bitswap/1.0.0

# send selection, upgrade connection, and start protocol traffic
> /ipfs/kad/0.2.3
> <kad-dht-request-0>
> <kad-dht-request-1>
> ...

# receive selection, and upgraded protocol traffic.
< /ipfs/kad/0.2.3
< <kad-dht-response-0>
< <kad-dht-response-1>
< ...

Note that ls support is OPTIONAL, i.e. implementations MAY support sending and/or receiving ls. This in turn implies that implementations MUST NOT depend on a remote node supporting ls.

Implementation Recommendations

Protocol Selection

When selecting a protocol, the initiator SHOULD send the multistream protocol AND the desired protocol in the same packet. Similarly, the responder SHOULD send its first response in the same packet as the multistream protocol.

For example:

# successful negotiation of the kad protocol
> /multistream/1.0.0\n/ipfs/kad/1.0.0\n
< /multistream/1.0.0\n/ipfs/kad/1.0.0\n

# successful negotiation of the kad protocol, with missing version support
> /multistream/1.0.0\n/ipfs/kad/1.0.0\n
< /multistream/1.0.0\nna\n
> /ipfs/kad/0.2.3\n
< /ipfs/kad/0.2.3\n

Listing

When requesting a list of protocols, if the multistream protocol has not yet been negotiated, the initiator SHOULD send the ls request in the same packet as the multistream protocol. Similarly, the responder SHOULD send its list of supported protocols in the same packet as the multistream protocol.

For example:

# request the list of protocols
> /multistream/1.0.0\nls\n

# response with supported protocols
< /multistream/1.0.0\n/ipfs/kad/0.2.3\n/ipfs/kad/1.0.0\n/ipfs/bitswap/0.4.3\n/ipfs/bitswap/1.0.0\n

ls example in detail

> <varint-ls-length>ls\n
< <varint-ls-response-length><varint-protocol-length><protocol>\n
< <varint-protocol-length><protocol>\n
# ...more protocols
< \n

varint-ls-response-length is the total size of the response message including all the protocols and the \n at the end.

Each protocol is an embedded multistream message (it has a length prefix and a new line suffix). So, unless there are 0 protocols the response will end with \n\n.

Implementations

js-multistream-select - JavaScript Implementation
go-multistream - Go Implementation
mss-nc - multistream-select netcat written in Go

Maintainers

Captain: @diasdavid.

Contribute

Contributions welcome. Please check out the issues.

Check out our contributing document for more information on how we work, and about contributing in general. Please be aware that all interactions related to multiformats are subject to the IPFS Code of Conduct.

Small note: If editing the README, please conform to the standard-readme specification.

multiformats/multistream-select