cl-multiplex

The active repository is at https://codeberg.org/glv/cl-multiplex

cl-multiplex is a library facilitating the multiplexing of the data of several channels into one octet stream, and the demultiplexing of the data read from the octet stream.

Installation

The only dependency is the cl-octet-streams library.

If you are using quicklisp, you can put cl-multiplex in the local-projects directory and use:

(ql:quickload "cl-multiplex")

API

The functions are in the multiplex package (you can also use the nickname cl-multiplex).

Multiplex streams

(make-multiplex-stream stream channels) => multiplex-stream

Return a multiplex stream. The data written to several channels will be multiplexed before being written to stream. The data read from stream will be demultiplexed and made available in the channels. The number of channels to use is indicated by channels.

(close-multiplex-stream multiplex-stream) => nil

Close all the channels of multiplex-stream.

(with-multiplex-stream (var stream channels) &body body)

Within body, var is bound to a multiplex stream defined by stream and channels. The result of the last form of body is returned.

(multiplex multiplex-stream &optional max-frame-size) => t

Multiplex the data that was written to the channels of multiplex-stream and write it to the underlying stream. The mutiplexed frames written to the underlying stream contain at most max-frame-size bytes of user data (1024 by default).

(finish-multiplex-output multiplex-stream) => nil

Multiplex the data that was written to the channels of multiplex-stream and return when everything has been written successfully to the underlying stream.

(clear-multiplex-input multiplex-stream) => nil

Clear the input of all the channels of multiplex-tream.

(demultiplex multiplex-stream &optional max-frames max-frame-size) => boolean

Read data from the underlying stream of multiplex-stream and demultiplex the frames. Return t if at least one frame was demultiplexed successfully, and nil otherwise. If max-frames is a positive integer, at most max-frames frames will be demultiplexed. If max-frame-size is a positive integer and a frame bigger than max-frame-size is detected, an error is signaled.

(write-data data multiplex-stream channel &key (start 0) end) => data

Like write-sequence for mutiplex streams. Write the bytes of data between start end end to a specific channel of multiplex-stream.

(read-data data multiplex-stream channel &key (start 0) end) => position

Like read-sequence for multiplex streams. Fill data between start and end with bytes read from a specific channel of multiplex-stream.

(clear-channel-input multiplex-stream channel) => nil

Clear the input of a specific channel of multiplex-tream.

(get-channel-stream multiplex-stream channel) => stream

Return a stream that can be used to read/write data from/to a specific channel of multiplex-stream.

Low-level functions

In case you need to read or write frames to a stream one by one, you can use the read-frame and write-frame functions.

(write-frame stream channel data &key (start 0) (end (length data))) => data

Make a frame for a channel with the bytes of data between start and end and write it to a stream.

(read-frame stream &key incomplete-frame max-frame-size buffer) => frame, boolean

Try to read a frame from a stream. The first returned value is a list of 4 elements representing a frame (channel, total length of data, data, length of data received so far). The second returned value is t if the frame is complete and nil otherwise. If incomplete-frame is specified (it must be an incomplete frame returned by a previous call to read-frame), the function tries to complete it with new data from the stream. If max-frame-size is a positive integer and a frame bigger than max-frame-size is detected, an error is signaled. If buffer is specified (it must be an array of (unsigned-byte 8)), the function tries to use it instead of allocating a new work area.

Conditions

A bad-channel-number error is signaled when trying to read or write data using an incorrect channel number.

A frame-to-big error is signaled when trying to read a frame containing more data than the specified max-frame-size. There are two restarts than can be invoked for this error:

process-frame which reads the frame as if there was no max-frame-size
drop-frame which reads the frame from the stream and discards it

Examples

Multiplex the contents of data-file1 and data-file2 to multiplexed-file:

(asdf:load-system "cl-multiplex")
(use-package :multiplex)

(with-open-file (d1 "data-file1" :element-type '(unsigned-byte 8))
  (with-open-file (d2 "data-file2" :element-type '(unsigned-byte 8))
    (with-open-file (f "multiplexed-file"
                       :direction :output
                       :if-exists :supersede
                       :element-type '(unsigned-byte 8))
      (with-multiplex-stream (mux f 2)
        (let ((buffer (make-array 2048 :element-type '(unsigned-byte 8)))
              (data-p t))
          (loop while data-p do
            (let ((n (read-sequence buffer d1)))
              (write-data buffer mux 0 :end n)
              (setf data-p (plusp n)))
            (let ((n (read-sequence buffer d2)))
              (write-data buffer mux 1 :end n)
              (setf data-p (or data-p (plusp n))))
            (multiplex mux)))))))

Demultiplex multiplexed-file to demux-file1 and demux-file2:

(with-open-file (f "multiplexed-file" :element-type '(unsigned-byte 8))
  (with-open-file (d1 "demux-file1"
                      :direction :output
                      :if-exists :supersede
                      :element-type '(unsigned-byte 8))
    (with-open-file (d2 "demux-file2"
                        :direction :output
                        :if-exists :supersede
                        :element-type '(unsigned-byte 8))
      (with-multiplex-stream (mux f 2)
        (let ((buffer (make-array 2048 :element-type '(unsigned-byte 8)))
              (data-p t))
          (loop while data-p do
            (setf data-p (handler-case
                             (demultiplex mux 10)
                           (end-of-file ())))
            (loop for n = (read-data buffer mux 0)
                  until (zerop n)
                  do (write-sequence buffer d1 :end n))
            (loop for n = (read-data buffer mux 1)
                  until (zerop n)
                  do (write-sequence buffer d2 :end n))))))))

data-file1 and demux-file1 should be identical, and data-file2 and demux-file2 should be identical.

Protocol

The multiplexed data is organized in frames having the following format:

+---------+------+------+
| channel | size | data |
+---------+------+------+

channel and size are integers encoded using the base 128 varint encoding. size indicates the number of bytes of data.

Tests

The tests require the fiveam library.

(asdf:test-system "cl-multiplex")

glv2/cl-multiplex