Audio plugin for NoVNC

Description

NoVNC drop-in plugin for out-of-band audio playback.

Tested with NoVNC 1.4.0

Features

Low-latency real-time audio playback using WebSocket and Media Source API.
Works with both PipeWire and PulseAudio.
No WebRTC shenanigans necessary.

Installation

NoVNC

Copy audio-plugin.js to the NoVNC client directory where vnc.html is located.

Add the following line to the end of the head section of vnc.html.

<script type="module" crossorigin="anonymous" src="audio-plugin.js"></script>

vnc_lite.html is not supported yet.

Host

VNC protocol in general only handles graphics and not audio. Audio must be transmitted to out-of-band using a separate connection.

PulseAudio/PipeWire

A few modifications to the configuration of PulseAudio/PipeWire is necessary to allow capturing audio and feeding it to the Audio Proxy.

If using PulseAudio:

Enable module-simple-protocol-tcp module on PulseAudio:

# echo "load-module module-simple-protocol-tcp listen=127.0.0.1 format=s16le channels=2 rate=48000 record=true playback=false" > /etc/pulse/default.pa.d/simple-protocol.pa
$ pulseaudio -k
$ pulseaudio --start

If using PipeWire:

Enable libpipewire-module-protocol-simple module on PipeWire:

# cat > /etc/pipewire/pipewire.conf.d/simple-protocol.conf << EOF
context.modules = [
    {
        name = libpipewire-module-protocol-simple
        args = {
            capture = true
            playback = false

            stream.capture.sink = true

            audio.rate = 48000
            audio.format = S16LE
            audio.channels = 2

            server.address = [
                "tcp:127.0.0.1:4711"
            ]
        }
    }
]
EOF
$ systemctl --user restart pipewire

Audio Proxy

Raw audio output from PulseAudio must be encoded to a codec supported by Media Source (e.g. WebM/Opus) before it can be played back on the browser.

audio-proxy.sh shell script can encode raw audio to the required format using GStreamer.

$ audio-proxy.sh -l 5711
Raw source port: 4711
Raw source format: s16le
Raw source sample rate: 48000
Raw source channels: 2
Server listening on 127.0.0.1:5711

audio-proxy.sh requires socat and gstreamer-tools (along with Base, Good and Bad plugins) to be installed on your system.

Installing dependencies on Ubuntu/Debian:
$ apt install socat gstreamer1.0-tools gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad
Installing dependencies on Alpine Linux:
$ apk add socat gstreamer-tools gst-plugins-base gst-plugins-good gst-plugins-bad

Websockify

Just like VNC, Audio Proxy only accepts raw TCP connections. Websockify will take care of proxying WebSocket traffic to and from audio-proxy.sh.

It's possible to proxy both VNC and Audio and serve the NoVNC client files using a single Websockify instance.

Assuming

there's a VNC server running on port 5900
Audio Proxy running on port 5711
noVNC client files are located at /var/www/noVNC

Websockify can be configured like this:

# cat > /etc/websockify/token.cfg << EOF
	vnc: 127.0.0.1:5900
	audio: 127.0.0.1:5711
EOF
$ websockify \
    --web=/var/www/noVNC \
    --token-plugin=TokenFile \
    --token-source=/etc/websockify/token.cfg \
    8080

NoVNC client will now be accessible at http://localhost:8080
VNC WebSocket path will be websockify?token=vnc
Audio WebSocket path will be websockify?token=audio

Known Issues

Audio may not start playing if autoconnect is enabled.

TODO

Add support for Windows
Add builtin WebSocket server to Audio Proxy

Disclaimer

This is an experimental and hacky piece of software, so expect a few bugs here and there.

Make sure to use TLS with proper HTTP authentication if this software is being exposed to the internet.

me-asri/noVNC-audio-plugin