This project implements an NBD server (almost
baseline)
to expose Sia cloud storage in the form of a
Linux block device /dev/nbd0 in combination with a local cache.
The Linux kernel module nbd provides a block device where every read and write
operation is turned into a request to a (potentially remote) server. This
project provides such a server - intended to be run locally, on the same
machine - and serves the requests by reading and writing to the Sia network. In
addition, a local cache is used to limit interaction with the Sia network and
increase performance. "Screenshot":
$ df -h /dev/nbd0
Filesystem Size Used Avail Use% Mounted on
/dev/nbd0 1.0T 34G 990G 4% /mnt
You will need a running Sia node that has formed storage contracts and is ready to store data. Then:
$ git clone https://github.com/javgh/sia-nbdserver.git
$ cd sia-nbdserver
$ go install # will - by default - install to ~/go/bin/
$ sia-nbdserver
As root:
# modprobe nbd
# nbd-client -b 4096 -t 3600 -u /run/user/1000/sia-nbdserver /dev/nbd0
# mkfs.xfs /dev/nbd0
# mount -o sync /dev/nbd0 /mnt
$ sia-nbdserver -h
NBD server backed by Sia storage + local cache.
Usage:
sia-nbdserver [flags]
Flags:
-H, --hard int hard limit for number of 64 MiB pages in the cache (default 128)
-h, --help help for sia-nbdserver
-i, --idle int seconds to wait before a cache page is marked idle and upload begins (default 120)
--sia-daemon string host and port of Sia daemon (default "localhost:9980")
--sia-password-file string path to Sia API password file (default "/home/jan/.sia/apipassword")
-s, --size uint size of block device; should ideally be a multiple of 67108864 (2 ^ 26) (default 1099511627776)
-S, --soft int soft limit for number of 64 MiB pages in the cache (default 96)
-u, --unix string unix domain socket (default "/run/user/1000/sia-nbdserver")
By default sia-nbdserver will export a block device with a size of 1 TiB. This
can be changed with the --size flag. The software divides this range up into a
number of 64 MiB pages. As Sia continues to push the minimum file size lower, it
will be possible to make the pages smaller, but for now this value is hardcoded.
Each page will be stored on Sia as a separate file under the directory nbd.
A page is only created once it has been accessed for the first time. The
directory ~/.local/share/sia-nbdserver/ serves as a local cache, where
recently accessed pages are kept to speed up read and write operations. The
maximum number of pages in this cache can be set with --soft and --hard.
The software will actively try to reduce the size of the cache once the soft
limit has been reached, but will still allow the cache to grow if necessary.
Once the hard limit is reached, it will block new operations until necessary
uploads have completed. Some time after the soft limit is exceeded, a "write
throttle" kicks in, which will artificially slow down write operations to allow
Sia to catch up. This is done in an attempt to avoid outright blocking write
operations, which is prone to trigger timeouts in the NBD client.
There is no specific lower bound for the cache size, but it should probably not be smaller than 16 pages and the hard limit should be an additional 8 pages for the write throttle mechanic to work correctly. For a short test run it can be helpful to reduce the time before uploads are initiated and use a fairly small cache:
$ sia-nbdserver --idle 30 -S 16 -H 32
Before shutting down the server, it is important to first unmount any filesystem
that might use /dev/nbd0 and then tell nbd-client to disconnect:
# umount /mnt
# nbd-client -d /dev/nbd0
The server can then be shutdown with ^C or using a kill command. This will
trigger a "fast" shutdown, where any unsynced data will simply remain in the
cache directory to be uploaded when the server is started again in the future.
To instead perform a "thorough" shutdown, it is possible to send SIGUSR1 to
the server (use kill -USR1 <pid of server>). This will cause the server to
wait for all uploads to finish before shutting down.
In theory any filesystem can be used on top of the block device. I first tried
ext4, but realized that mkfs.ext4 will immediately cause a huge amount of
upload activity by writing to a lot of different pages. I have found xfs to be
a good choice, as mkfs.xfs only touches a few pages.
Linux will provide caching for the NBD block device, which is great for
performance. However, it can cause a problem in this setting where nbd-client
and sia-nbdserver are on the same machine. If the write cache fills up too
much, it can use up all available memory, which in turn prevents siad from
making any progress. Now the write cache can not get smaller and the system is
stuck. To prevent this I pass -o sync to xfs to force it to directly flush
all writes to the block device. This unfortunately impacts performance
negatively, but seems to avoid the low memory situation. A better approach is to
use cgroups to limit memory for whatever application is using the block device,
which will also limit the size of the filesystem cache. Yet another approach would
be to have nbd-client and sia-nbdserver on two separate machines. This would
require to first modify sia-nbdserver to support exporting over the network.
In my testing (in December 2019) previous versions of Sia would sometimes make no
progress on uploads or downloads for several tens of minutes. This will then usually
cause nbd-client to timeout. As a workaround I set a very high timeout value like
1 hour (3600 seconds) with nbd-client -t 3600. This might not be necessary
with a recent version of Sia, but I would still recommend to set a timeout of
at least a few minutes.