Userspace block driver(ublk)
1 Introduction
This is the userspace daemon part(ublksrv) of the ublk framework, the other
part is ublk blk-mq block driver[1] which supports multiple queue.
The two parts communicate by io_uring's IORING_OP_URING_CMD with one
per-queue shared cmd buffer for storing io command, and the buffer is
read only for ublksrv, each io command can be indexed by io request tag
directly, and the command is written by ublk driver, and read by ublksrv
after getting notification from ublk driver.
For example, when one READ io request is submitted to ublk block driver, ublk
driver stores the io command into cmd buffer first, then completes one
IORING_OP_URING_CMD for notifying ublksrv, and the URING_CMD is issued to
ublk driver beforehand by ublksrv for getting notification of any new io
request, and each URING_CMD is associated with one io request by tag,
so depth for URING_CMD is same with queue depth of ublk block device.
After ublksrv gets the io command, it translates and handles the ublk io
request, such as, for the ublk-loop target, ublksrv translates the request
into same request on another file or disk, like the kernel loop block
driver. In ublksrv's implementation, the io is still handled by io_uring,
and share same ring with IORING_OP_URING_CMD command. When the target io
request is done, the same IORING_OP_URING_CMD is issued to ublk driver for
both committing io request result and getting future notification of new
io request.
So far, the ublk driver needs to copy io request pages into userspace buffer
(pages) first for write before notifying the request to ublksrv, and copy
userspace buffer(pages) to the io request pages after ublksrv handles
READ. Also looks linux-mm can't support zero copy for this case yet[2].
More ublk targets will be added with this framework in future even though only
ublk-loop and ublk-null are implemented now.
libublksrv is also generated, and it helps to integrate ublk into existed
project. One example of demo_null is provided for how to make a ublk
device over libublksrv.
2 Quick start
2.1 how to build ublksrv:
autoreconf -i
./configure
make
'./configure' requires liburing 2.2 package installed, if liburing 2.2 isn't
available in your distribution, please configure via the following command:
PKG_CONFIG_PATH=${LIBURING_DIR} \
./configure \
CFLAGS="-I${LIBURING_DIR}/src/include" \
CXXFLAGS="-I${LIBURING_DIR}/src/include" \
LDFLAGS="-L${LIBURING_DIR}/src"
and LIBURING_DIR points to directory of liburing source code, and liburing
needs to be built before running above commands. Also IORING_SETUP_SQE128
has to be supported in the liburing source.
2.2 help
- ublk help
2.3 add one ublk-null disk
- ublk add -t null
2.4 add one ublk-loop disk
- ublk add -t loop -f /dev/vdb
or
- ublk add -t loop -f 1.img
2.5 remove one ublk disk
- ublk del -n 0 #remove /dev/ublkb0
- ublk del -a #remove all ublk devices
2.6 list ublk devices
- ublk list
- ublk list -v #with all device info dumped
3 build
3.1 run 'make' directly
3.2 dependency
1) liburing with IORING_SETUP_SQE128 support
2) linux kernel v5.19(IORING_SETUP_SQE128 support)
3) linux kernel v5.20 with ublk kernel driver(drivers/block/ublk_drv.c)
merged
4 test
4.1 run all built tests
make test T=all
4.2 run test group
make test T=null
make test T=loop
make test T=generic
4.3 run single test
make test T=generic/001
make test T=null/001
make test T=loop/001
...
4.4 run specified tests or test groups
make test T=generic:loop/001:null
5 License
nlohmann(include/nlohmann/json.hpp) is from [3], which is covered by MIT
license.
The library functions (all code in lib/ directory and include/ublksrv.h)
are covered by dual licensed LGPL and MIT, see COPYING.LGPL and LICENSE.
All other source code are covered by dual licensed GPL and MIT, see
COPYING and LICENSE.
[1] https://lore.kernel.org/linux-block/20220509092312.254354-1-ming.lei@redhat.com/
[2] https://lore.kernel.org/all/20220318095531.15479-1-xiaoguang.wang@linux.alibaba.com/
[3] https://github.com/nlohmann/json