Management gateway daemon to setup access to Ceph storage over NVMeoF
This daemon runs as root. It provides the ability to export existing RBD images as NVMeoF namespaces. Creation of RBD images is not within the scope of this daemon.
-
The daemon is a gRPC server, so the host running the server will need to install gRPC packages:
$ make setup -
Modify the config file (default ceph-nvmeof.conf) to reflect the IP/ Port where the server can be reached:
addr = <IP address at which the client can reach the gateway> port = <port at which the client can reach the gateway> -
To enable mTLS using self signed certificates, edit the config file to set:
enable_auth = True # Setting this to False will open an insecure port -
Compile protobuf files for gRPC:
$ make grpc -
SPDK is included in this repository as a submodule. Edit the config file to set:
spdk_path = <complete path to SPDK parent directory> spdk_tgt = <relative path to SPDK target executable> -
Setup SPDK
Navigate to the spdk folder & install dependencies:
$ ./scripts/pkgdep.shInitialize configuration:
$ apt install librbd-dev $ ./configure --with-rbdBuild the SPDK app:
$ makeSPDK requires hugepages to be set up:
$ sh -c 'echo 4096 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages' -
Start the gateway server daemon:
$ python3 -m control [-c CONFIG]
The CLI tool can be used to initiate a connection to the gateway and run commands to configure the NVMe targets.
Run the tool with the -h flag to see a list of available commands:
$ python3 -m control.cli -h
usage: python3 -m control.cli [-h] [-c CONFIG]
{create_bdev,delete_bdev,create_subsystem,delete_subsystem,add_namespace,remove_namespace,add_host,remove_host,create_listener,delete_listener,get_subsystems} ...
CLI to manage NVMe gateways
positional arguments:
{create_bdev,delete_bdev,create_subsystem,delete_subsystem,add_namespace,remove_namespace,add_host,remove_host,create_listener,delete_listener,get_subsystems}
optional arguments:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
Path to config file
Example:
$ python3 -m control.cli create_bdev -h
usage: python3 -m control.cli create_bdev [-h] -i IMAGE -p POOL [-b BDEV_NAME] [-s BLOCK_SIZE]
optional arguments:
-h, --help show this help message and exit
-i IMAGE, --image IMAGE
RBD image name
-p POOL, --pool POOL Ceph pool name
-b BDEV_NAME, --bdev BDEV_NAME
Bdev name
-s BLOCK_SIZE, --block_size BLOCK_SIZE
Block size
For testing purposes, self signed certificates and keys can be generated locally using OpenSSL.
For the server, generate credentials for server name 'my.server' in files called server.key and server.crt:
$ openssl req -x509 -newkey rsa:4096 -nodes -keyout server.key -out server.crt -days 3650 -subj '/CN=my.server'
For client:
$ openssl req -x509 -newkey rsa:4096 -nodes -keyout client.key -out client.crt -days 3650 -subj '/CN=client1'
Indicate the location of the keys and certificates in the config file:
[mtls]
server_key = ./server.key
client_key = ./client.key
server_cert = ./server.crt
client_cert = ./client.crt
-
Start the gateway server:
$ python3 -m control INFO:root:SPDK PATH: /path/to/spdk INFO:root:Starting /path/to/spdk/tgt/nvmf_tgt all -u INFO:root:Attempting to initialize SPDK: server_addr: /var/tmp/spdk.sock, port: 5260, conn_retries: 3, timeout: 60.0 INFO: Setting log level to ERROR INFO:JSONRPCClient(/var/tmp/spdk.sock):Setting log level to ERROR -
Run the CLI (ensure a ceph pool 'rbd' with an rbdimage 'mytestdevimage' is created prior to this step):
$ python3 -m control.cli create_bdev -i mytestdevimage -p rbd -b Ceph0 INFO:root:Created bdev Ceph0: True $ python3 -m control.cli create_subsystem -n nqn.2016-06.io.spdk:cnode1 -s SPDK00000000000001 INFO:root:Created subsystem nqn.2016-06.io.spdk:cnode1: True $ python3 -m control.cli add_namespace -n nqn.2016-06.io.spdk:cnode1 -b Ceph0 INFO:root:Added namespace 1 to nqn.2016-06.io.spdk:cnode1: True $ python3 -m control.cli add_host -n nqn.2016-06.io.spdk:cnode1 -t '*' INFO:root:Allowed open host access to nqn.2016-06.io.spdk:cnode1: True $ python3 -m control.cli create_listener -n nqn.2016-06.io.spdk:cnode1 -s 5001 INFO:root:Created nqn.2016-06.io.spdk:cnode1 listener: True -
On the storage client system (ubuntu-21.04):
-
Install requisite packages
$ apt install nvme-cli $ modprobe nvme-fabrics -
Run nvme command to discover available subsystems
$ nvme discover -t tcp -a 192.168.50.4 -s 5001 Discovery Log Number of Records 1, Generation counter 6 =====Discovery Log Entry 0====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 0 trsvcid: 5001 subnqn: nqn.2016-06.io.spdk:cnode1 traddr: 192.168.50.4 sectype: none -
Connect to desired subsystem
$ nvme connect -t tcp --traddr 192.168.50.4 -s 5001 -n nqn.2016-06.io.spdk:cnode1 -
List targets that are available
$ nvme list Node SN Model Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 SPDK00000000000001 SPDK bdev Controller 1 6.44 GB / 6.44 GB 4 KiB + 0 B 21.04 -
Create a filesystem on the desired target
$ mkfs /dev/nvme0n1 mke2fs 1.45.7 (28-Jan-2021) Creating filesystem with 1572864 4k blocks and 393216 inodes Filesystem UUID: 1308f6ff-621b-4d17-b127-65eded31abe2 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736 Allocating group tables: done Writing inode tables: done Writing superblocks and filesystem accounting information: done -
Mount and use the storage volume
$ mount /dev/nvme0n1 /mnt $ ls /mnt lost+found $ echo "NVMe volume" > /mnt/test.txt $ ls /mnt lost+found test.txt
-