The goal of the SRIOV Network device plugin is to manage the lifecycle of SRIOV VFs on a Kubernetes node. The device plugin discovers, advertises and allocates SRIOV VFs to requesting pods and the Multus CNI retrieves the allocated Device information from the Kubernetes APIServer and delegates SRIOV CNI to plumb the VF to the pods network namespace.
-
Device Plugin/Device Manager
-
Discovery of SRIOV NIC devices in a node
-
Advertisement of number of SRIOV VFs configured on a node
-
Allocation of SRIOV VF to a pod
-
-
Multus CNI
-
Retrieve allocatd VF information for a pod from Kubernetes APIServer
-
Passing retrieved information to SRIOV CNI
-
-
SRIOV CNI
-
On Cmd Add, using information passed from Multus CNI plumb allocated SRIOV VF to the pods network namespace
-
On Cmd Del, release VF from the pods network namespace
-
This implementation follows the directions of this proposal document.
There are list of items should be required before installing the SRIOV Network device plugin
- SRIOV NICs - (Tested with Intel NIC, should support all the NICs)
- Intel SRIOV CNI - dev/k8s-deviceid-model branch
- Kubernetes version - 1.11 or higher(with patch applied)
- Meta plugin - Multus dev/k8s-deviceid-model branch
Make sure to implement the steps described in Quick Start for Kubernetes cluster to support multi network. SRIOV network device plugin is a collective plugin model to work with device manager, Meta-plugin and SRIOV CNI plugin.
The following NICs were tested with this implementation. However, other SRIOV capable NICs should work as well.
- Intel® Ethernet Controller X710 Series 4x10G - PF driver : v2.4.6 - VF driver: v3.5.6
please refer to Intel download center for installing latest Intel Ethernet Controller-X710-Series drivers
- Intel® 82599ES 10 Gigabit Ethernet Controller
- PF driver : v4.4.0-k
- VF driver: v3.2.2-k
please refer to Intel download center for installing latest Intel-® 82599ES 10 Gigabit Ethernet drivers
This section explains how to set up SRIOV Network device plugin in Kubernetes. Required YAML files can be found in deployments/ directory.
Kubernetes out of the box only allows to have one network interface per pod. In order to add multiple interfaces in a Pod we need to configure Kubernetes with a CNI meta plugin that enables invoking multiple CNI plugins to add additional interfaces. Multus is only meta plugin that supports this mechanism. Multus uses Kubernetes Custom Resource Definition or CRDs to define network objects. For more information see Multus documentation.
- Compile Meta Plugin CNI (Multus dev/k8s-deviceid-model branch):
$ git clone https://github.com/intel/multus-cni.git
$ cd multus-cni
$ git fetch
$ git checkout dev/k8s-deviceid-model
$ ./build
$ cp bin/multus /opt/cni/bin
- Configure Kubernetes network CRD with Multus
Compile SRIOV-CNI (dev/k8s-deviceid-model branch):
$ git clone https://github.com/intel/sriov-cni.git
$ cd sriov-cni
$ git fetch
$ git checkout dev/k8s-deviceid-model
$ ./build
$ cp bin/sriov /opt/cni/bin
- Clone the sriov-network-device-plugin repository and checkout dev/k8s-deviceid-model branch
$ git clone https://github.com/intel/sriov-network-device-plugin.git
$ cd sriov-network-device-plugin
$ git fetch
$ git checkout dev/k8s-deviceid-model
- Run the build script, this will build the SRIOV Network Device Plugin binary
$ ./build.sh
- Copy the multus Configuration file from the Deployments folder to the CNI Configuration diectory
$ cp deployments/cni-conf.json /etc/cni/net.d/
- Create the SRIOV Network CRD
$ kubectl create -f deployments/crdnetwork.yaml
$ kubectl create -f deployments/sriov-crd.yaml
- Run build docker script to create SRIOV Network Device Plugin Docker image
$ cd deployments/
$ ./build_docker.sh
- Create SRIOV Network Device Plugin Pod
$ kubectl create -f pod-sriovdp.yaml
Note: This is for demo purposes, the SRIOV Device Plugin binary must be executed from within the pod
- Get a bash terminal to the SRIOV Network Device Plugin Pod
$ kubectl exec -it sriov-device-plugin bash
- Execute the SRIOV Network Device Plugin binary from within the Pod
$ ./usr/bin/sriovdp --logtostderr -v 10
sriov-device-plugin.go:380] SRIOV Network Device Plugin started...
sriov-device-plugin.go:190] Discovering SRIOV network device[s]
sriov-device-plugin.go:92] Checking for file /sys/class/net/enp0s31f6/device/sriov_numvfs
sriov-device-plugin.go:92] Checking for file /sys/class/net/enp14s0/device/sriov_numvfs
sriov-device-plugin.go:92] Checking for file /sys/class/net/enp5s0f0/device/sriov_numvfs
sriov-device-plugin.go:92] Checking for file /sys/class/net/enp5s0f1/device/sriov_numvfs
sriov-device-plugin.go:92] Checking for file /sys/class/net/enp5s0f2/device/sriov_numvfs
sriov-device-plugin.go:92] Checking for file /sys/class/net/enp5s0f3/device/sriov_numvfs
sriov-device-plugin.go:92] Checking for file /sys/class/net/enp6s2/device/sriov_numvfs
sriov-device-plugin.go:92] Checking for file /sys/class/net/enp6s2f1/device/sriov_numvfs
sriov-device-plugin.go:121] Sriov Capable Path: /sys/class/net/enp5s0f0/device/sriov_totalvfs
sriov-device-plugin.go:133] Total number of VFs for device enp5s0f0 is 32
sriov-device-plugin.go:135] SRIOV capable device discovered: enp5s0f0
sriov-device-plugin.go:148] Number of Configured VFs for device enp5s0f0 is 2
sriov-device-plugin.go:171] PCI Address for device enp5s0f0, VF 0 is 0000:06:02.0
sriov-device-plugin.go:171] PCI Address for device enp5s0f0, VF 1 is 0000:06:02.1
sriov-device-plugin.go:121] Sriov Capable Path: /sys/class/net/enp5s0f1/device/sriov_totalvfs
sriov-device-plugin.go:133] Total number of VFs for device enp5s0f1 is 32
sriov-device-plugin.go:135] SRIOV capable device discovered: enp5s0f1
sriov-device-plugin.go:148] Number of Configured VFs for device enp5s0f1 is 0
sriov-device-plugin.go:121] Sriov Capable Path: /sys/class/net/enp5s0f2/device/sriov_totalvfs
sriov-device-plugin.go:133] Total number of VFs for device enp5s0f2 is 32
sriov-device-plugin.go:135] SRIOV capable device discovered: enp5s0f2
sriov-device-plugin.go:148] Number of Configured VFs for device enp5s0f2 is 0
sriov-device-plugin.go:121] Sriov Capable Path: /sys/class/net/enp5s0f3/device/sriov_totalvfs
sriov-device-plugin.go:133] Total number of VFs for device enp5s0f3 is 32
sriov-device-plugin.go:135] SRIOV capable device discovered: enp5s0f3
sriov-device-plugin.go:148] Number of Configured VFs for device enp5s0f3 is 0
sriov-device-plugin.go:195] Starting SRIOV Network Device Plugin server at: /var/lib/kubelet/device-plugins/sriovNet.sock
sriov-device-plugin.go:220] SRIOV Network Device Plugin server started serving
sriov-device-plugin.go:402] SRIOV Network Device Plugin registered with the Kubelet
sriov-device-plugin.go:291] ListAndWatch: send devices &ListAndWatchResponse{Devices:[&Device{ID:0000:06:02.0,Health:Healthy,} &Device{ID:0000:06:02.1,Health:Healthy,}],}
Leave the sriov device plugin running and open a new terminal session for following steps.
$ kubectl create -f pod-tc1.yaml
pod "testpod1" created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
sriov-device-plugin 1/1 Running 0 7h
testpod1 1/1 Running 0 3s
$ kubectl exec -it testpod1 -- ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth0@if17511: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
link/ether 0a:58:c0:a8:4a:b1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.74.177/24 scope global eth0
valid_lft forever preferred_lft forever
17508: net0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
link/ether ce:d8:06:08:e6:3f brd ff:ff:ff:ff:ff:ff
inet 10.56.217.179/24 scope global net0
valid_lft forever preferred_lft forever
$ kubectl exec -it testpod1 -- route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.74.1 0.0.0.0 UG 0 0 0 eth0
10.56.217.0 0.0.0.0 255.255.255.0 U 0 0 0 net0
192.168.0.0 192.168.74.1 255.255.0.0 UG 0 0 0 eth0
192.168.74.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
We welcome your feedback and contributions to this project. Please see the CONTRIBUTING.md for contribution guidelines.
Copyright 2018 © Intel Corporation.