THIS REPOSITORY IS ARCHIVED. Please see the active repository @ https://github.com/openshift/sriov-network-device-plugin
The SRIOV network device plugin is Kubernetes device plugin for discovering and advertising SRIOV network virtual functions (VFs) in a Kubernetes host. To deploy workloads with SRIOV VF this plugin needs to work together with the following two CNI plugins:
-
Multus CNI
-
Retrieves allocated network device information of a Pod
-
Passes allocated SRIOV VF information to SRIOV CNI
-
-
SRIOV CNI
-
During Pod creation, plumbs allocated SRIOV VF to a Pods network namespace using VF information given by Multus
-
On Pod deletion, reset and release the VF from the Pod
-
This implementation follows the design discuessed in this proposal document.
Please follow the Multus Quick Start for multi network interface support in Kubernetes.
The following NICs were tested with this implementation. However, other SRIOV capable NICs should work as well.
- Intel® Ethernet Controller X710 Series 4x10G - PF driver : v2.4.6 - VF driver: v3.5.6
please refer to Intel download center for installing latest Intel Ethernet Controller-X710-Series drivers
- Intel® 82599ES 10 Gigabit Ethernet Controller
- PF driver : v4.4.0-k
- VF driver: v3.2.2-k
please refer to Intel download center for installing latest Intel-® 82599ES 10 Gigabit Ethernet drivers
This section explains an exmaple deployment of SRIOV Network device plugin in Kubernetes. Required YAML files can be found in deployments/ directory.
Multus uses Custom Resource Definitions(CRDs) for defining additional network attachements. These network attachment CRDs follow the standards defined by K8s Network Plumbing Working Group(NPWG). Please refer to Multus documentation for more information.
- Compile Multus executable:
$ git clone https://github.com/intel/multus-cni.git
$ cd multus-cni
$ ./build
$ cp bin/multus /opt/cni/bin
- Copy the multus Configuration file from the Deployments folder to the CNI Configuration diectory
$ cp deployments/cni-conf.json /etc/cni/net.d/
- Configure Kubernetes network CRD with Multus
$ kubectl create -f deployments/crdnetwork.yaml
- Compile SRIOV-CNI (dev/k8s-deviceid-model branch):
$ git clone https://github.com/intel/sriov-cni.git
$ cd sriov-cni
$ git fetch
$ git checkout dev/k8s-deviceid-model
$ ./build
$ cp bin/sriov /opt/cni/bin
- Create the SRIOV Network CRD
$ kubectl create -f deployments/sriov-crd.yaml
- Clone the sriov-network-device-plugin
$ git clone https://github.com/intel/sriov-network-device-plugin.git
$ cd sriov-network-device-plugin
- Build executable binary using
make
$ make
On successful build the
sriovdp
executable can be found in./build
directory. It is recommended to run the plugin in a container or K8s Pod. The follow on steps cover how to build and run the Docker image of the plugin.
- Build docker image
$ make image
- Create SRIOV network device plugin Pod and the SRIOV network device plugin binary will be exectued from within the Pod
$ kubectl create -f pod-sriovdp.yaml
- Check for the logs of the SRIOV network device plugin binary from within the pod
$ kubectl logs sriov-device-plugin
I0323 05:49:12.547174 324440 sriov-device-plugin.go:469] Starting SRIOV Network Device Plugin...
I0323 05:49:12.550968 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/cni0
I0323 05:49:12.551082 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/cni0/device/sriov_numvfs
I0323 05:49:12.551159 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/docker0
I0323 05:49:12.551224 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/docker0/device/sriov_numvfs
I0323 05:49:12.551328 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/eno1
I0323 05:49:12.551445 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/eno1/device/sriov_numvfs
I0323 05:49:12.551846 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/eno2
I0323 05:49:12.551922 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/eno2/device/sriov_numvfs
I0323 05:49:12.551999 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/enp135s0f0
I0323 05:49:12.552074 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/enp135s0f0/device/sriov_numvfs
I0323 05:49:12.552152 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/enp135s0f1
I0323 05:49:12.552225 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/enp135s0f1/device/sriov_numvfs
I0323 05:49:12.552306 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/enp135s2
I0323 05:49:12.552382 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/enp135s2/device/sriov_numvfs
I0323 05:49:12.552463 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/enp135s2f1
I0323 05:49:12.552538 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/enp135s2f1/device/sriov_numvfs
I0323 05:49:12.552623 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/enp135s2f2
I0323 05:49:12.552698 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/enp135s2f2/device/sriov_numvfs
I0323 05:49:12.552777 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/enp135s2f3
I0323 05:49:12.552852 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/enp135s2f3/device/sriov_numvfs
I0323 05:49:12.552930 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/enp135s2f4
I0323 05:49:12.553010 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/enp135s2f4/device/sriov_numvfs
I0323 05:49:12.553090 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/enp135s2f5
I0323 05:49:12.553165 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/enp135s2f5/device/sriov_numvfs
I0323 05:49:12.553243 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/enp175s0f0
I0323 05:49:12.553321 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/enp175s0f0/device/sriov_numvfs
I0323 05:49:12.553403 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/enp175s0f1
I0323 05:49:12.553478 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/enp175s0f1/device/sriov_numvfs
I0323 05:49:12.553556 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/enp177s0f0
I0323 05:49:12.553635 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/enp177s0f0/device/sriov_numvfs
I0323 05:49:12.553714 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/enp177s0f1
I0323 05:49:12.553791 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/enp177s0f1/device/sriov_numvfs
I0323 05:49:12.553870 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/lo
I0323 05:49:12.553942 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/lo/device/sriov_numvfs
I0323 05:49:12.554015 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/veth389adcb
I0323 05:49:12.554087 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/veth389adcb/device/sriov_numvfs
I0323 05:49:12.554159 324440 sriov-device-plugin.go:106] Checking inside dir /sys/class/net/vethb391bc4b
I0323 05:49:12.554231 324440 sriov-device-plugin.go:117] Checking for file /sys/class/net/vethb391bc4b/device/sriov_numvfs
I0323 05:49:12.554307 324440 sriov-device-plugin.go:209] Discovering all capable and configured devices
I0323 05:49:12.554340 324440 sriov-device-plugin.go:218] Sriov Capable Path: /sys/class/net/eno1/device/sriov_totalvfs
I0323 05:49:12.554489 324440 sriov-device-plugin.go:230] Total number of VFs for device eno1 is 32
I0323 05:49:12.554523 324440 sriov-device-plugin.go:232] SRIOV capable device discovered: eno1
I0323 05:49:12.554641 324440 sriov-device-plugin.go:245] Number of Configured VFs for device eno1 is 0
I0323 05:49:12.554675 324440 sriov-device-plugin.go:218] Sriov Capable Path: /sys/class/net/eno2/device/sriov_totalvfs
I0323 05:49:12.554780 324440 sriov-device-plugin.go:230] Total number of VFs for device eno2 is 32
I0323 05:49:12.554812 324440 sriov-device-plugin.go:232] SRIOV capable device discovered: eno2
I0323 05:49:12.554916 324440 sriov-device-plugin.go:245] Number of Configured VFs for device eno2 is 0
I0323 05:49:12.554944 324440 sriov-device-plugin.go:218] Sriov Capable Path: /sys/class/net/enp135s0f0/device/sriov_totalvfs
I0323 05:49:12.555044 324440 sriov-device-plugin.go:230] Total number of VFs for device enp135s0f0 is 64
I0323 05:49:12.555075 324440 sriov-device-plugin.go:232] SRIOV capable device discovered: enp135s0f0
I0323 05:49:12.555204 324440 sriov-device-plugin.go:245] Number of Configured VFs for device enp135s0f0 is 6
I0323 05:49:12.557289 324440 sriov-device-plugin.go:218] Sriov Capable Path: /sys/class/net/enp135s0f1/device/sriov_totalvfs
I0323 05:49:12.557439 324440 sriov-device-plugin.go:230] Total number of VFs for device enp135s0f1 is 64
I0323 05:49:12.557485 324440 sriov-device-plugin.go:232] SRIOV capable device discovered: enp135s0f1
I0323 05:49:12.557645 324440 sriov-device-plugin.go:245] Number of Configured VFs for device enp135s0f1 is 0
I0323 05:49:12.557690 324440 sriov-device-plugin.go:218] Sriov Capable Path: /sys/class/net/enp175s0f0/device/sriov_totalvfs
I0323 05:49:12.557819 324440 sriov-device-plugin.go:230] Total number of VFs for device enp175s0f0 is 8
I0323 05:49:12.557861 324440 sriov-device-plugin.go:232] SRIOV capable device discovered: enp175s0f0
I0323 05:49:12.558021 324440 sriov-device-plugin.go:245] Number of Configured VFs for device enp175s0f0 is 0
I0323 05:49:12.558075 324440 sriov-device-plugin.go:218] Sriov Capable Path: /sys/class/net/enp175s0f1/device/sriov_totalvfs
I0323 05:49:12.558230 324440 sriov-device-plugin.go:230] Total number of VFs for device enp175s0f1 is 8
I0323 05:49:12.558276 324440 sriov-device-plugin.go:232] SRIOV capable device discovered: enp175s0f1
I0323 05:49:12.558421 324440 sriov-device-plugin.go:245] Number of Configured VFs for device enp175s0f1 is 0
I0323 05:49:12.558470 324440 sriov-device-plugin.go:218] Sriov Capable Path: /sys/class/net/enp177s0f0/device/sriov_totalvfs
I0323 05:49:12.558635 324440 sriov-device-plugin.go:230] Total number of VFs for device enp177s0f0 is 6
I0323 05:49:12.558680 324440 sriov-device-plugin.go:232] SRIOV capable device discovered: enp177s0f0
I0323 05:49:12.558821 324440 sriov-device-plugin.go:245] Number of Configured VFs for device enp177s0f0 is 0
I0323 05:49:12.558861 324440 sriov-device-plugin.go:218] Sriov Capable Path: /sys/class/net/enp177s0f1/device/sriov_totalvfs
I0323 05:49:12.558978 324440 sriov-device-plugin.go:230] Total number of VFs for device enp177s0f1 is 6
I0323 05:49:12.559007 324440 sriov-device-plugin.go:232] SRIOV capable device discovered: enp177s0f1
I0323 05:49:12.559092 324440 sriov-device-plugin.go:245] Number of Configured VFs for device enp177s0f1 is 0
I0323 05:49:12.559119 324440 sriov-device-plugin.go:262] Discovered SR-IOV PF devices: [enp135s0f0]
I0323 05:49:12.559166 324440 sriov-device-plugin.go:308] Starting SRIOV Network Device Plugin server at: /var/lib/kubelet/plugins_registry/sriovNet.sock
I0323 05:49:12.560940 324440 sriov-device-plugin.go:333] SRIOV Network Device Plugin server started serving
I0323 05:49:12.563297 324440 sriov-device-plugin.go:370] Plugin: sriovNet.sock gets registered successfully at Kubelet
I0323 05:49:12.563499 324440 sriov-device-plugin.go:385] ListAndWatch: send initial devices &ListAndWatchResponse{Devices:[&Device{ID:0000:87:02.3,Health:Healthy,} &Device{ID:0000:87:02.4,Health:Healthy,} &Device{ID:0000:87:02.5,Health:Healthy,} &Device{ID:0000:87:02.0,Health:Healthy,} &Device{ID:0000:87:02.1,Health:Healthy,} &Device{ID:0000:87:02.2,Health:Healthy,}],}
Leave the sriov device plugin running and open a new terminal session for following steps.
$ kubectl create -f pod-tc1.yaml
pod "testpod1" created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
sriov-device-plugin 1/1 Running 0 7h
testpod1 1/1 Running 0 3s
$ kubectl exec -it testpod1 -- ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth0@if17511: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
link/ether 0a:58:c0:a8:4a:b1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.74.177/24 scope global eth0
valid_lft forever preferred_lft forever
17508: net0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
link/ether ce:d8:06:08:e6:3f brd ff:ff:ff:ff:ff:ff
inet 10.56.217.179/24 scope global net0
valid_lft forever preferred_lft forever
$ kubectl exec -it testpod1 -- route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.74.1 0.0.0.0 UG 0 0 0 eth0
10.56.217.0 0.0.0.0 255.255.255.0 U 0 0 0 net0
192.168.0.0 192.168.74.1 255.255.0.0 UG 0 0 0 eth0
192.168.74.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
We welcome your feedback and contributions to this project. Please see the CONTRIBUTING.md for contribution guidelines.
Copyright 2018 © Intel Corporation.