MULTUS CNI plugin

MULTUS CNI plugin

Multus is a latin word for "Multi"
As the name suggests, it acts as a Multi plugin in Kubernetes and provides the multiple network interface support in a pod
Multus supports all reference plugins (eg. Flannel, DHCP, Macvlan) that implement the CNI specification and all 3rd party plugins (eg. Calico, Weave, Cilium, Contiv). In addition to it, Multus supports SRIOV, SRIOV-DPDK, OVS-DPDK & VPP workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes
It is a contact between the container runtime and other plugins, and it doesn't have any of its own net configuration, it calls other plugins like flannel/calico to do the real net conf job.
Multus reuses the concept of invoking delegates as used in flannel by grouping multiple plugins into delegates and invoking them in the sequential order of the CNI configuration file provided in json format
The default network gets "eth0" and additional network Pod interface name as “net0”, “net1”,… “netX and so on. Multus also support interface names from the user.
Multus is one of the projects in the Baremetal Container Experience kit.

Please check the CNI documentation for more information on container networking.

Quickstart Guide

Multus may be deployed as a Daemonset, and is provided in this guide along with Flannel. Flannel is deployed as a pod-to-pod network that is used as our "default network". Each network attachment is made in addition to this default network.

Firstly, clone this GitHub repository. We'll apply files to kubectl from this repo.

We apply these files as such:

$ cat ./images/{multus-daemonset.yml,flannel-daemonset.yml} | kubectl apply -f -

Create a CNI configuration loaded as a CRD object, in this case a macvlan CNI configuration is defined. You may replace the config field with any valid CNI configuration where the CNI binary is available on the nodes.

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "eth0",
      "mode": "bridge",
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.1.0/24",
        "rangeStart": "192.168.1.200",
        "rangeEnd": "192.168.1.216",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ],
        "gateway": "192.168.1.1"
      }
    }'
EOF

You may then create a pod which attached this additional interface, where the annotation correlates to the name in the NetworkAttachmentDefinition above.

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
  containers:
  - name: samplepod
    command: ["/bin/bash", "-c", "sleep 2000000000000"]
    image: dougbtv/centos-network
EOF

You may now inspect the pod and see that there is an additional interface configured, like so:

$ kubectl exec -it samplepod -- ip a

Kubernetes Network Custom Resource Definition De-facto Standard - Reference implementation

This project is a reference implementation for Kubernetes Network Custom Resource Definition De-facto Standard. For more information refer Network Plumbing Working Group Agenda
Kubernetes Network Custom Resource Definition De-facto Standard documentation link
Reference implementation support following modes
- CNI config JSON in network object
- Not using CNI config (“thick” plugin usecase)
- CNI configuration stored in on-disk file
refer the section 3.2 Network Object Definition for more details in Kubernetes Network Custom Resource Definition De-facto Standard
Refer the reference implementation presentation and demo details - link
Release version from v3.0 is not compatible with v1.0 and v2.0 version Network Object CRD
- MULTUS CNI pluginspecifications.

Multi-Homed pod

Building from source

This plugin requires Go 1.8 (or later) to build.

#./build

Work flow

Network configuration reference

Following is the example of multus config file, in /etc/cni/net.d/.

{
    "name": "node-cni-network",
    "type": "multus",
    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
    "confDir": "/etc/cni/multus/net.d",
    "cniDir": "/var/lib/cni/multus",
    "binDir": "/opt/cni/bin",
    "logFile": "/var/log/multus.log",
    "logLevel": "debug",
    /* NOTE: you can set clusterNetwork+defaultNetworks OR delegates!! (this is only for manual) */
    "clusterNetwork": "defaultCRD",
    "defaultNetworks": ["sidecarCRD", "flannel"],
    "delegates": [{
        "type": "weave-net",
        "hairpinMode": true
    }, {
        "type": "macvlan",
        ... (snip)
    }]
}

name (string, required): the name of the network
type (string, required): "multus"
confDir (string, optional): directory for CNI config file that multus reads. default /etc/cni/multus/net.d
cniDir (string, optional): Multus CNI data directory, default /var/lib/cni/multus
binDir (string, optional): directory for CNI plugins which multus calls. default /opt/cni/bin
kubeconfig (string, optional): kubeconfig file for the out of cluster communication with kube-apiserver. See the example kubeconfig. If you would like to use CRD (i.e. network attachment definition), this is required
logFile (string, optional): file path for log file. multus puts log in given file
logLevel (string, optional): logging level ("debug", "error" or "panic")
capabilities ({}list, optional): capabilities supported by at least one of the delegates. (NOTE: Multus only supports portMappings capability for now). See the example.

User should chose following parameters combination (clusterNetwork+defaultNetworks or delegates):

clusterNetwork (string, required): default CNI network for pods, used in kubernetes cluster (Pod IP and so on): name of network-attachment-definition, CNI json file name (without extention, .conf/.conflist) or directory for CNI config file
defaultNetworks ([]string, required): default CNI network attachment: name of network-attachment-definition, CNI json file name (without extention, .conf/.conflist) or directory for CNI config file
delegates ([]map,required): number of delegate details in the Multus

Network selection flow of clusterNetwork/defaultNetworks

Multus will find network for clusterNetwork/defaultNetworks as following sequences:

CRD object for given network name
CNI json config file in confDir. Given name should be without extention, like .conf/.conflist. (e.g. "test" for "test.conf")
Directory for CNI json config file. Multus will find alphabetically first file for the network.
Multus raise error message

Usage with Kubernetes CRD based network objects

Kubelet is responsible for establishing network interfaces for pods; it does this by invoking its configured CNI plugin. When Multus is invoked it retrieves network references from Pod annotation. Multus then uses these network references to get network configurations. Network configurations are defined as Kubernetes Custom Resource Object (CRD). These configurations describe which CNI plugins to invoke and what their configurations are. The order of plugin invocation is important as it identifies the primary plugin. This order is taken from network object references given in a Pod spec.

Creating "Network" resources in Kubernetes

You may wish to create the network-attachment-definition manually if you haven't installed using the daemonset technique, which includes the CRD, and you can verify if it's loaded with kubectl get crd and look for the presence of network-attachment-definition.

Create a Custom Resource Definition (CRD) crdnetwork.yaml; for the network object using the YAML from the examples directory.

$ kubectl create -f ./examples/crd.yml
customresourcedefinition.apiextensions.k8s.io/network-attachment-definitions.k8s.cni.cncf.io created

Run kubectl get command to check the Network CRD creation

$ kubectl get crd
NAME                      KIND
network-attachment-definitions.k8s.cni.cncf.io   CustomResourceDefinition.v1beta1.apiextensions.k8s.io

Creating CRD network resources in Kubernetes

After creating CRD network object you can create network resources in Kubernetes. These network resources may contain additional underlying CNI plugin parameters given in JSON format. In the following example shown below the args field contains parameters that will be passed into Flannel plugin.
Save the following YAML to flannel-network.yaml

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: flannel-networkobj
spec:
  config: '{
    "cniVersion": "0.3.0",
    "type": "flannel",
    "delegate": {
      "isDefaultGateway": true
    }
  }'

Create the custom resource definition

$ kubectl create -f ./flannel-network.yaml
network "flannel-networkobj" created

$ kubectl get net-attach-def
NAME                         AGE
flannel-networkobj           26s

Get the custom network object details

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  clusterName: ""
  creationTimestamp: 2018-05-17T09:13:20Z
  deletionGracePeriodSeconds: null
  deletionTimestamp: null
  initializers: null
  name: flannel-networkobj
  namespace: default
  resourceVersion: "21176114"
  selfLink: /apis/k8s.cni.cncf.io/v1/namespaces/default/networks/flannel-networkobj
  uid: 8ac8f873-59b2-11e8-8308-a4bf01024e6f
spec:
  config: '{ "cniVersion": "0.3.0", "type": "flannel", "delegate": { "isDefaultGateway":
    true } }'

Save the following YAML to sriov-network.yaml to creating sriov network object. ( Refer to Intel - SR-IOV CNI or contact @kural in Intel-Corp Slack for running the DPDK based workloads in Kubernetes)

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-conf
spec:
  config: '{
    "type": "sriov",
    "if0": "enp12s0f1",
    "ipam": {
            "type": "host-local",
            "subnet": "10.56.217.0/24",
            "rangeStart": "10.56.217.171",
            "rangeEnd": "10.56.217.181",
            "routes": [
                    { "dst": "0.0.0.0/0" }
            ],
            "gateway": "10.56.217.1"
    }
  }'

Likewise save the following YAML to sriov-vlanid-l2enable-network.yaml to create another sriov based network object:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-vlanid-l2enable-conf
spec:
  config: '{
    "type": "sriov",
    "if0": "enp2s0",
    "vlan": 210,
    "l2enable": true
  }'

Follow step 3 above to create "sriov-vlanid-l2enable-conf" and "sriov-conf" network objects
View network objects using kubectl

# kubectl get net-attach-def
NAME                         AGE
flannel-networkobj           29m
sriov-conf                   6m
sriov-vlanid-l2enable-conf   2m

Configuring Multus to use the kubeconfig

Create a Mutlus CNI configuration file on each Kubernetes node. This file should be created in: /etc/cni/net.d/multus-cni.conf with the content shown below. Use only the absolute path to point to the kubeconfig file (as it may change depending upon your cluster env). We are assuming all CNI plugin binaries are default location (/opt/cni/bin dir)

{
    "name": "node-cni-network",
    "type": "multus",
    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml"
}

Configuring Multus to use kubeconfig and a default network

Many users want Kubernetes default networking feature along with network objects. Refer to issues #14 & #17 for more information. In the following Multus configuration, Weave act as the default network in the absence of network field in the pod metadata annotation.

{
    "name": "node-cni-network",
    "type": "multus",
    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
    "delegates": [{
        "type": "weave-net",
        "hairpinMode": true
    }]
}

Configurations referenced in annotations are created in addition to the default network.

Configuring Pod to use the CRD network objects

Save the following YAML to pod-multi-network.yaml. In this case flannel-conf network object acts as the primary network.

# cat pod-multi-network.yaml
apiVersion: v1
kind: Pod
metadata:
  name: multus-multi-net-poc
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
            { "name": "flannel-conf" },
            { "name": "sriov-conf" },
            { "name": "sriov-vlanid-l2enable-conf",
              "interface": "north" }
    ]'
spec:  # specification of the pod's contents
  containers:
  - name: multus-multi-net-poc
    image: "busybox"
    command: ["top"]
    stdin: true
    tty: true

Create Multiple network based pod from the master node

# kubectl create -f ./pod-multi-network.yaml
pod "multus-multi-net-poc" created

Get the details of the running pod from the master

# kubectl get pods
NAME                   READY     STATUS    RESTARTS   AGE
multus-multi-net-poc   1/1       Running   0          30s

Pod Annotation Parameters

JSON formated network annotation in Pod can have several parameters as following:

namespace: Kubernetes namespace that the target network attach definition is defined in.
mac: MAC address (e.g "c2:11:22:33:44:66") for target network interface
interface: interface name for target network interface

Note: If you add mac, please add 'tuning' plugin into target network attach definition as CNI plugin chaining as following.

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan with tuning
spec:
  config: '{
      "cniVersion": "0.3.0",
      "name": "chains",
      "plugins": [ {
        "type": "macvlan",
        "master": "eth0",
        "mode": "bridge",
          "ipam": {
            "type": "host-local",
            "subnet": "192.168.1.0/24",
            "rangeStart": "192.168.1.200",
            "rangeEnd": "192.168.1.216",
            "routes": [
              { "dst": "0.0.0.0/0" }
            ],
            "gateway": "192.168.1.1"
          }
        },
        {
          "type":"tuning"
        }]

  }'

Verifying Pod network interfaces

Run ifconfig command in Pod:

# kubectl exec -it multus-multi-net-poc -- ifconfig
eth0      Link encap:Ethernet  HWaddr C6:43:7C:09:B4:9C
          inet addr:10.128.0.4  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:42 (42.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

net0      Link encap:Ethernet  HWaddr 06:21:91:2D:74:B9
          inet addr:192.168.42.3  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::421:91ff:fe2d:74b9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

net1      Link encap:Ethernet  HWaddr D2:94:98:82:00:00
          inet addr:10.56.217.171  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::d094:98ff:fe82:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:120 (120.0 B)  TX bytes:648 (648.0 B)

north     Link encap:Ethernet  HWaddr BE:F2:48:42:83:12
          inet6 addr: fe80::bcf2:48ff:fe42:8312/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1420 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1276 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:95956 (93.7 KiB)  TX bytes:82200 (80.2 KiB)

Interface name	Description
lo	loopback
eth0	weave network interface
net0	Flannel network tap interface
net1	VF0 of NIC 1 assigned to the container by Intel - SR-IOV CNI plugin
north	VF0 of NIC 2 assigned with VLAN ID 210 to the container by SR-IOV CNI plugin

Check the vlan ID of the NIC 2 VFs

# ip link show enp2s0
20: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 24:8a:07:e8:7d:40 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, vlan 210, spoof checking off, link-state auto
    vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
    vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
    vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto

Using with Multus conf file

Given the following network configuration:

# tee /etc/cni/net.d/multus-cni.conf <<-'EOF'
{
    "name": "multus-demo-network",
    "type": "multus",
    "delegates": [
        {
                "type": "sriov",
                #part of sriov plugin conf
                "if0": "enp12s0f0",
                "ipam": {
                        "type": "host-local",
                        "subnet": "10.56.217.0/24",
                        "rangeStart": "10.56.217.131",
                        "rangeEnd": "10.56.217.190",
                        "routes": [
                                { "dst": "0.0.0.0/0" }
                        ],
                        "gateway": "10.56.217.1"
                }
        },
        {
                "type": "ptp",
                "ipam": {
                        "type": "host-local",
                        "subnet": "10.168.1.0/24",
                        "rangeStart": "10.168.1.11",
                        "rangeEnd": "10.168.1.20",
                        "routes": [
                                { "dst": "0.0.0.0/0" }
                        ],
                        "gateway": "10.168.1.1"
                }
        },
        {
                "type": "flannel",
                "delegate": {
                        "isDefaultGateway": true
                }
        }
    ]
}
EOF

Logging Options

You may wish to enable some enhanced logging for Multus, especially during the process where you're configuring Multus and need to understand what is or isn't working with your particular configuration.

Multus will always log via STDERR, which is the standard method by which CNI plugins communicate errors, and these errors are logged by the Kubelet. This method is always enabled.

Writing to a Log File

Optionally, you may have Multus log to a file on the filesystem. This file will be written locally on each node where Multus is executed. You may configure this via the logFile option in the CNI configuration. By default this additional logging to a flat file is disabled.

For example in your CNI configuration, you may set:

    "logFile": "/var/log/multus.log",

Logging Level

The default logging level is set as panic -- this will log only the most critical errors, and is the least verbose logging level.

The available logging level values, in decreasing order of verbosity are:

debug
error
panic

You may configure the logging level by using the LogLevel option in your CNI configuration. For example:

    "LogLevel": "debug",

CNI running with Network device plugin

Allocation of the Network device(such as SRIOV VFs) are done by Device plugins(Eg.SRIOV Network device plugin), Multus developed to work in the co-existence enviroment to work with device plugin by passing down the allocated device information to the CNI plugins.

Device plugin & CNI, NUMA Manager alignment - technical architecture document
Reference implementation : SRIOV Network device plugin
Example: How to make Multus work with device plugin?

Default Network Readiness Checks

You may wish for your "default network" (that is, the CNI plugin & its configuration you specify as your default delegate) to become ready before you attach networks with Multus. This is disabled by default and not used unless you add the readiness check option(s) to your CNI configuration file.

For example, if you use Flannel as a default network, the recommended method for Flannel to be installed is via a daemonset that also drops a configuration file in /etc/cni/net.d/. This may apply to other plugins that place that configuration file upon their readiness, hence, Multus uses their configuration filename as a semaphore and optionally waits to attach networks to pods until that file exists.

In this manner, you may prevent pods from crash looping, and instead wait for that default network to be ready.

Only one option is necessary to configure this functionality:

readinessindicatorfile: The path to a file whose existance denotes that the default network is ready.

NOTE: If readinessindicatorfile is unset, or is an empty string, this functionality will be disabled, and is disabled by default.

Testing Multus CNI

Multiple flannel networks

Github user YYGCui has used multiple flannel network to work with Multus CNI plugin. Please refer to this closed issue for ,multiple overlay network support with Multus CNI.

Make sure that the multus, sriov, flannel, and ptp binaries are in the /opt/cni/bin directories and follow the steps as mentioned in the CNI

Configure Kubernetes with CNI

Kubelet must be configured to run with the CNI network plugin. Edit /etc/kubernetes/kubelet file and add --network-plugin=cni flags in KUBELET\_OPTS as shown below:

KUBELET_OPTS="...
--network-plugin-dir=/etc/cni/net.d
--network-plugin=cni
"

Refer to the Kubernetes User Guide and network plugin for more information.

Launching workloads in Kubernetes

With Multus CNI configured as described in sections above each workload launched via a Kubernetes Pod will have multiple network interfacesLaunch the workload using yaml file in the kubernetes master, with above configuration in the multus CNI, each pod should have multiple interfaces.

Note: To verify whether Multus CNI plugin is working correctly, create a pod containing one busybox container and execute ip link command to check if interfaces management follows configuration.

Create multus-test.yaml file containing below configuration. Created pod will consist of one busybox container running top command.

apiVersion: v1
kind: Pod
metadata:
  name: multus-test
spec:  # specification of the pod's contents
  restartPolicy: Never
  containers:
  - name: test1
    image: "busybox"
    command: ["top"]
    stdin: true
    tty: true

Create pod using command:

# kubectl create -f multus-test.yaml
pod "multus-test" created

Run "ip link" command inside the container:

# 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth0@if41: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
    link/ether 26:52:6b:d8:44:2d brd ff:ff:ff:ff:ff:ff
20: net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
    link/ether f6:fb:21:4f:1d:63 brd ff:ff:ff:ff:ff:ff
21: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
    link/ether 76:13:b1:60:00:00 brd ff:ff:ff:ff:ff:ff

Interface name	Description
lo	loopback
eth0@if41	Flannel network tap interface
net0	VF assigned to the container by SR-IOV CNI plugin
net1	ptp localhost interface

Multus additional plugins

DPDK-SRIOV CNI
Vhostuser CNI - a Dataplane network plugin - Supports OVS-DPDK & VPP
Bond CNI - For fail-over and high availability of networking

NFV based networking in Kubernetes

KubeCon workshop on "Enabling NFV features in Kubernetes" presentation slide deck
Feature brief
- Multiple Network Interface Support in Kubernetes
- Enhanced Platform Awareness in Kubernetes
Application note
- Multiple Network Interfaces in Kubernetes and Container Bare Metal
- Enhanced Platform Awareness Features in Kubernetes
White paper
- Enabling New Features with Kubernetes for NFV
Multus's related project github pages

Need help

Read Containers Experience Kits
Try our container exp kit demo - KubeCon workshop on Enabling NFV Features in Kubernetes
Join us on #intel-sddsg-slack slack channel and ask question in #general-discussion
You can also email us
Feel free to submit an issue

Please fill in the Questions/feedback - google-form!

Contacts

For any questions about Multus CNI, please reach out on github issue or feel free to contact the developer @kural in our Intel-Corp Slack

AlonaKaplan/multus-cni