Kube-node
and node-annotation
are IPAM CNI-plugins that assigns IP addresses
to PODs based on the Kubernetes node object. Node-annotation
is an
experimental plugin written with shell scripts. It is kept as a
reference. Kube-node
is written in go and should be prefered.
For addresses on the main Kubernetes network, eth0
in PODs, the
address ranges (subnets) are taken from node.spec.podCIDRs
. This is
automatically generated if the kube-controller-manager
is started
with --allocate-node-cidrs=true
. Example:
# kubectl get node vm-002 -o json | jq .spec.podCIDRs
[
"11.0.1.0/24",
"1100:0:0:1::/64"
]
{
"name": "k8snet",
"cniVersion": "1.0.0",
"isDefaultGateway": true,
"ipam": {
"type": "kube-node",
"kubeconfig": "/etc/kubernetes/kubeconfig",
"dataDir": "/run/container-ipam-state/k8snet"
}
}
A few things should be noted. Kube-node
must be able to read node
objects from the K8s API-server, so it must have a working
kubeconfig
. This may be omitted if the kubeconfig can be found in
other ways, for instance using the $KUBECONFIG
environment variable.
Kube-node
delegates address allocation to the host-local ipam, so the dataDir
should be set to a directory that is cleared on node reboot (e.g. a
tmpfs
).
In large clusters IPv4 addresses may become a limiting resource.
Kube-node
can be configured to only assign IPv4 address to some namespaces.
{
"name": "k8snet",
"cniVersion": "1.0.0",
"isDefaultGateway": true,
"ipam": {
"type": "kube-node",
"kubeconfig": "/etc/kubernetes/kubeconfig",
"dataDir": "/run/container-ipam-state/k8snet",
"ipv4-namespaces": [
"old-application"
]
}
}
WARNING: If this is used, Kubernetes must be configured to use
IPv6 addresses for access to the API-server and as default for
services. This is done by specifying an IPv6 address for the
--advertise-address
and set an IPv6 address first in
--service-cluster-ip-range
when the kube-apiserver
is started.
if kube-node
is used to assign addresses on interfaces which are
not the Kubernetes network, for instance interfaces added with Multus, the address
ranges (subnets) can be specified in an annotation on the node object.
kubectl annotate node vm-002 kube-node.nordix.org/net1=172.20.2.0/24,fd00::2:0:0/96
{
"name": "net1",
"cniVersion": "1.0.0",
"isDefaultGateway": false,
"ipam": {
"type": "kube-node",
"kubeconfig": "/etc/kubernetes/kubeconfig",
"dataDir": "/run/container-ipam-state/net1",
"annotation": "kube-node.nordix.org/net1"
}
}
Note that isDefaultGateway
is set to "false" for secondary networks.
./build.sh # Print help
./build.sh binaries
The node-annotation
is an IPAM CNI-plugin that reads the addresses
from an annotation in the Kubernetes node object.
Example:
{
"name": "mynet",
"type": "ipvlan",
"master": "eth0",
"ipam": {
"type": "node-annotation",
"annotation": "example.com/ipvlan-ranges"
}
}
And in the node object:
# kubectl get node $(hostname) -o json | jq .metadata.annotations
{
"example.com/ipvlan-ranges": "ranges: [[{\"subnet\": \"172.16.5.0/24\"}]]",
"node.alpha.kubernetes.io/ttl": "0",
"volumes.kubernetes.io/controller-managed-attach-detach": "true"
}
The node-annotation
ipam is a wrapper for the host-local ipam (by
default). Anything in the annotation will be inserted in a
host-local
config. In the figure everything in bold is taken from
the annotation.
This gives you the freedom to use any host-local
configuration.
For now node-annotation
is implemented as a shell script. It is
intended mainly for testing.
node-annotation
shall be installed in the cni-bin directory, usually
"/opt/cni/bin". node-annotation
must be able to get the K8s node
objects using kubectl get nodes -o json
and analyze with jq.
Configuration is in json
format and is read from
/etc/cni/node-annotation.conf
or $NODE_ANNOTATION_CFG
. Example;
{
"kubeconfig": "/etc/kubernetes/kubeconfig",
"nextipam": "/opt/cgi/bin/host-local"
}
kubeconfig
is needed unless $KUBECONFIG
is defined. nextipam
is
optional and may be used to chain with another plugin than host-local
.
NOTE; a "key" must only contain characters that are valid in a shell script variable. That means no dash (-).
The node-annotation
script can be invoked with a parameter to test
some things on a cluster.
# /opt/cni/bin/node-annotation -h
node-annotation --
An IPAM CNI-plugin that uses annotations on the K8s node object
https://github.com/Nordix/ipam-node-annotation
Commands;
ipam
Act as an ipam CNI-plugin. This is the default command
error_quit [msg]
Print an error in standard CNI json format and quit
my_node
Print the own node object
get_annotation [--node=node] <annotation>
Print the value of the annotation in the K8s node object
# kubectl annotate node $(hostname) example.com/ipvlan-ranges="\"ranges\": [
{ \"subnet\": \"4000::16.0.0.0/120\" },
{ \"subnet\": \"16.0.0.0/24\" }
]"
# /opt/cni/bin/node-annotation get_annotation example.com/ipvlan-ranges
"ranges": [
[{ "subnet": "4000::16.0.0.0/120" }],
[{ "subnet": "16.0.0.0/24" }]
]
A test of the entire function;
# cat > test.cfg <<EOF
{
"name": "mynet",
"type": "ipvlan",
"master": "eth0",
"ipam": {
"type": "node-annotation",
"annotation": "example.com/ipvlan-ranges"
}
}
EOF
# export NODE_ANNOTATION_CFG=./nacfg
# cat > $NODE_ANNOTATION_CFG <<EOF
{
"nextipam": "cat"
}
EOF
cat test.cfg | /opt/cni/bin/node-annotation
{
"name": "mynet",
"type": "ipvlan",
"master": "eth0",
"ipam": {
"type": "cat",
"ranges": [
{
"subnet": "4000::16.0.0.0/120"
},
{
"subnet": "16.0.0.0/24"
}
]
}
}
If you have Multus
running in kind there are some steps to make
node-annotation
work.
First copy the node-annotation
to "/opt/cni/bin" on all workers.
Only the controller have a kubeconfig at "/etc/kubernetes/admin.conf".
Create one at the workers and configure node-annotation
to use it;
kind get kubeconfig --internal | \
docker exec -i kind-worker tee /etc/kubernetes/kubeconfig > /dev/null
echo "{ \"kubeconfig\": \"/etc/kubernetes/kubeconfig\" }" | docker exec -i kind-worker \
tee /etc/cni/node-annotation.conf > /dev/null
# (repeat for all workers)
The Multus kubeconfig (/etc/cni/net.d/multus.d/multus.kubeconfig) can't be used due to restrictions.
Now you can annotate the workers and create a NAD. Example;
kubectl annotate node worker example.com/bridge="\"ranges\": [
[{ \"subnet\":\"4000::16.0.0.0/120\", \"rangeStart\":\"4000::16.0.0.0\" , \"rangeEnd\":\"4000::16.0.0.7\"}],
[{ \"subnet\":\"16.0.0.0/24\", \"rangeStart\":\"16.0.0.0\" , \"rangeEnd\":\"16.0.0.7\"}]
]"
cat | kubectl apply -f - <<EOF
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: meridio-bridge
spec:
config: '{
"cniVersion": "0.4.0",
"type": "bridge",
"bridge": "cbr2",
"isGateway": true,
"ipam": {
"type": "node-annotation",
"annotation": "example.com/bridge"
}
}'
EOF