Whereabouts IPAM not working on a IPV6 only K8s cluster
Closed this issue · 5 comments
Hi, am using a combination of k8s (v1.18.6) + docker (v19.03.6) + calico (v3.15.1) + multus (v3.41), and trying to get IP from whereabouts IPAM via NetworkAttachmentDefinition, as explained in the docs. All works fine when am on a IPV4 only based K8s cluster, but the moment I change to IPV6 only networking, I see failures in whereabouts IPAM logs, and the pods is always stuck in container creating state:
Pod Status
NAME READY STATUS RESTARTS AGE
sample 0/1 ContainerCreating 0 2m45s
Kubernetes Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP fd33:ca9b:3a09:868c:172:18:0:8001 443/TCP 21m
Pod Description
Events:
Type Reason Age From Message
Normal Scheduled 3m4s default-scheduler Successfully assigned default/sample to w2
Warning FailedCreatePodSandBox 2m30s kubelet,w2 Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "363cae6628c8b9facb0f24dd621f8df7e6d7a5570d5366f88e8ce8e0cbf14886" network for pod "sample": networkPlugin cni failed to set up pod "sample_default" network: Multus: [default/sample]: error adding container to network "custom-network": delegateAdd: error invoking DelegateAdd - "sriov": error in getting result from AddNetwork: failed to set up IPAM plugin type "whereabouts" from the device "sriov1": Error assigning IP: Get https://fd33:ca9b:3a09:868c:172:18:0:8001:443/api?timeout=32s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers), failed to clean up sandbox container "363cae6628c8b9facb0f24dd621f8df7e6d7a5570d5366f88e8ce8e0cbf14886" network for pod "sample": networkPlugin cni failed to teardown pod "sample_default" network: delegateDel: error invoking DelegateDel - "sriov": error in getting result from DelNetwork: error reading cached NetConf in /var/lib/cni/sriov with name 363cae6628c8b9facb0f24dd621f8df7e6d7a5570d5366f88e8ce8e0cbf14886-net1]
Whereabouts IPAM Logs :
2020-08-13T05:03:36+09:00 [debug] ADD - IPAM configuration successfully read: {Name:custom-network Type:whereabouts Routes:[] Datastore:kubernetes Addresses:[] OmitRanges:[] DNS:{Nameservers:[] Domain: Search:[] Options:[]} Range:f456::/124 RangeStart:f456:: RangeEnd: GatewayStr: EtcdHost: EtcdUsername: EtcdPassword:********* EtcdKeyFile: EtcdCertFile: EtcdCACertFile: LogFile:/tmp/whereabouts.log LogLevel:debug Gateway: Kubernetes:{KubeConfigPath:/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig K8sAPIRoot:} ConfigurationPath:}
2020-08-13T05:03:36+09:00 [debug] Beginning IPAM for ContainerID: ec60f9971715d518425570124b76c65dbce43a6a0fbce0a8b1d49e9a8e948003
2020-08-13T05:03:36+09:00 [debug] IPManagement -- mode: 0 / host: / containerID: ec60f9971715d518425570124b76c65dbce43a6a0fbce0a8b1d49e9a8e948003
**2020-08-13T05:04:08+09:00 [error] IPAM kubernetes client initialization error: Get https://fd33:ca9b:3a09:868c:172:18:0:8001:443/api?timeout=32s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-08-13T05:04:08+09:00 [error] Error assigning IP: Get https://fd33:ca9b:3a09:868c:172:18:0:8001:443/api?timeout=32s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)**
Pod Yaml
apiVersion: v1
kind: Pod
metadata:
name: sample
annotations:
k8s.v1.cni.cncf.io/networks: nad-whereabouts
spec:
containers:
- name: sample
image: centos:latest
imagePullPolicy: IfNotPresent
command: ["/bin/bash","-c","while true; do sleep 1000; done"]
resources:
requests:
intel.com/sriov_netdevice_vf: 1
limits:
intel.com/sriov_netdevice_vf: 1
NAD Yaml
kind: NetworkAttachmentDefinition
metadata:
name: nad-whereabouts
annotations:
k8s.v1.cni.cncf.io/resourceName: intel.com/sriov_netdevice_vf
spec:
config: '{
"cniVersion": "0.3.1",
"type": "sriov",
"name": "custom-network",
"ipam": {
"type": "whereabouts",
"datastore": "kubernetes",
"kubernetes": { "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig" },
"range": "f456::/124",
"log_file" : "/tmp/whereabouts.log",
"log_level" : "debug"
}
}'
I hope whereabouts works on a IPV6 only cluster as well. Any help will be appreciated, thanks in advance !
i think it is the same issue: https://github.com/dougbtv/whereabouts/pull/47
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle rotten
/remove-lifecycle stale
Rotten issues close after 30d of inactivity.
Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.
/close
@openshift-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue by commenting
/reopen.
Mark the issue as fresh by commenting/remove-lifecycle rotten.
Exclude this issue from closing again by commenting/lifecycle frozen./close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.