Integration with CNI + calico v3.0 fail

Question

Integration with CNI + calico v3.0 fail

sandromello opened this issue 7 years ago · 6 comments

I've followed the deploy.md guide and installed frakti with success along with kubernetes (dns, calico, etc). Instead of running the self hosted install of calico in the docs/networking.md instructions, I followed the v3.0 guide of calico: kubectl apply -f https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/kubeadm/1.7/calico.yaml
The logs of frakti informs an error when updating the data store (calico etcd).

Because of this error the POD doesn't start. Above there're the logs of frakti:

Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.129 [INFO][19037] calico.go 68: Extracted identifiers EndpointIDs=&utils.WEPIdentifiers{Namespace:"default", WEPName:"", WorkloadEndpointIdentifiers:names.WorkloadEndpointIdentifiers{Node:"ip-10-0-83-184", Orchestrator:"k8s", Endpoint:"eth0", Workload:"", Pod:"nginx", ContainerID:"k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173"}}
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.129 [INFO][19037] calico.go 69: Loaded CNI NetConf NetConfg=types.NetConf{CNIVersion:"0.3.0", Name:"k8s-pod-network", Type:"calico", IPAM:struct { Name string; Type string "json:\"type\""; Subnet string "json:\"subnet\""; AssignIpv4 *string "json:\"assign_ipv4\""; AssignIpv6 *string "json:\"assign_ipv6\""; IPv4Pools []string "json:\"ipv4_pools,omitempty\""; IPv6Pools []string "json:\"ipv6_pools,omitempty\"" }{Name:"", Type:"calico-ipam", Subnet:"", AssignIpv4:(*string)(nil), AssignIpv6:(*string)(nil), IPv4Pools:[]string(nil), IPv6Pools:[]string(nil)}, MTU:1500, Hostname:"", Nodename:"", DatastoreType:"", EtcdAuthority:"", EtcdEndpoints:"http://10.96.232.136:6666", LogLevel:"info", Policy:types.Policy{PolicyType:"k8s", K8sAPIRoot:"https://10.96.0.1:443", K8sAuthToken:"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjYWxpY28tY25pLXBsdWdpbi10b2tlbi1zdDd3cyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjYWxpY28tY25pLXBsdWdpbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjAxNmUyOGMwLTA4MjQtMTFlOC1iM2M0LTBlYTU3MzFmMmY1ZSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTpjYWxpY28tY25pLXBsdWdpbiJ9.AC5lyPL1v-y6_XHBQyAdMr4OOJGDCEWuxfA4NhO_GvbqbwVpYC5ekj8meYIsh1bjEgSdVOU5fN8AQYpGA1qR8JCMPhifKUbIzvlcJVNoBbscDEpvsXjKEhv-Aev_LQ1yrCClczSXg77xwhwXcz5vaQx_gQd5u-BIYtp2V0ZqT3aQJL1WOMF-VZdbOuGST1hMOWsj6uGnSw2gZJmmbRYvS3idfTsKNgUNVm5XB2yfDxmQE7DCHWcXf-KMW8cJ7n6pHocu22IBwngj3aNUfRqcVbsvmryTxOB5tuiir6HmDu8S7KVQlL_DdxMCF9CumNPMkB28Kur0vqDLKyFfmCGSOw", K8sClientCertificate:"", K8sClientKey:"", K8sCertificateAuthority:""}, Kubernetes:types.Kubernetes{K8sAPIRoot:"", Kubeconfig:"/etc/cni/net.d/calico-kubeconfig", NodeName:""}, Args:types.Args{Mesos:types.Mesos{NetworkInfo:types.NetworkInfo{Name:"", Labels:struct { Labels []struct { Key string "json:\"key\""; Value string "json:\"value\"" } "json:\"la
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: bels,omitempty\"" }{Labels:[]struct { Key string "json:\"key\""; Value string "json:\"value\"" }(nil)}}}}, EtcdScheme:"", EtcdKeyFile:"", EtcdCertFile:"", EtcdCaCertFile:""}
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.129 [INFO][19037] utils.go 327: Configured environment: [CNI_COMMAND=ADD CNI_CONTAINERID=k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173 CNI_NETNS=/var/run/netns/cni-76bf6400-2f69-64ad-77b1-8cd337b88a4b CNI_ARGS=IgnoreUnknown=1;IgnoreUnknown=1;K8S_POD_NAME=nginx;K8S_POD_NAMESPACE=default;K8S_POD_INFRA_CONTAINER_ID=k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173 CNI_IFNAME=eth0 CNI_PATH=/opt/cni/bin:/opt/calico/bin PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ETCD_ENDPOINTS=http://10.96.232.136:6666 KUBECONFIG=/etc/cni/net.d/calico-kubeconfig K8S_API_TOKEN=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjYWxpY28tY25pLXBsdWdpbi10b2tlbi1zdDd3cyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjYWxpY28tY25pLXBsdWdpbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjAxNmUyOGMwLTA4MjQtMTFlOC1iM2M0LTBlYTU3MzFmMmY1ZSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTpjYWxpY28tY25pLXBsdWdpbiJ9.AC5lyPL1v-y6_XHBQyAdMr4OOJGDCEWuxfA4NhO_GvbqbwVpYC5ekj8meYIsh1bjEgSdVOU5fN8AQYpGA1qR8JCMPhifKUbIzvlcJVNoBbscDEpvsXjKEhv-Aev_LQ1yrCClczSXg77xwhwXcz5vaQx_gQd5u-BIYtp2V0ZqT3aQJL1WOMF-VZdbOuGST1hMOWsj6uGnSw2gZJmmbRYvS3idfTsKNgUNVm5XB2yfDxmQE7DCHWcXf-KMW8cJ7n6pHocu22IBwngj3aNUfRqcVbsvmryTxOB5tuiir6HmDu8S7KVQlL_DdxMCF9CumNPMkB28Kur0vqDLKyFfmCGSOw]
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.131 [INFO][19037] k8s.go 59: Extracted identifiers for CmdAddK8s ContainerID="k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" Namespace="default" Pod="nginx" WorkloadEndpoint="ip--10--0--83--184-k8s-nginx-eth0"
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.182 [INFO][19037] k8s.go 648: pod info &Pod{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:nginx,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/pods/nginx,UID:6430a93d-0824-11e8-b3c4-0ea5731f2f5e,ResourceVersion:757,Generation:0,CreationTimestamp:2018-02-02 14:21:42 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:PodSpec{Volumes:[{default-token-sfnpc {nil nil nil nil nil SecretVolumeSource{SecretName:default-token-sfnpc,Items:[],DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{nginx nginx:1.7.9 [] []  [{ 0 80 TCP }] [] [] {map[] map[]} [{default-token-sfnpc true /var/run/secrets/kubernetes.io/serviceaccount  <nil>}] nil nil nil /dev/termination-log File IfNotPresent nil false false false}],RestartPolicy:Always,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:ip-10-0-83-184,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,},ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[{node.kubernetes.io/not-ready Exists  NoExecute 0xc4203df360} {node.kubernetes.io/unreachable Exists  NoExecute 0xc4203df380}],HostAliases:[],PriorityClassName:,Priority:nil,},Status:PodStatus{Phase:Pending,Conditions:[{Initialized True 0001-01-01 00:00:00 +0000 UTC 2018-02-02 14:21:42 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2018-02-02 14:21:42 +0000 UTC ContainersNotReady containers with unready status: [nginx]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2018-02-02 14:21:42 +0000 UTC  }],Message
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: :,Reason:,HostIP:10.0.83.184,PodIP:,StartTime:2018-02-02 14:21:42 +0000 UTC,ContainerStatuses:[{nginx {ContainerStateWaiting{Reason:ContainerCreating,Message:,} nil nil} {nil nil nil} false 0 nginx:1.7.9  }],QOSClass:BestEffort,InitContainerStatuses:[],},}
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.245 [INFO][19043] utils.go 327: Configured environment: [CNI_COMMAND=ADD CNI_CONTAINERID=k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173 CNI_NETNS=/var/run/netns/cni-76bf6400-2f69-64ad-77b1-8cd337b88a4b CNI_ARGS=IgnoreUnknown=1;IgnoreUnknown=1;K8S_POD_NAME=nginx;K8S_POD_NAMESPACE=default;K8S_POD_INFRA_CONTAINER_ID=k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173 CNI_IFNAME=eth0 CNI_PATH=/opt/cni/bin:/opt/calico/bin PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ETCD_ENDPOINTS=http://10.96.232.136:6666 KUBECONFIG=/etc/cni/net.d/calico-kubeconfig K8S_API_TOKEN=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjYWxpY28tY25pLXBsdWdpbi10b2tlbi1zdDd3cyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjYWxpY28tY25pLXBsdWdpbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjAxNmUyOGMwLTA4MjQtMTFlOC1iM2M0LTBlYTU3MzFmMmY1ZSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTpjYWxpY28tY25pLXBsdWdpbiJ9.AC5lyPL1v-y6_XHBQyAdMr4OOJGDCEWuxfA4NhO_GvbqbwVpYC5ekj8meYIsh1bjEgSdVOU5fN8AQYpGA1qR8JCMPhifKUbIzvlcJVNoBbscDEpvsXjKEhv-Aev_LQ1yrCClczSXg77xwhwXcz5vaQx_gQd5u-BIYtp2V0ZqT3aQJL1WOMF-VZdbOuGST1hMOWsj6uGnSw2gZJmmbRYvS3idfTsKNgUNVm5XB2yfDxmQE7DCHWcXf-KMW8cJ7n6pHocu22IBwngj3aNUfRqcVbsvmryTxOB5tuiir6HmDu8S7KVQlL_DdxMCF9CumNPMkB28Kur0vqDLKyFfmCGSOw]
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: Calico CNI IPAM request count IPv4=1 IPv6=0
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: Calico CNI IPAM handle=k8s-pod-network.k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.246 [INFO][19043] calico-ipam.go 186: Auto assigning IP ContainerID="k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" HandleID="k8s-pod-network.k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" Workload="ip--10--0--83--184-k8s-nginx-eth0" assignArgs=ipam.AutoAssignArgs{Num4:1, Num6:0, HandleID:(*string)(0xc420188130), Attrs:map[string]string(nil), Hostname:"ip-10-0-83-184", IPv4Pools:[]net.IPNet{}, IPv6Pools:[]net.IPNet{}}
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.246 [INFO][19043] ipam.go 66: Auto-assign 1 ipv4, 0 ipv6 addrs for host 'ip-10-0-83-184'
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.247 [INFO][19043] ipam_block.go 344: New allocation attribute: {AttrPrimary:0xc420188130 AttrSecondary:map[]}
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.248 [INFO][19043] ipam.go 752: Creating new handle: k8s-pod-network.k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.252 [INFO][19043] ipam.go 223: Auto-assigned 1 out of 1 IPv4s: [192.168.142.195]
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: Calico CNI IPAM assigned addresses IPv4=[192.168.142.195] IPv6=[]
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.252 [INFO][19043] calico-ipam.go 214: IPAM Result ContainerID="k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" HandleID="k8s-pod-network.k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" Workload="ip--10--0--83--184-k8s-nginx-eth0" result.IPs=[]*current.IPConfig{(*current.IPConfig)(0xc4203fc180)}
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.254 [INFO][19037] k8s.go 268: Populated endpoint ContainerID="k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" Namespace="default" Pod="nginx" WorkloadEndpoint="ip--10--0--83--184-k8s-nginx-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"ip--10--0--83--184-k8s-nginx-eth0", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"projectcalico.org/namespace":"default", "projectcalico.org/orchestrator":"k8s"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"ip-10-0-83-184", ContainerID:"", Pod:"nginx", Endpoint:"eth0", IPNetworks:[]string{"192.168.142.195/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"kns.default"}, InterfaceName:"", MAC:"", Ports:[]v3.EndpointPort(nil)}}
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: Calico CNI using IPs: [192.168.142.195/32]
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.254 [INFO][19037] network.go 31: Setting the host side veth name to calic440f455693 ContainerID="k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" Namespace="default" Pod="nginx" WorkloadEndpoint="ip--10--0--83--184-k8s-nginx-eth0"
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.263 [INFO][19037] k8s.go 303: Added Mac, interface name, and active container ID to endpoint ContainerID="k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" Namespace="default" Pod="nginx" WorkloadEndpoint="ip--10--0--83--184-k8s-nginx-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"ip--10--0--83--184-k8s-nginx-eth0", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"projectcalico.org/namespace":"default", "projectcalico.org/orchestrator":"k8s"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"ip-10-0-83-184", ContainerID:"k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173", Pod:"nginx", Endpoint:"eth0", IPNetworks:[]string{"192.168.142.195/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"kns.default"}, InterfaceName:"calic440f455693", MAC:"e2:ed:d6:21:06:6d", Ports:[]v3.EndpointPort(nil)}}
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.264 [ERROR][19037] k8s.go 307: Error creating/updating endpoint in datastore. ContainerID="k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" Namespace="default" Pod="nginx" WorkloadEndpoint="ip--10--0--83--184-k8s-nginx-eth0" error=error with field containerID = 'k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173'
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.264 [INFO][19037] k8s.go 281: Releasing IPAM allocation after failure ContainerID="k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" IPs=[]string{"192.168.142.195/32"} Namespace="default" Pod="nginx" WorkloadEndpoint="ip--10--0--83--184-k8s-nginx-eth0"
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.264 [INFO][19037] utils.go 346: Cleaning up IP allocations for failed ADD ContainerID="k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" IPs=[]string{"192.168.142.195/32"} Namespace="default" Pod="nginx" WorkloadEndpoint="ip--10--0--83--184-k8s-nginx-eth0"
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.328 [INFO][19050] utils.go 327: Configured environment: [CNI_COMMAND=DEL CNI_CONTAINERID=k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173 CNI_NETNS=/var/run/netns/cni-76bf6400-2f69-64ad-77b1-8cd337b88a4b CNI_ARGS=IgnoreUnknown=1;IgnoreUnknown=1;K8S_POD_NAME=nginx;K8S_POD_NAMESPACE=default;K8S_POD_INFRA_CONTAINER_ID=k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173 CNI_IFNAME=eth0 CNI_PATH=/opt/cni/bin:/opt/calico/bin PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ETCD_ENDPOINTS=http://10.96.232.136:6666 KUBECONFIG=/etc/cni/net.d/calico-kubeconfig K8S_API_TOKEN=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjYWxpY28tY25pLXBsdWdpbi10b2tlbi1zdDd3cyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjYWxpY28tY25pLXBsdWdpbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjAxNmUyOGMwLTA4MjQtMTFlOC1iM2M0LTBlYTU3MzFmMmY1ZSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTpjYWxpY28tY25pLXBsdWdpbiJ9.AC5lyPL1v-y6_XHBQyAdMr4OOJGDCEWuxfA4NhO_GvbqbwVpYC5ekj8meYIsh1bjEgSdVOU5fN8AQYpGA1qR8JCMPhifKUbIzvlcJVNoBbscDEpvsXjKEhv-Aev_LQ1yrCClczSXg77xwhwXcz5vaQx_gQd5u-BIYtp2V0ZqT3aQJL1WOMF-VZdbOuGST1hMOWsj6uGnSw2gZJmmbRYvS3idfTsKNgUNVm5XB2yfDxmQE7DCHWcXf-KMW8cJ7n6pHocu22IBwngj3aNUfRqcVbsvmryTxOB5tuiir6HmDu8S7KVQlL_DdxMCF9CumNPMkB28Kur0vqDLKyFfmCGSOw]
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.329 [INFO][19050] calico-ipam.go 258: Releasing address using handleID ContainerID="k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" HandleID="k8s-pod-network.k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" Workload="ip--10--0--83--184-k8s-nginx-eth0"
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.329 [INFO][19050] ipam.go 676: Releasing all IPs with handle 'k8s-pod-network.k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173'
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.332 [INFO][19050] ipam.go 813: Decremented handle 'k8s-pod-network.k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173' by 1
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.332 [INFO][19050] calico-ipam.go 267: Released address using handleID ContainerID="k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" HandleID="k8s-pod-network.k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" Workload="ip--10--0--83--184-k8s-nginx-eth0"
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.332 [INFO][19050] calico-ipam.go 276: Releasing address using workloadID ContainerID="k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" HandleID="k8s-pod-network.k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173" Workload="ip--10--0--83--184-k8s-nginx-eth0"
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: 2018-02-02 14:21:43.332 [INFO][19050] ipam.go 676: Releasing all IPs with handle 'default.nginx'
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: E0202 14:21:43.334799   16160 ocicni.go:197] Pod: k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173, Netns: /var/run/netns/cni-76bf6400-2f69-64ad-77b1-8cd337b88a4b, Error adding network: error with field containerID = 'k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: E0202 14:21:43.334828   16160 ocicni.go:170] Error while adding to cni network: error with field containerID = 'k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173'
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: E0202 14:21:43.334895   16160 sandbox.go:87] Setup network for sandbox "&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:nginx,Uid:6430a93d-0824-11e8-b3c4-0ea5731f2f5e,Namespace:default,Attempt:0,},Hostname:nginx,LogDirectory:/var/log/pods/6430a93d-0824-11e8-b3c4-0ea5731f2f5e,DnsConfig:&DNSConfig{Servers:[10.96.0.10],Searches:[default.svc.cluster.local svc.cluster.local cluster.local ec2.internal],Options:[ndots:5],},PortMappings:[&PortMapping{Protocol:TCP,ContainerPort:80,HostPort:0,HostIp:,}],Labels:map[string]string{NETNS: /var/run/netns/cni-76bf6400-2f69-64ad-77b1-8cd337b88a4b,io.kubernetes.frakti.annotations: {\"kubernetes.io/config.seen\":\"2018-02-02T14:21:42.763310438Z\",\"kubernetes.io/config.source\":\"api\"},io.kubernetes.pod.name: nginx,io.kubernetes.pod.namespace: default,io.kubernetes.pod.uid: 6430a93d-0824-11e8-b3c4-0ea5731f2f5e,},Annotations:map[string]string{kubernetes.io/config.seen: 2018-02-02T14:21:42.763310438Z,kubernetes.io/config.source: api,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/besteffort/pod6430a93d-0824-11e8-b3c4-0ea5731f2f5e,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{HostNetwork:false,HostPid:false,HostIpc:false,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:false,SeccompProfilePath:,},Sysctls:map[string]string{},},}" by cni plugin failed: error with field containerID = 'k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173'
Feb 02 14:21:43 ip-10-0-83-184 frakti[16160]: E0202 14:21:43.347186   16160 manager.go:214] RunPodSandbox from hyper runtime service failed: error with field containerID = 'k8s_POD.0_nginx_default_6430a93d-0824-11e8-b3c4-0ea5731f2f5e_04273173'

Note: When testing with v2.4 the problem doesn't happen.

I've tracked down the source of error and found the error comes from this line:
https://github.com/projectcalico/cni-plugin/blob/fc96bde550232f5fddb13109b844e63ae8c0377e/k8s/k8s.go#L306

I've tested with a Ubuntu 16.04 LTS machine on AWS.

Answer 1 · 2018-02-03T18:42:07.000Z

@liwei We may need to check cni lib, which seems unmatch with latest Calico.

Answer 2 · 2018-02-04T05:40:17.000Z

it seems calico considered our crafted containerID is not valid:

https://github.com/projectcalico/libcalico-go/blob/d2fe70a7fc26fff6b08f2f91214cd129ec12426e/lib/validator/v3/validator.go#L239

Answer 3 · 2018-09-03T03:23:51.000Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Answer 4 · 2018-10-03T03:46:03.000Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

Answer 5 · 2018-11-02T04:33:33.000Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Answer 6 · 2018-11-02T04:33:40.000Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.