felix/table.go 997: Failed to program iptables, will retry error=exit status 1 ipVersion=0x6 table="filter"
denghuancong opened this issue · 2 comments
denghuancong commented
we deploy a k8s cluster using alicloud ecs node , reference this page :https://github.com/AliyunContainerService/terway/blob/main/README-zh_CN.md
terway image version: terway:v1.2.3
cluster info
[root@harbor-vm ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true","reason":""}
node info
[root@harbor-vm ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
172.16.0.182 Ready control-plane 21m v1.24.2
172.16.0.183 Ready control-plane 15m v1.24.2
172.16.0.184 Ready control-plane 11m v1.24.2
172.16.0.187 Ready <none> 7m31s v1.24.2
check terway pod in node 172.16.0.187, it flash many warning
2022-07-14 10:18:23.273 [WARNING][14795] felix/table.go 997: Failed to program iptables, will retry error=exit status 1 ipVersion=0x6 table="raw"
2022-07-14 10:18:23.274 [WARNING][14795] felix/table.go 1000: Retrying... error=exit status 1 ipVersion=0x6 table="raw"
2022-07-14 10:18:23.283 [WARNING][14795] felix/table.go 1016: Succeeded after retry. ipVersion=0x4 table="filter"
2022-07-14 10:18:23.289 [WARNING][14795] felix/table.go 1016: Succeeded after retry. ipVersion=0x6 table="raw"
2022-07-14 10:18:23.289 [WARNING][14795] felix/table.go 1324: Failed to execute ip(6)tables-restore command error=exit status 1 errorOutput="ip6tables-nft-restore: line 42 failed\n" input="*filter\n:cali-from-endpoint-mark - -\n:cali-INPUT - -\n:cali-to-wl-dispatch - -\n:cali-cidr-block - -\n:cali-to-host-endpoint - -\n:cali-from-hep-forward - -\n:cali-forward-check - -\n:cali-from-host-endpoint - -\n:cali-forward-endpoint-mark - -\n:cali-FORWARD - -\n:cali-OUTPUT - -\n:cali-from-wl-dispatch - -\n:cali-to-hep-forward - -\n:cali-wl-to-host - -\n:cali-set-endpoint-mark - -\n-A cali-to-wl-dispatch -m comment --comment \"cali:7KNphB1nNHw80nIO\" -m comment --comment \"Unknown interface\" --jump DROP\n-A cali-forward-endpoint-mark -m comment --comment \"cali:O0SmFDrnm7KggWqW\" -m mark ! --mark 0x100000/0xfff00000 --jump cali-from-endpoint-mark\n-A cali-forward-endpoint-mark -m comment --comment \"cali:aFl0WFKRxDqj8oA6\" --out-interface cali+ --jump cali-to-wl-dispatch\n-A cali-forward-endpoint-mark -m comment --comment \"cali:AZKVrO3i_8cLai5f\" --jump cali-to-hep-forward\n-A cali-forward-endpoint-mark -m comment --comment \"cali:96HaP1sFtb-NYoYA\" --jump MARK --set-mark 0/0xfff00000\n-A cali-forward-endpoint-mark -m comment --comment \"cali:VxO6hyNWz62YEtul\" -m comment --comment \"Policy explicitly accepted packet.\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-A cali-FORWARD -m comment --comment \"cali:vjrMJCRpqwy5oRoX\" --jump MARK --set-mark 0/0xe0000\n-A cali-FORWARD -m comment --comment \"cali:A_sPAO0mcxbT9mOV\" -m mark --mark 0/0x10000 --jump cali-from-hep-forward\n-A cali-FORWARD -m comment --comment \"cali:8ZoYfO5HKXWbB3pk\" --in-interface cali+ --jump cali-from-wl-dispatch\n-A cali-FORWARD -m comment --comment \"cali:jdEuaPBe14V2hutn\" --out-interface cali+ --jump cali-to-wl-dispatch\n-A cali-FORWARD -m comment --comment \"cali:12bc6HljsMKsmfr-\" --jump cali-to-hep-forward\n-A cali-FORWARD -m comment --comment \"cali:NOSxoaGx8OIstr1z\" --jump cali-cidr-block\n-A cali-OUTPUT -m comment --comment \"cali:Mq1_rAdXXH3YkrzW\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-A cali-OUTPUT -m comment --comment \"cali:5Z67OUUpTOM7Xa1a\" -m mark ! --mark 0/0xfff00000 --goto cali-forward-endpoint-mark\n-A cali-OUTPUT -m comment --comment \"cali:M2Wf0OehNdig8MHR\" --out-interface cali+ --jump RETURN\n-A cali-OUTPUT -m comment --comment \"cali:qO3aVIhjZ5EawFCC\" --jump MARK --set-mark 0/0xf0000\n-A cali-OUTPUT -m comment --comment \"cali:f8FynPuJ4oqVC2fm\" -m conntrack ! --ctstate DNAT --jump cali-to-host-endpoint\n-A cali-OUTPUT -m comment --comment \"cali:XrSZqRHtp-7rL_2w\" -m comment --comment \"Host endpoint policy accepted packet.\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-A cali-forward-check -m comment --comment \"cali:Pbldlb4FaULvpdD8\" -m conntrack --ctstate RELATED,ESTABLISHED --jump RETURN\n-A cali-forward-check -m comment --comment \"cali:Gxs1CMJwgvM0AlFN\" -m comment --comment \"To kubernetes NodePort service\" -p tcp -m multiport --destination-ports 30000:32767 -m set --match-set cali60this-host dst --goto cali-set-endpoint-mark\n-A cali-forward-check -m comment --comment \"cali:dve-xa1EDg-t6Iw9\" -m comment --comment \"To kubernetes NodePort service\" -p udp -m multiport --destination-ports 30000:32767 -m set --match-set cali60this-host dst --goto cali-set-endpoint-mark\n-A cali-forward-check -m comment --comment \"cali:s5OAa1zjNk2pPPHq\" -m comment --comment \"To kubernetes service\" -m set ! --match-set cali60this-host dst --jump cali-set-endpoint-mark\n-A cali-set-endpoint-mark -m comment --comment \"cali:MN61lcxFj1yWuYBo\" -m comment --comment \"Unknown endpoint\" --in-interface cali+ --jump DROP\n-A cali-set-endpoint-mark -m comment --comment \"cali:nKOjq8N2yzfmS3jk\" -m comment --comment \"Non-Cali endpoint mark\" --jump MARK --set-mark 0x100000/0xfff00000\n-A cali-from-wl-dispatch -m comment --comment \"cali:zTj6P0TIgYvgz-md\" -m comment --comment \"Unknown interface\" --jump DROP\n-A cali-wl-to-host -m comment --comment \"cali:TYeA_BqDrPHaAt6E\" -p 58 -m icmp6 --icmpv6-type 130 --jump ACCEPT\n-A cali-wl-to-host -m comment --comment \"cali:5ugan8LfmJg_BiJc\" -p 58 -m icmp6 --icmpv6-type 131 --jump ACCEPT\n-A cali-wl-to-host -m comment --comment \"cali:Fl5LHxdlOnUNgCc4\" -p 58 -m icmp6 --icmpv6-type 132 --jump ACCEPT\n-A cali-wl-to-host -m comment --comment \"cali:tNvzCkGVISJ3ZXdS\" -p 58 -m icmp6 --icmpv6-type 133 --jump ACCEPT\n-A cali-wl-to-host -m comment --comment \"cali:86e1wB5w3SEOMrZb\" -p 58 -m icmp6 --icmpv6-type 135 --jump ACCEPT\n-A cali-wl-to-host -m comment --comment \"cali:kCq3XXx0yCb5mSXt\" -p 58 -m icmp6 --icmpv6-type 136 --jump ACCEPT\n-A cali-wl-to-host -m comment --comment \"cali:qQJuyC_KUUNb16sA\" --jump cali-from-wl-dispatch\n-A cali-wl-to-host -m comment --comment \"cali:TEAa8oLTO9cQ8kDr\" -m comment --comment \"Configured DefaultEndpointToHostAction\" --jump ACCEPT\n-A cali-from-endpoint-mark -m comment --comment \"cali:9dpftzl-pNycbr37\" -m comment --comment \"Unknown interface\" --jump DROP\n-A cali-INPUT -m comment --comment \"cali:d4znnv6_6rx6sE6M\" --jump MARK --set-mark 0/0xfff00000\n-A cali-INPUT -m comment --comment \"cali:YHXh2XvaasL3jbTp\" --jump cali-forward-check\n-A cali-INPUT -m comment --comment \"cali:eL3eAQBTXQrID5PB\" -m mark ! --mark 0/0xfff00000 --jump RETURN\n-A cali-INPUT -m comment --comment \"cali:hwvMPJWpIRFo77b4\" --in-interface cali+ --goto cali-wl-to-host\n-A cali-INPUT -m comment --comment \"cali:c3dtuPGUL9TVsB6Y\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-A cali-INPUT -m comment --comment \"cali:czgL26xl8reOnh13\" --jump MARK --set-mark 0/0xf0000\n-A cali-INPUT -m comment --comment \"cali:EylNwA1nPRRCgK9T\" --jump cali-from-host-endpoint\n-A cali-INPUT -m comment --comment \"cali:JEbIi4mUTjL17qKC\" -m comment --comment \"Host endpoint policy accepted packet.\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-I FORWARD -m comment --comment \"cali:wUHhoiAYhphO9Mso\" --jump cali-FORWARD\n-A FORWARD -m comment --comment \"cali:S93hcgKJrXEqnTfs\" -m comment --comment \"Policy explicitly accepted packet.\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-A FORWARD -m comment --comment \"cali:mp77cMpurHhyjLrM\" --jump MARK --set-mark 0x10000/0x10000\n-I OUTPUT -m comment --comment \"cali:tVnHkvAo15HuiPy0\" --jump cali-OUTPUT\n-I INPUT -m comment --comment \"cali:Cz_u1IQiXIMmKD4c\" --jump cali-INPUT\nCOMMIT\n" ipVersion=0x6 output="" table="filter"
and pod don't get the ip from alicloud vpc/vswitch
[root@harbor-vm ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-5757d955b4-kjb76 0/1 ContainerCreating 0 2m52s <none> 172.16.0.187 <none> <none>
nginx-deployment-5757d955b4-t482v 0/1 ContainerCreating 0 2m52s <none> 172.16.0.187 <none> <none>
terway config mao info
kind: ConfigMap
apiVersion: v1
metadata:
name: eni-config
namespace: kube-system
data:
eni_conf: |
{
"version": "1",
"access_key": "ak",
"access_secret": "as",
"security_group": "sg-xx",
"service_cidr": "192.168.0.0/16",
"vswitches": {
"eu-central-1a": ["vsw-xx"]
},
"max_pool_size": 5,
"min_pool_size": 0
}
10-terway.conf: |
{
"cniVersion": "0.3.1",
"name": "terway",
"type": "terway",
"eniip_virtual_type": "IPVlan",
"ip_stack": "ipv4"
}
# eniip_virtual_type: virtual type for eni multi ip "Veth" || "IPVlan"
disable_network_policy: "false"
denghuancong commented
│ level=info msg=GitCommit 14641f3a0d549c13b0b59e230ea73971fce5fa49 BuildDate 2022-06-28T02:31:58Z Platform linux/amd64
│ W0715 05:59:05.557368 506018 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
│ level=info msg=got config: &{Version:1 AccessID:xxx AccessSecret:xxxCredentialPath: ServiceCIDR:192.168.0.0/16 VSwitches:map[eu-central-1a:[vsw-xx]] ENITags:map[] Max
│ PoolSize:5 MinPoolSize:0 MinENI:0 MaxENI:0 Prefix: SecurityGroup:sg-xx SecurityGroups:[] EniCapRatio:0 EniCapShift:0 VSwitchSelectionPolicy: EnableEIPPool: IPStack: AllowEIPRob: EnableENITrunking:false CustomStateful
│ WorkloadKinds:[] IPAMType: ENICapPolicy: BackoffOverride:map[] ExtraRoutes:[]} from: /etc/eni/eni.json subSys=network-service
│ level=info msg=instance metadata instance-id=i-gw85sex2ooqidjm8ef0e instance-type=ecs.c7.large primary-mac=00:16:3e:01:3e:06 region-id=eu-central-1 vpc-id=vpc-gw8up1gm0vwelesmzm47g vswitch-id=vsw-gw81uhte2fpqnspz8gjxr zone-id=eu-centr
│ al-1a
│ level=info msg=using AKPairProvider provider subSys=clientMgr
│ level=info msg=credential update expireAt=2122-07-15 05:59:05.600945812 +0000 UTC subSys=clientMgr updateAt=2022-07-15 05:59:05.601025163 +0000 UTC m=+0.053751036
│ level=fatal msg=error get instance type Post "https://ecs-vpc.eu-central-1.aliyuncs.com/?AccessKeyId=xxx&Action=DescribeInstanceTypes&Format=JSON&InstanceTypes.1=ecs.c7.large&RegionId=eu-central-1&Signature=x
│ xx%3D&SignatureMethod=HMAC-SHA1&SignatureNonce=xxx&SignatureType=&SignatureVersion=1.0&Timestamp=2022-07-15T06%3A00%3A03Z&Version=2014-05-26": dial tcp: lookup ecs-vpc.eu-central-1.aliyu
│ ncs.com: i/o timeout,timed out waiting for the condition
l1b0k commented
Your ECS have connection problem to openAPI.
Make sure you have SecurityGroup Egress direction set to allow , and valid DNS config on node