Deletion of Mapping does not actually delete it
Opened this issue · 3 comments
cn-mayank commented
I created a mapping, did a telnet to the service and used escape character to quit the telnet connection.
After that when i tried to delete the mapping, the kubectl returned message that successfully deleted but I could still get/describe the mapping in next kubectl command.
Logs on smartNat service showed following error message:
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=debug msg="Starting
to delete system configuration" name=mapping-sample9 namespace=devspaces-deploy-dev type=Mapping
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=debug msg="Starting
to delete DNAT rules" name=mapping-sample9 namespace=devspaces-deploy-dev type=Mapping
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=debug msg="Starting
to delete per-service DNAT with chain name MAP-WQVIHMDAZHKRBDJBYMBE3KFU" name=mapping-sample9 namespace=devspaces-
deploy-dev type=Mapping
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=debug msg="Flushing
chain MAP-WQVIHMDAZHKRBDJBYMBE3KFU" name=mapping-sample9 namespace=devspaces-deploy-dev type=Mapping
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=debug msg="Executin
g iptables in table nat and chain MAP-WQVIHMDAZHKRBDJBYMBE3KFU; rule option -F, selector: , action: , comment: "
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=debug msg="Destroyi
ng ipset DNAT-WQVIHMDAZHKRBDJBYMBE3KFU"
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=debug msg="Error re
moving ipset DNAT-WQVIHMDAZHKRBDJBYMBE3KFU: exit status 1, stdErr: ipset v6.30: Set cannot be destroyed: it is in u
se by a kernel component\n"
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=error msg="Error re
moving ipset DNAT-WQVIHMDAZHKRBDJBYMBE3KFU: exit status 1" name=mapping-sample9 namespace=devspaces-deploy-dev type
=Mapping
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=error msg="Error de
leting DNAT: exit status 1" name=mapping-sample9 namespace=devspaces-deploy-dev type=Mapping
Jun 10 14:22:05 aurea-dvsp-smartnat-5 smartnat-manager[3614]: time="2019-06-10T14:22:05Z" level=warning msg="Failed
to cleanup mapping from the operating system: &{%!c(*os.ProcessState=&{8187 256 0xc4202fc990}) [i p s e t v 6 .
3 0 : S e t c a n n o t b e d e s t r o y e d : i t i s i n u s e b y a k e r n e l c o m p
o n e n t \n]}" name=mapping-sample9 namespace=devspaces-deploy-dev type=Mapping
cn-mayank commented
output of some iptables command
$ sudo iptables -t nat -vnL
Chain PREROUTING (policy ACCEPT 2 packets, 76 bytes)
pkts bytes target prot opt in out source destination
88298 4795K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
88328 4801K SNM-PREROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* "for SNM" */
Chain INPUT (policy ACCEPT 2 packets, 76 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
41809 2541K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
41825 2541K KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */
41861 2544K KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0
41860 2544K SNM-POSTROUTING-MASQ all -- * * 0.0.0.0/0 0.0.0.0/0 /* "for SNM" */
Chain KUBE-FIREWALL (0 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-LOAD-BALANCER (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* Kubernetes service load balancer ip + port with externalTrafficPolicy=local */ match-set KUBE-LOAD-BALANCER-LOCAL dst,dst
0 0 KUBE-MARK-MASQ all -- * * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-MARK-DROP (1 references)
pkts bytes target prot opt in out source destination
Chain KUBE-MARK-MASQ (3 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000
Chain KUBE-NODE-PORT (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* Kubernetes nodeport TCP port with externalTrafficPolicy=local */ match-set KUBE-NODE-PORT-LOCAL-TCP dst
0 0 KUBE-MARK-MASQ all -- * * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-POSTROUTING (2 references)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose */ match-set KUBE-LOOP-BACK dst,dst,src
Chain KUBE-SERVICES (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-LOAD-BALANCER all -- * * 0.0.0.0/0 0.0.0.0/0 /* Kubernetes service lb portal */ match-set KUBE-LOAD-BALANCER dst,dst
0 0 KUBE-NODE-PORT all -- * * 0.0.0.0/0 0.0.0.0/0 /* Kubernetes nodeport TCP port for masquerade purpose */ match-set KUBE-NODE-PORT-TCP dst
0 0 KUBE-MARK-MASQ all -- * * 0.0.0.0/0 0.0.0.0/0 /* Kubernetes service cluster ip + port for masquerade purpose */ match-set KUBE-CLUSTER-IP dst,dst
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 match-set KUBE-CLUSTER-IP dst,dst
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 match-set KUBE-LOAD-BALANCER dst,dst
Chain MAP-KJI62Y2IAW63Q3D6QBAYI3AV (1 references)
pkts bytes target prot opt in out source destination
Chain MAP-LO7TEVUCNQ3B7Q6Q3CQ5RG4B (1 references)
pkts bytes target prot opt in out source destination
Chain MAP-OSDFTRYVHZ7TSEKNWVJUQZZO (1 references)
pkts bytes target prot opt in out source destination
Chain MAP-VMEYKBHWZV2RLE7XCT25UIPS (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* "mark for masquerade in SNM-POSTROUTING-MASQ" */ MARK or 0x100000
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:9097 /* "for mapping devspaces-deploy-dev/mapping-sample10 [tcp:9097:80]" */ to:10.19.249.9:80
Chain MAP-WQVIHMDAZHKRBDJBYMBE3KFU (1 references)
pkts bytes target prot opt in out source destination
Chain SNM-POSTROUTING-MASQ (1 references)
pkts bytes target prot opt in out source destination
16 840 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x100000/0x100000 /* "masquerade traffic marked in PREROUTING rules as destined for services" */
Chain SNM-PREROUTING (1 references)
pkts bytes target prot opt in out source destination
10 532 MAP-KJI62Y2IAW63Q3D6QBAYI3AV all -- * * 0.0.0.0/0 10.128.0.7 match-set DNAT-KJI62Y2IAW63Q3D6QBAYI3AV src,dst /* "for mapping devspaces-deploy-dev/mapping-sample7" */
8 396 MAP-OSDFTRYVHZ7TSEKNWVJUQZZO all -- * * 0.0.0.0/0 10.128.0.7 match-set DNAT-OSDFTRYVHZ7TSEKNWVJUQZZO src,dst /* "for mapping devspaces-deploy-dev/mapping-sample8" */
0 0 MAP-LO7TEVUCNQ3B7Q6Q3CQ5RG4B all -- * * 0.0.0.0/0 10.128.0.7 match-set DNAT-LO7TEVUCNQ3B7Q6Q3CQ5RG4B src,dst /* "for mapping devspace-dev/devspaces-8ckhdxyp-service-mp" */
1 64 MAP-WQVIHMDAZHKRBDJBYMBE3KFU all -- * * 0.0.0.0/0 10.128.0.7 match-set DNAT-WQVIHMDAZHKRBDJBYMBE3KFU src,dst /* "for mapping devspaces-deploy-dev/mapping-sample9" */
0 0 MAP-VMEYKBHWZV2RLE7XCT25UIPS all -- * * 0.0.0.0/0 10.128.0.7 match-set DNAT-VMEYKBHWZV2RLE7XCT25UIPS src,dst /* "for mapping devspaces-deploy-dev/mapping-sample10" */
cn-mayank commented
$ sudo iptables -t mangle -vnL
Chain PREROUTING (policy ACCEPT 7261 packets, 3729K bytes)
pkts bytes target prot opt in out source destination
4791K 2611M SNM-PREROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* "for SNM" */
Chain INPUT (policy ACCEPT 7261 packets, 3729K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 6920 packets, 488K bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 6920 packets, 488K bytes)
pkts bytes target prot opt in out source destination
Chain SNM-PREROUTING (1 references)
pkts bytes target prot opt in out source destination
51 5738 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 match-set KJI62Y2IAW63Q3D6QBAYI3AV src /* "for Maping devspaces-deploy-dev/mapping-sample7 [KJI62Y2IAW63Q3D6QBAYI3AV] mark 0x803" */ MARK or 0x803
36 4004 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 match-set OSDFTRYVHZ7TSEKNWVJUQZZO src /* "for Maping devspaces-deploy-dev/mapping-sample8 [OSDFTRYVHZ7TSEKNWVJUQZZO] mark 0x803" */ MARK or 0x803
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 match-set LO7TEVUCNQ3B7Q6Q3CQ5RG4B src /* "for Maping devspace-dev/devspaces-8ckhdxyp-service-mp [LO7TEVUCNQ3B7Q6Q3CQ5RG4B] mark 0x803" */ MARK or 0x803
3 164 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 match-set WQVIHMDAZHKRBDJBYMBE3KFU src /* "for Maping devspaces-deploy-dev/mapping-sample9 [WQVIHMDAZHKRBDJBYMBE3KFU] mark 0x803" */ MARK or 0x803
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 match-set VMEYKBHWZV2RLE7XCT25UIPS src /* "for Maping devspaces-deploy-dev/mapping-sample10 [VMEYKBHWZV2RLE7XCT25UIPS] mark 0x803" */ MARK or 0x803