openvstorage/alba

Decommissioning a backend/disk causes connections leaks

Opened this issue · 5 comments

kinvaris commented

On a certain environment I've decommissioning a backend that is still running but policy wise broken.
After the proxy removed the backend from list-osds we would expect all connections died with it.
But as seen here it isn't. (These results are for 12 ASDS)

root@ovs-node02-1604:~# lsof -p 21427 | grep 199.18 | wc -l
100

root@ovs-node02-1604:~# lsof -p 21427 | grep 199.18
alba    21427 root  108u     IPv4           27207601      0t0    TCP ovs-node02-1604:58040->10.100.199.181:26404 (ESTABLISHED)
alba    21427 root  111u     IPv4           27207603      0t0    TCP ovs-node02-1604:58044->10.100.199.181:26404 (ESTABLISHED)
alba    21427 root  114u     IPv4           27207602      0t0    TCP ovs-node02-1604:58042->10.100.199.181:26404 (ESTABLISHED)
alba    21427 root  122u     IPv4           27207614      0t0    TCP ovs-node02-1604:45634->10.100.199.182:8601 (ESTABLISHED)
alba    21427 root  123u     IPv4           27207615      0t0    TCP ovs-node02-1604:33738->10.100.199.182:8600 (ESTABLISHED)
alba    21427 root  124u     IPv4           27207616      0t0    TCP ovs-node02-1604:41708->10.100.199.182:8603 (ESTABLISHED)
alba    21427 root  125u     IPv4           27207617      0t0    TCP ovs-node02-1604:56814->10.100.199.182:8602 (ESTABLISHED)
alba    21427 root  126u     IPv4           27207618      0t0    TCP ovs-node02-1604:35958->10.100.199.183:8600 (ESTABLISHED)
wimpers commented

@JeffreyDevloo , @pploegaert Do we have a test that deletes backends so we can reproduce this?

JeffreyDevloo commented

Not to this extend. We currently just setup and alba backend, add a preset and then remove it.

Also this ticket is rather vague. What happened here for instance? "After the proxy removed the backend from list-osds" would imply a global backend having other backends as osds where one of the backends would be removed from it while the global backend is being used on a vpool (hence the proxy reference)
So a test for this case would be:

  • Setup 2 backends + presets + asds
  • Link 2 backends to a third, global backend
  • Add vpool to this backend
  • Remove a local backend from the global one (by unlinking or removing it fully <- unclear)
  • Check connection
wimpers commented

@JeffreyDevloo please reproduce the scenario first so we know exactly how we got into it.

domsj commented

We (pluralis majestatis) already know exactly how we got into it

wimpers commented

@domsj please explain. Is this still an issue?