I've deployed it on our cluster, it founds volumes but isn't deleting
cusspvz opened this issue · 18 comments
log:
18/11/2015 22:34:38�[34mINFO�[0m[26790] Waking up to check for volumes...
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: 1a39a9441c0b125844847711d3481571b7bb68b1ba1b078f845e5992367c7ccf
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: 32fbc1ab1c4cb66a5dd8641a8fc4903ae4530fe4cfadcf790e1e3e28de6e1b0d
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: 3335ef272ed6d51176e5819cb4356910548cc444fad817544e38fb26f6422991
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: 3f821e4047f7ed0cde28f5a722a1f237930a9014cd9aa116a3267bdf10363e15
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: 4303f0160842d608f3ebbb324bb77e36611dc955a9d25238ed0f4b619284b047
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: 46064c2fa0ba0d39f3bdc6a3c2093c14f8eb035194f18a2d5be158111447d052
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: 4be446e60f7964791d7dfd57b67ce1f8c8361dca5521eef4587f6119b52584cd
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: 5ff11243c18a1e17ddc06268ff4c205a9d52827f7e60d65fe755223ca73caf4d
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: 7599e33a8e316f0e1a34d9b0005d030c5ad2848e1653640f6e306de169fe1567
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: 7d8f61566feb8b5e2bf22082c0993d147fbb4e8ab0db8777c1594b95414b8561
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: 822570e38b3c21eaf48206ebb475383f2fecd5828cdb05a201b257713ff5e7d6
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: 919bc1b3b1f3b82f1e6ee5e5d17a334bdce9d9aa727a351ca1180455f5d6367c
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: 95c9762e86d42d4433c3d8f4990da497ea038728fee36eaba50035f1c6a8434c
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: a330d3f814c2a019569bc25d358cb5f370b14b0a62eab15f67a569db3fa8aae3
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: bb930a6ae80420932427250cf285b0b244437681d3be50f5e275720574122dc4
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: bddede3996c52b1d6e6342ae8f90dc29cf0abd342381fb441a0f96360914b65a
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: d22b02625e2c3a3e4899a72cef172e29064dc17ad2065a793ee84d6c9755470b
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: d7e69e6f0e500678e1c8005713f5cc5530ec03fe70f5196ce2d7496e71871306
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: e235349627af3317748254e5d858627ed4a1aed10e5d6745ed1f4f1a942afa1b
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: e295b9459f81b507200def18d9b0cfd2a4cd49234c2e1e2c14158cacfefa03aa
18/11/2015 22:34:39�[34mINFO�[0m[26791] Found volume: f515511f1294ba4a787d0a0effb2e479d135a383ae582fbeab2eaa9a07a69f50
18/11/2015 22:39:39�[34mINFO�[0m[27091] Waking up to check for volumes...
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: 1a39a9441c0b125844847711d3481571b7bb68b1ba1b078f845e5992367c7ccf
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: 32fbc1ab1c4cb66a5dd8641a8fc4903ae4530fe4cfadcf790e1e3e28de6e1b0d
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: 3335ef272ed6d51176e5819cb4356910548cc444fad817544e38fb26f6422991
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: 3f821e4047f7ed0cde28f5a722a1f237930a9014cd9aa116a3267bdf10363e15
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: 4303f0160842d608f3ebbb324bb77e36611dc955a9d25238ed0f4b619284b047
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: 46064c2fa0ba0d39f3bdc6a3c2093c14f8eb035194f18a2d5be158111447d052
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: 4be446e60f7964791d7dfd57b67ce1f8c8361dca5521eef4587f6119b52584cd
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: 5ff11243c18a1e17ddc06268ff4c205a9d52827f7e60d65fe755223ca73caf4d
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: 7599e33a8e316f0e1a34d9b0005d030c5ad2848e1653640f6e306de169fe1567
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: 7d8f61566feb8b5e2bf22082c0993d147fbb4e8ab0db8777c1594b95414b8561
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: 822570e38b3c21eaf48206ebb475383f2fecd5828cdb05a201b257713ff5e7d6
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: 919bc1b3b1f3b82f1e6ee5e5d17a334bdce9d9aa727a351ca1180455f5d6367c
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: 95c9762e86d42d4433c3d8f4990da497ea038728fee36eaba50035f1c6a8434c
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: a330d3f814c2a019569bc25d358cb5f370b14b0a62eab15f67a569db3fa8aae3
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: bb930a6ae80420932427250cf285b0b244437681d3be50f5e275720574122dc4
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: bddede3996c52b1d6e6342ae8f90dc29cf0abd342381fb441a0f96360914b65a
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: d22b02625e2c3a3e4899a72cef172e29064dc17ad2065a793ee84d6c9755470b
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: d7e69e6f0e500678e1c8005713f5cc5530ec03fe70f5196ce2d7496e71871306
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: e235349627af3317748254e5d858627ed4a1aed10e5d6745ed1f4f1a942afa1b
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: e295b9459f81b507200def18d9b0cfd2a4cd49234c2e1e2c14158cacfefa03aa
18/11/2015 22:39:39�[34mINFO�[0m[27091] Found volume: f515511f1294ba4a787d0a0effb2e479d135a383ae582fbeab2eaa9a07a69f50
18/11/2015 22:44:40�[34mINFO�[0m[27392] Waking up to check for volumes...
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: 1a39a9441c0b125844847711d3481571b7bb68b1ba1b078f845e5992367c7ccf
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: 32fbc1ab1c4cb66a5dd8641a8fc4903ae4530fe4cfadcf790e1e3e28de6e1b0d
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: 3335ef272ed6d51176e5819cb4356910548cc444fad817544e38fb26f6422991
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: 3f821e4047f7ed0cde28f5a722a1f237930a9014cd9aa116a3267bdf10363e15
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: 4303f0160842d608f3ebbb324bb77e36611dc955a9d25238ed0f4b619284b047
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: 46064c2fa0ba0d39f3bdc6a3c2093c14f8eb035194f18a2d5be158111447d052
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: 4be446e60f7964791d7dfd57b67ce1f8c8361dca5521eef4587f6119b52584cd
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: 5ff11243c18a1e17ddc06268ff4c205a9d52827f7e60d65fe755223ca73caf4d
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: 7599e33a8e316f0e1a34d9b0005d030c5ad2848e1653640f6e306de169fe1567
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: 7d8f61566feb8b5e2bf22082c0993d147fbb4e8ab0db8777c1594b95414b8561
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: 822570e38b3c21eaf48206ebb475383f2fecd5828cdb05a201b257713ff5e7d6
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: 919bc1b3b1f3b82f1e6ee5e5d17a334bdce9d9aa727a351ca1180455f5d6367c
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: 95c9762e86d42d4433c3d8f4990da497ea038728fee36eaba50035f1c6a8434c
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: a330d3f814c2a019569bc25d358cb5f370b14b0a62eab15f67a569db3fa8aae3
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: bb930a6ae80420932427250cf285b0b244437681d3be50f5e275720574122dc4
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: bddede3996c52b1d6e6342ae8f90dc29cf0abd342381fb441a0f96360914b65a
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: d22b02625e2c3a3e4899a72cef172e29064dc17ad2065a793ee84d6c9755470b
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: d7e69e6f0e500678e1c8005713f5cc5530ec03fe70f5196ce2d7496e71871306
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: e235349627af3317748254e5d858627ed4a1aed10e5d6745ed1f4f1a942afa1b
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: e295b9459f81b507200def18d9b0cfd2a4cd49234c2e1e2c14158cacfefa03aa
18/11/2015 22:44:40�[34mINFO�[0m[27392] Found volume: f515511f1294ba4a787d0a0effb2e479d135a383ae582fbeab2eaa9a07a69f50
18/11/2015 22:49:40�[34mINFO�[0m[27692] Waking up to check for volumes...
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: 1a39a9441c0b125844847711d3481571b7bb68b1ba1b078f845e5992367c7ccf
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: 32fbc1ab1c4cb66a5dd8641a8fc4903ae4530fe4cfadcf790e1e3e28de6e1b0d
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: 3335ef272ed6d51176e5819cb4356910548cc444fad817544e38fb26f6422991
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: 3f821e4047f7ed0cde28f5a722a1f237930a9014cd9aa116a3267bdf10363e15
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: 4303f0160842d608f3ebbb324bb77e36611dc955a9d25238ed0f4b619284b047
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: 46064c2fa0ba0d39f3bdc6a3c2093c14f8eb035194f18a2d5be158111447d052
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: 4be446e60f7964791d7dfd57b67ce1f8c8361dca5521eef4587f6119b52584cd
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: 5ff11243c18a1e17ddc06268ff4c205a9d52827f7e60d65fe755223ca73caf4d
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: 6fdb4bbe11a302791f23b064bcca555f47c4952fef942d4640c70edec4f03c0c
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: 7599e33a8e316f0e1a34d9b0005d030c5ad2848e1653640f6e306de169fe1567
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: 7d8f61566feb8b5e2bf22082c0993d147fbb4e8ab0db8777c1594b95414b8561
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: 822570e38b3c21eaf48206ebb475383f2fecd5828cdb05a201b257713ff5e7d6
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: 919bc1b3b1f3b82f1e6ee5e5d17a334bdce9d9aa727a351ca1180455f5d6367c
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: 95c9762e86d42d4433c3d8f4990da497ea038728fee36eaba50035f1c6a8434c
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: a330d3f814c2a019569bc25d358cb5f370b14b0a62eab15f67a569db3fa8aae3
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: bb930a6ae80420932427250cf285b0b244437681d3be50f5e275720574122dc4
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: bddede3996c52b1d6e6342ae8f90dc29cf0abd342381fb441a0f96360914b65a
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: d22b02625e2c3a3e4899a72cef172e29064dc17ad2065a793ee84d6c9755470b
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: d7e69e6f0e500678e1c8005713f5cc5530ec03fe70f5196ce2d7496e71871306
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: e235349627af3317748254e5d858627ed4a1aed10e5d6745ed1f4f1a942afa1b
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: e295b9459f81b507200def18d9b0cfd2a4cd49234c2e1e2c14158cacfefa03aa
18/11/2015 22:49:41�[34mINFO�[0m[27693] Found volume: f515511f1294ba4a787d0a0effb2e479d135a383ae582fbeab2eaa9a07a69f50
docker-compose.yml
vol-cleanup:
restart:
name: always
labels:
io.rancher.scheduler.global: 'true'
tty: true
command:
- -i
- '5'
image: cloudnautique/vol-cleanup
volumes:
- /var/lib/docker:/var/lib/docker
- /var/run/docker.sock:/var/run/docker.sock
stdin_open: true
Which version of Docker? I need to find some time to look at this with 1.9.x now.
The default behavior is to log volumes that it finds, but it only deletes volumes that are NOT connected to a container.
If you have running containers, do an inspect and see if you find one of those volumes in use.
The default behavior is to log volumes that it finds, but it only deletes volumes that are NOT connected to a container.
they're not, since we run lots of builds on those specific hosts, we have LOOOOTS of non-used volumes x)
(Off-topic: do you want me to push a scalable stack for gitlab on compose templates? 😄)
Which version of Docker? I need to find some time to look at this with 1.9.x now.
Docker 1.9.0, build 76d6bc9
On the off topic note, I'd love to see a scalable Gitlab template. How are you handling storage/etc? I'm not familiar with Gitlabs architecture at the moment. If you have something to look at, open a PR, we can work through it in the compose-templates repo or if you want to make it a full on catalog entry, that would be good too. We have someone working with us on Mongo DB, and it is one of our first PRs from outside. There is some back and forth that might be helpful outlining what we are looking for.
Related to this issue, in Docker 1.9 you can name volumes, which makes them a little less like orphans and more like pets. Thoughts on what the appropriate behavior is for handling them?
On another unrelated note, have you looked at cloudnautique/cowbell? It is a webhook listener to scale up services in a stack. I'm interested in thoughts/feedback.
How are you handling storage/etc?
btsync handles syncing between hosts and could interact between containers over volume or nfs sharing. For us is great since we just have to run another container on one of our physical hosts and it backups entire storage.
If you have something to look at, open a PR, we can work through it in the compose-templates repo or if you want to make it a full on catalog entry
Will do that, but I would prefer a full catalog entry hehehe :)
We have someone working with us on Mongo DB
Nice to hear! We will need that as well!
Related to this issue, in Docker 1.9 you can name volumes, which makes them a little less like orphans and more like pets. Thoughts on what the appropriate behavior is for handling them?
I know there were changes from 1.8 to 1.9 related to volumes but I'm not on that scope yet, we've been tooooo busy over here.
have you looked at cloudnautique/cowbell?
Saw it now, nice! I've been thinking on how rancher could scale without having human intervention, but it would give some work, things such as using lb and hosts metrics for scaling services, and the most complex, for having pluggable hosts (sort of EC2 auto scalling for those who aren't on AWS) which could be maintained by the system by using docker-machine for creating them or deleting them.
Just an update on this issue, it looks like in 1.9.1 it is actually deleting the volume folder, but docker volume ls still shows the docker volume ls command.
Will need to update the volume stuff to use top level volume command functionality.
Is that implementation hard? (didn't read your repo code yet)
I don't think so, will try and get to it this weekend.
That would be awesome @cloudnautique !
@fernandoneto this is the repo I was talking about, a few days ago, that does some cleanup on unused docker volumes. Do you mind to stay on charge of performing an one-shot upgrade in case @cloudnautique releases a new version?
@cusspvz ok. If @cloudnautique release a new version i make the upgrade.
@fernandoneto @cusspvz I built a v0.3.0 container, and put the fixes on master. (I haven't moved the latest docker tag yet) Would you mind helping to test it out? Perhaps deploy it to a few nodes to make sure it doesn't do anything crazy =D I'm going to roll it to our CI environment as well, but we are still on Docker 1.8.x.
@cloudnautique i will test and as soon as i have news i give you feedback, and thanks for your availability.
@cloudnautique I did some tests and it seems that everything is working properly. i've created and remove some stack's and all volumes associated were deleted.
this is a part of the logs
02/12/2015 16:38:51�[34mINFO�[0m[18601] Waking up to check for volumes...
02/12/2015 16:38:51�[34mINFO�[0m[18601] Found volume: be7d6ecd8a2cbe33a482936ad62dc40ecd697f72c7460e4668bfa45aa225d52e
02/12/2015 16:38:51�[34mINFO�[0m[18601] Found volume: fcf346a0352a59189b209dcee930f319fad5b3ee382c5b5c866bda02c1439661
02/12/2015 16:38:51�[34mINFO�[0m[18601] Found volume: ca27d4db8d7db27f71619a003c74655bc4e46d7bb79e430ceef6aa1c18c4536b
02/12/2015 16:38:51�[34mINFO�[0m[18601] Found volume: d564ed7e44fa775f0cfdffb24eade164b3b007af97388333df11991975a0d136
02/12/2015 16:38:51�[34mINFO�[0m[18601] Found volume: 22e45035f581a714e51a4cfcffb12d7ed16076ae12bbdd5bb4f2f6da789b3f30
02/12/2015 16:38:51�[34mINFO�[0m[18601] Found volume: 9544a7fd1bad6c47575b66ab58c3947a14547bd04440843938d69311cb4c895b
02/12/2015 16:38:51�[34mINFO�[0m[18601] Found volume: 62b7e65132f9696edcca286e95ad94024db3165e5980ca401476f0d7d8aa507c
02/12/2015 16:38:51�[34mINFO�[0m[18601] Found volume: 1d322fa0944dcf22411d3585ca33ec3c847c580fc2413d569cf74f8c4c46571e
02/12/2015 16:38:51�[34mINFO�[0m[18601] Found volume: 73154430190e8a6ead279a97d0529b732a2c01f279c657752d0dc36d8f8afd4c
02/12/2015 16:38:51�[34mINFO�[0m[18601] Found volume: 00e801d78a395735243314c33657ec162fab92d755728028a824f7c8d8bb5962
02/12/2015 16:38:51�[34mINFO�[0m[18601] Found volume: 825640efdc4d4f22f714b5496af10f17af958d30415ffa44304781d73eccda07
02/12/2015 16:38:51�[34mINFO�[0m[18601] Found volume: 33eae05604c0bb2cfbb9378c9bb822201bf7ffef4571345df2f98030c61cb268
02/12/2015 16:43:51�[34mINFO�[0m[18901] Waking up to check for volumes...
02/12/2015 16:43:51�[34mINFO�[0m[18901] Found volume: be7d6ecd8a2cbe33a482936ad62dc40ecd697f72c7460e4668bfa45aa225d52e
02/12/2015 16:43:51�[34mINFO�[0m[18901] Found volume: 22e45035f581a714e51a4cfcffb12d7ed16076ae12bbdd5bb4f2f6da789b3f30
02/12/2015 16:43:51�[34mINFO�[0m[18901] Found volume: 1d322fa0944dcf22411d3585ca33ec3c847c580fc2413d569cf74f8c4c46571e
02/12/2015 16:43:51�[34mINFO�[0m[18901] Found volume: 73154430190e8a6ead279a97d0529b732a2c01f279c657752d0dc36d8f8afd4c
02/12/2015 16:43:51�[34mINFO�[0m[18901] Found volume: 00e801d78a395735243314c33657ec162fab92d755728028a824f7c8d8bb5962
02/12/2015 16:48:51�[34mINFO�[0m[19201] Waking up to check for volumes...
02/12/2015 16:48:51�[34mINFO�[0m[19201] Found volume: 00e801d78a395735243314c33657ec162fab92d755728028a824f7c8d8bb5962
02/12/2015 16:48:51�[34mINFO�[0m[19201] Found volume: 1d322fa0944dcf22411d3585ca33ec3c847c580fc2413d569cf74f8c4c46571e
02/12/2015 16:48:51�[34mINFO�[0m[19201] Found volume: 73154430190e8a6ead279a97d0529b732a2c01f279c657752d0dc36d8f8afd4c
02/12/2015 16:48:51�[34mINFO�[0m[19201] Found volume: 22e45035f581a714e51a4cfcffb12d7ed16076ae12bbdd5bb4f2f6da789b3f30
02/12/2015 16:48:51�[34mINFO�[0m[19201] Found volume: be7d6ecd8a2cbe33a482936ad62dc40ecd697f72c7460e4668bfa45aa225d52eCool, thanks!
Based on the logs, I don't think this container did any of the deleting. Looks like Rancher might have done it for you, or docker rm -v was used? In Rancher environments typically systems that run docker natively are the culprits when there are orphaned volumes.
No i didn't use docker rm -v.
OK, I'll move this version to latest, and close out this issue.
Thanks for the help testing.
Just to confirm everything is working perfect with last version of docker
Great! thanks for following up @fernandoneto
@cloudnautique Thank you!!1 :)