seaweedfs/seaweedfs-csi-driver

Docker Swarm compatibility

pedroetb opened this issue · 30 comments

pedroetb commented

There is any way to make this CSI driver work for Docker Swarm volumes?

s4ke commented

Leaving my vote here as well. Docker 23.0.0 is finally out, and having a SeaweedFS CSI driver would be useful.

Link to the changelog:

https://github.com/moby/moby/releases/tag/v23.0.0
https://docs.docker.com/engine/release-notes/23.0/

olljanat commented

Yes Docker Swarm supports CSI interface starting on 23.0.0 but CSI drivers still need to be packages as Docker plugins. Official documentation about that process exists on https://github.com/moby/moby/blob/master/docs/cluster_volumes.md and I have build some example scripts and guidance to https://github.com/olljanat/csi-plugins-for-docker-swarm

chrislusf commented

Is there anything this repo can do to smooth the whole process? @olljanat @s4ke

olljanat commented

@chrislusf preferably someone who have existing SeaweedFS environment and understanding what are requirements of this CSI driver should:

  1. Take copy of one of those examples which I have shared (there is three now).
  2. Modify it scripts and config on way that it works with this CSI plugin.
  3. Eventually came as pull request here which adds those as part of new version release process.
s4ke commented

Judging by the fact that this driver already implements a monolith for both the controller and node server, this plugin should be quite easy to repackage. All we would have to do is repackage the image as a docker plugin, see the following example:

https://github.com/hetznercloud/csi-driver/tree/main/deploy/docker-swarm/pkg

  • 👍3
prologic commented

Has anyone tried to package this up yet for Docker?

s4ke commented

Sorry, currently busy with other things. If anyone needs guidance on how to do this, let me know. I am happy to help but just dont have the time to take care of all of it.

  • ❤️1
Gradlon commented

I want this plugin, so I guess I will do it then.
@s4ke Can you point me in the right direction to get started?

  • 🎉1
s4ke commented

For the Hetzner plugin I ported, you can look here: https://github.com/hetznercloud/csi-driver/blob/main/cmd/aio/main.go

Packaging the plugin is done here: https://github.com/hetznercloud/csi-driver/blob/main/deploy/docker-swarm/pkg/README.md

You can find more details here: https://github.com/olljanat/csi-plugins-for-docker-swarm

Also, feel free to reach out to me via discord over at devops.fan. I'd be happy to jump onto a call if you'd like as well.

Gradlon commented

@s4ke I gave it a go.
As far as I understand it, it should be straightforward.
For the moment I am stuck at the point of creating the Socket, somehow Docker can't connect to it.
The code can be found here
https://github.com/Gradlon/swarm-csi-swaweedfs
Any suggestions on what I am missing?

s4ke commented

Do you have some docker daemon logs for me? For me it was quite helpful to enable "debug": true on the docker daemon. Also, if you need to control the containerd container that is used for the driver, you should look into using the containerd ctr command. It has similar options as the docker command.

  • 👍1
Gradlon commented

I gave it another go.

I believe that tit fails on

Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="F0711 18:53:44.301927 server.go:61 invalid endpoint: /var/run/csi.sock" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102

But I could be wrong, not used to reading go Error's yet ;)

Jul 11 20:53:43 builder dockerd[4764]: time="2024-07-11T20:53:43.967351563+02:00" level=debug msg="Calling POST /v1.46/plugins/gradlon/swarm-csi-swaweedfs:v1.2.0/enable?timeout=30"
Jul 11 20:53:43 builder dockerd[4764]: time="2024-07-11T20:53:43.969385385+02:00" level=debug msg="bundle dir created" bundle=/run/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102 module=libcontainerd namespace=plugins.moby root=/var/lib/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102/rootfs
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.102223488+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/create
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.117858083+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/start
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138077 main.go:73 will run node: true, controller: true, attacher: true" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138293 main.go:79 connect to filer 10.0.1.1:8888" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138304 driver.go:53 Driver: seaweedfs-csi-driver version: 1.0.0" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138595 driver.go:127 Enabling volume access mode: MULTI_NODE_MULTI_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138603 driver.go:127 Enabling volume access mode: SINGLE_NODE_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138607 driver.go:127 Enabling volume access mode: SINGLE_NODE_MULTI_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138610 driver.go:127 Enabling volume access mode: SINGLE_NODE_SINGLE_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138614 driver.go:134 Enabling controller service capability: CREATE_DELETE_VOLUME" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138619 driver.go:134 Enabling controller service capability: SINGLE_NODE_MULTI_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138622 driver.go:134 Enabling controller service capability: EXPAND_VOLUME" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138626 driver.go:134 Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138636 driver.go:90 starting" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="F0711 18:53:44.138815 server.go:61 invalid endpoint: /var/run/csi.sock" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="goroutine 55 [running]:" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.stacks(0x0)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/seaweedfs@v0.0.0-20240622153018-82ef66cc194f/weed/glog/glog.go:768 +0x85" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).output(0x2905bc0, 0x3, 0xc0003fc540, {0x1fda19b?, 0x1?}, 0xc00005b530?, 0x0)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/seaweedfs@v0.0.0-20240622153018-82ef66cc194f/weed/glog/glog.go:719 +0x38a" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).printDepth(0x2905bc0, 0x3, 0x0?, {0xc00008bf38, 0x1, 0x1})" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/seaweedfs@v0.0.0-20240622153018-82ef66cc194f/weed/glog/glog.go:648 +0xea" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).print(...)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/seaweedfs@v0.0.0-20240622153018-82ef66cc194f/weed/glog/glog.go:639" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.Fatal(...)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/seaweedfs@v0.0.0-20240622153018-82ef66cc194f/weed/glog/glog.go:1134" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver.(*nonBlockingGRPCServer).serve(0xc000421518, {0x7fff74e29ef0?, 0x0?}, {0x1be7b28, 0xc0004b63e8}, {0x1bffe08, 0xc0004b63e0}, {0x1bfca78, 0xc00011fdd0})" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/src/github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver/server.go:61 +0xd4" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver.(*nonBlockingGRPCServer).Start.func1()" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/src/github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver/server.go:41 +0x6f" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="created by github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver.(*nonBlockingGRPCServer).Start in goroutine 1" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/src/github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver/server.go:39 +0xf3" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.149726373+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/exit
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.175998440+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/delete
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.176031563+02:00" level=info msg="ignoring event" container=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102 module=libcontainerd namespace=plugins.moby topic=/tasks/delete type="*events.TaskDelete"
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.197385259+02:00" level=debug msg="bundle dir created" bundle=/run/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102 module=libcontainerd namespace=plugins.moby root=/var/lib/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102/rootfs
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.262861122+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/create
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.283055724+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/start
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.300898 main.go:73 will run node: true, controller: true, attacher: true" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301158 main.go:79 connect to filer 10.0.1.1:8888" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301166 driver.go:53 Driver: seaweedfs-csi-driver version: 1.0.0" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301542 driver.go:127 Enabling volume access mode: MULTI_NODE_MULTI_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301555 driver.go:127 Enabling volume access mode: SINGLE_NODE_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301558 driver.go:127 Enabling volume access mode: SINGLE_NODE_MULTI_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301562 driver.go:127 Enabling volume access mode: SINGLE_NODE_SINGLE_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301581 driver.go:134 Enabling controller service capability: CREATE_DELETE_VOLUME" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301586 driver.go:134 Enabling controller service capability: SINGLE_NODE_MULTI_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301590 driver.go:134 Enabling controller service capability: EXPAND_VOLUME" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301594 driver.go:134 Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301601 driver.go:90 starting" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="F0711 18:53:44.301927 server.go:61 invalid endpoint: /var/run/csi.sock" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="goroutine 53 [running]:" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.stacks(0x0)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/seaweedfs@v0.0.0-20240622153018-82ef66cc194f/weed/glog/glog.go:768 +0x85" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).output(0x2905bc0, 0x3, 0xc00016c310, {0x1fda19b?, 0x1?}, 0xc000103760?, 0x0)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/seaweedfs@v0.0.0-20240622153018-82ef66cc194f/weed/glog/glog.go:719 +0x38a" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).printDepth(0x2905bc0, 0x3, 0x0?, {0xc000088f38, 0x1, 0x1})" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/seaweedfs@v0.0.0-20240622153018-82ef66cc194f/weed/glog/glog.go:648 +0xea" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).print(...)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/seaweedfs@v0.0.0-20240622153018-82ef66cc194f/weed/glog/glog.go:639" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.Fatal(...)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/seaweedfs@v0.0.0-20240622153018-82ef66cc194f/weed/glog/glog.go:1134" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver.(*nonBlockingGRPCServer).serve(0xc0001346a8, {0x7ffe5e1b0ef0?, 0x0?}, {0x1be7b28, 0xc00012a1c0}, {0x1bffe08, 0xc00012a1b8}, {0x1bfca78, 0xc0001d3da0})" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/src/github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver/server.go:61 +0xd4" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver.(*nonBlockingGRPCServer).Start.func1()" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/src/github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver/server.go:41 +0x6f" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="created by github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver.(*nonBlockingGRPCServer).Start in goroutine 1" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/src/github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver/server.go:39 +0xf3" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.314400485+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/exit
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.621472118+02:00" level=debug msg="error net dialing plugin: dial unix /run/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102/csi/csi.sock: connect: no such file or directory"
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.640165673+02:00" level=error msg="Sending SIGTERM to plugin failed with error: NotFound: process already finished: not found"
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.640338522+02:00" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!" error="dial unix /run/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102/csi/csi.sock: connect: no such file or directory" error_type="*net.OpError" module=api
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.640441142+02:00" level=error msg="Handler for POST /v1.46/plugins/gradlon/swarm-csi-swaweedfs:v1.2.0/enable returned error: dial unix /run/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102/csi/csi.sock: connect: no such file or directory"
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.786914522+02:00" level=debug msg="error net dialing plugin: dial unix /run/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102/csi/csi.sock: connect: no such file or directory"
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.801264468+02:00" level=error msg="Sending SIGTERM to plugin failed with error: NotFound: process already finished: not found"
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.827514808+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/delete
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.827621777+02:00" level=info msg="ignoring event" container=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102 module=libcontainerd namespace=plugins.moby topic=/tasks/delete type="*events.TaskDelete"
s4ke commented

Have you debugged the logs of the plugin with ctr already? could be that the socket is never created. Maybe the process is crashing?

Gradlon commented

Running

docker run -it --rm --privileged --pid=host -v /proc:/host/proc -v /var/run/docker/netns:/var/run/docker/netns -v /var/lib/docker/plugins/<ID>/rootfs:/plugin-root alpine chroot plugin-root sh

./seaweedfs-csi-driver -nodeid=$HOSTNAME -filer=10.0.1.1:8888

I can see that the process works in this context at least.
The socket is created at /tmp/seaweedfs-csi.sock

the problem is that docker is looking for the socket at /run/docker/plugins/
but there i only get the stderr/stdout

After some testing, I figured out that "-endpoint=unix://run/docker/plugins/seaweed.sock" make the socket available on the host.

s4ke commented

After some testing, I figured out that "-endpoint=unix://run/docker/plugins/seaweed.sock" make the socket available on the host.

Looking at https://github.com/hetznercloud/csi-driver/blob/b6cf8a8764dd2c05bc290dec68adf2ae5b2fcc69/deploy/docker-swarm/pkg/config.json#L16 this looks about right.

Have you already checked out how to debug the plugin using ctr?

Looks like you are unblocked, though, right?

Gradlon commented

No, did not understand how I should be doing this
Currently, I am trying to pass arguments to the CSI.
Socket connection now works, now I want to make it configurable.

Gradlon commented

Figured it out.

docker plugin install --disable --alias seaweedfs-csi:swarm --grant-all-permissions gradlon/swarm-csi-swaweedfs:v1.2.0
docker plugin set seaweedfs-csi:swarm FILER=IP1:8888,IP2:8888
docker plugin set seaweedfs-csi:swarm CACHE_SIZE=512
docker plugin enable seaweedfs-csi:swarm
docker volume create --driver gradlon/swarm-csi-swaweedfs:v1.2.0 --availability active --scope single --sharing none --type mount test-volume

This works to the point that I get a cluster Volume but it is stuck.

test-volume             seaweedfs-csi:swarm   active         pending creation
Gradlon commented

@chrislusf
@s4ke

I can't figure out why it is failing.
Any ideas?
Code is here
https://github.com/Gradlon/swarm-csi-swaweedfs

Jul 12 15:00:02 builder dockerd[840]: time="2024-07-12T15:00:02.384813636+02:00" level=debug msg="Calling POST /v1.45/volumes/create" spanID=0c2d0a4015fe5925 traceID=d17b16c9a9bcb320497e2c716d3a1dcd
Jul 12 15:00:02 builder dockerd[840]: time="2024-07-12T15:00:02.385015174+02:00" level=debug msg="form data: {\"ClusterVolumeSpec\":{\"AccessMode\":{\"MountVolume\":{},\"Scope\":\"single\",\"Sharing\":\"all\"},\"AccessibilityRequirements\":{},\"Availability\":\"active\",\"CapacityRange\":{\"LimitBytes\":0,\"RequiredBytes\":0}},\"Driver\":\"gradlon/swarm-csi-swaweedfs:v1.2.0\",\"DriverOpts\":{\"path\":\"/data\"},\"Name\":\"testVolume\"}" spanID=0c2d0a4015fe5925 traceID=d17b16c9a9bcb320497e2c716d3a1dcd
Jul 12 15:00:02 builder dockerd[840]: time="2024-07-12T15:00:02.385158055+02:00" level=debug msg="using cluster volume" spanID=0c2d0a4015fe5925 traceID=d17b16c9a9bcb320497e2c716d3a1dcd


Jul 12 15:00:26 builder dockerd[840]: time="2024-07-12T15:00:26.912339604+02:00" level=debug msg="2024/07/12 15:00:26 WARNING: [core] [Channel #12 SubChannel #13] grpc: addrConn.createTransport failed to connect to {Addr: \"/run/docker/plugins/f3306c1f2f0db70878a674d9073dd19b1d0109c7b6a70c7b648d9756a69757df/seaweed.sock\", ServerName: \"localhost\", Attributes: {\"<%!p(networktype.keyType=grpc.internal.transport.networktype)>\": \"unix\" }, }. Err: connection error: desc = \"transport: Error while dialing: dial unix /run/docker/plugins/f3306c1f2f0db70878a674d9073dd19b1d0109c7b6a70c7b648d9756a69757df/seaweed.sock: connect: no such file or directory\"" library=grpc
chrislusf commented

seems no error messages from seaweed itself.

  • 👍1
Gradlon commented

I found the issue.
There was an outdated driver version that was blocking the process.
Probably related to this: moby/swarmkit#3120

Now the volume is created(I see a directory in the Filer).
But I have two issues.

  1. Path argument is not applied, the volume is always created in /buckets/
  2. And I am unable to mount the volume onto a Container.
Jul 13 13:10:45 builder dockerd[69345]: time="2024-07-13T13:10:45.313975523+02:00" level=debug msg="form data: {\"ClusterVolumeSpec\":{\"AccessMode\":{\"MountVolume\":{},\"Scope\":\"single\",\"Sharing\":\"all\"},\"AccessibilityRequirements\":{},\"Availability\":\"active\",\"CapacityRange\":{\"LimitBytes\":0,\"RequiredBytes\":0}},\"Driver\":\"gradlon/swarm-csi-swaweedfs:v1.2.0\",\"DriverOpts\":{\"path\":\"/docker/volumes\"},\"Name\":\"testVolume\"}" spanID=217124d632e3265b traceID=beab043ad2e94e70f1416a5dd0a32067
Jul 13 13:10:45 builder dockerd[69345]: time="2024-07-13T13:10:45.314104200+02:00" level=debug msg="using cluster volume" spanID=217124d632e3265b traceID=beab043ad2e94e70f1416a5dd0a32067
Jul 13 13:10:45 builder dockerd[69345]: time="2024-07-13T13:10:45.316060868+02:00" level=info msg="creating volume" attempt=0 driver="gradlon/swarm-csi-swaweedfs:v1.2.0" module=csi/manager node.id=xajfahck1rppdkh1s88haz400 volume.id=goezb38l6wk050isx2qwfkxqv
Jul 13 13:10:45 builder dockerd[69345]: time="2024-07-13T13:10:45+02:00" level=error msg="I0713 11:10:45.316983 controllerserver.go:27 create volume req: testVolume" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:10:45 builder dockerd[69345]: time="2024-07-13T13:10:45.324127667+02:00" level=debug msg="updated volume" module=scheduler node.id=xajfahck1rppdkh1s88haz400 volume.id=goezb38l6wk050isx2qwfkxqv

Wehen trying to mount the volume i get this error

Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08.741832071+02:00" level=info msg="attempting to publish volume" attempt=7 module=node/agent/csi volume.id=tpw9rgtlofwp7i1pt1j4r4ak2
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="I0713 11:21:08.746976 nodeserver.go:34 node stage volume testvolume to /data/staged/tpw9rgtlofwp7i1pt1j4r4ak2" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="I0713 11:21:08.747216 mounter_seaweedfs.go:45 mounting [10.0.0.3:8888] /buckets/testvolume to /data/staged/tpw9rgtlofwp7i1pt1j4r4ak2" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="W0713 11:21:08.747300 mounter_seaweedfs.go:139 VolumeContext 'path' ignored" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="I0713 11:21:08.747384 mounter.go:53 Mounting fuse with command: weed and args: [-logtostderr=true mount -dirAutoCreate=true -umask=000 -dir=/data/staged/tpw9rgtlofwp7i1pt1j4r4ak2 -localSocket=/tmp/seaweedfs-mount-196015100.sock -cacheDir=/tmp/testvolume -collection=testvolume -filer.path=/buckets/testvolume -concurrentWriters=32 -filer=10.0.0.3:8888 -cacheCapacityMB=0]" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=info msg="mount point owner uid=0 gid=0 mode=drwxrwxrwx" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=info msg="current uid=0 gid=0" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="I0713 11:21:08.809603 leveldb_store.go:47 filer store dir: /tmp/testvolume/774824a5/meta" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="I0713 11:21:08.809678 file_util.go:27 Folder /tmp/testvolume/774824a5/meta Permission: -rwxr-xr-x" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="/bin/fusermount: fuse device not found, try 'modprobe fuse' first" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="F0713 11:21:08.820251 mount_std.go:257 Mount fail: fusermount exited with code 256" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="E0713 11:21:08.822759 mounter.go:75 weed mount exit, pid: 104, path: /data/staged/tpw9rgtlofwp7i1pt1j4r4ak2, error: exit status 255" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Gradlon commented

@s4ke
The CSI Plugin is working!

I mapped most of the supported Driver options.
As you can see in the config.

I did no excessive testing yet (Works on my machine;)

There is one thing that bothers me is that the target directory on seaweed is always /buckets/{VolumeName}
@chrislusf
After reading more into the mounter code, I can see that is configurable, can you help me on this one?

Also, do you want to integrate my code into this repo, should I create a pull request?

  • 🎉2
chrislusf commented

/buckets/ was originally configuration, but later removed because of complexity.

Yes. This PR is welcome!

Gradlon commented

Do you have any idea how I can pass this as an argument?
Somehow i can't find out how to make it work.

chrislusf commented

/buckets/ is not configurable.

Gradlon commented

How hard would it be to make it configurable?

chrislusf commented

fairly hard.

Gradlon commented

Ok, that is unfortunate.

I wunder, when looking at the code

	if params == nil {
		params = make(map[string]string)
	}
	glog.V(4).Infof("params:%v", params)
	capacity := req.GetCapacityRange().GetRequiredBytes()

	if err := filer_pb.Mkdir(cs.Driver, "/buckets", volumeId, nil); err != nil {
		return nil, fmt.Errorf("error setting bucket metadata: %v", err)
	}

It looks feasible to me.
But also I don't have the entire picture and a rather naive approach to it.
params = make(map[string]string) those should be the parameters passed with the -opt flag while creating the volume, right?

Please excuse me for insisting, for me, it makes not that much sense to have my docker volumes stored in folder called buckets :/

Would you be willing to invest the time to implement this?

Gradlon commented

Pull request is ready.
The plugin works as expected.
I had some issues using it, but they are probably more related to some issue with the CSI implementation itself.
#173

  • 👍1
Stitch10925 commented

Is this plugin POSIX compliant like the FUSE mount is?

Gradlon commented

As far as I can say, it is as compliant as the Original Seewead client. I am not sure if the seaweed client is 100% POSIX compliant though.