Multicast in Overlay driver
nicklaslof opened this issue · 59 comments
Multicast network packages doesn't seem to be transported to all containers in an Overlay network configuration.
I have investigated this a bit trying various manual config inside the network name space but haven't got anywhere.
@nicklaslof Yes, this is the current behavior. overlay is implemented using vxlan unicast. So handling multicast need some form of packet replication. We are looking into possible options to support multicast on top of overlay.
Just to make it clear (especially since I wrote host instead of container in my original text) is that I mean using multicast between containers whilst still having vxlan unicast between the docker hosts.
@sanimej @mrjana @mavenugo hey guys, any update on whether there a solution in sight for this one? Per #740, this impacts use of the official elasticsearch image for creating a cluster.
If we can outline a possible solution here, perhaps someone from the community can attempt a fix if we don't have bandwidth for 1.10
@dave-tucker Multiple vxlan fdb entries can be created for all zero mac, which is the default destination. This gives an option (with some drawbacks) to handle multicast without the complexities of snooping. I have to try this out manually to see if it works.
@sanimej, @dave-tucker: any news on this one? We are looking for this so as to support containerization, with minimal refactoring, of a multicast-based service discovery integration in our stack. We can probably make unicast work but would prefer to not incur such a refactoring so as to avoid unintended consequences and/or further refactoring in our stack.
@dweomer this is not planned for the upcoming release. But, have added the help-wanted label to request help from any interested dev. If someone is interested in contributing this feature, we can help with design review & get this moving forward for the upcoming release.
+1 - some infrastructure components (like Aerospike) rely on Multicast for cluster discovery.
+1 - This would be very useful. At the very least the documentation should note that this is not currently supported.
Note there are other Docker Network plugins available which do support multicast. For instance the one I work on.
+1 in order to adapt Guidewire Application multihost clustering, multihost is must
+1 Can't get Wildfly and mod_cluster to work on Swam Mode, because the overlay network doesn't support multicast.
One could fall back to Unicast, but since one would also need to provide a proxy list with the ip addresses of all httpd load balancers and it would be very difficult to figure them out beforehand, I would say that Wildfly and Modcluster don't currently work on Swarm Mode. Regards.
+1 to support http://crate.io
any update on getting the multicast implemented ?
@DanyC97 used macvlan underlay instead for my needs as a quick solution and worked fine.
@codergr : Were you able to use macvlan in a swarm with services ? Can you easily change the scope of a network ?
@jocelynthode that is not supported yet unfortunately. PTAL moby/moby#27266 and it is one of supporting such a need. But it needs some discussion to get it right.
Another option is to use the Weave Net networking plugin for Docker, which has multicast support. Full disclosure: I work for Weaveworks.
@mjlodge but there is no support for network plugins on swarm mode, no? So weave can't be used in swarm mode and docker services. It's pity that in such environment (docker swarm and services) much of the applications that support clustering can't be used due to the lack of multicast.
@jonlatorre local scope plugins do work, so Weave Net can be used with ordinary containers when Docker is in swarm mode. But not services.
With Docker 1.12 it is not possible to use Docker services with plugins.
Docker 1.13 is intended to support network plugins, but as of today, using Docker 1.13-rc5, we have been unable to get a network plugin to do anything with Docker services.
(note I work for Weaveworks)
@bboreham thanks for the update. I hope that in final release wave could work with docker swarm and services, i'm impatient to use it :)
I filed moby/moby#30024 to describe what I see.
+1
+1
@jonlatorre the Weave Net plugin is now compatible with swarm mode, and can be downloaded via docker plugin install weaveworks/net-plugin
That is a "latest" build; there will be a frozen release version at some point in the near future.
@bboreham looks like the latest built (last built 6 hours ago as of now) is stuck with a registry problem.
#/ docker network create --driver weaveworks/net-plugin:latest my-net
Error response from daemon: rpc error: code = 3 desc = error during lookup of plugin weaveworks/net-plugin:latest
@azaars I believe this is fixed in weaveworks/weave#2950
Not sure how long that will take to propagate to the Store.
OK, I don't think that was the same problem, but I just tried it myself and seems ok:
root@bryan-dev2:~# docker plugin install weaveworks/net-plugin:latest
Plugin "weaveworks/net-plugin:latest" is requesting the following privileges:
- network: [host]
- mount: [/proc/]
- mount: [/var/run/docker.sock]
- mount: [/var/lib/]
- mount: [/etc/]
- capabilities: [CAP_SYS_ADMIN CAP_NET_ADMIN]
Do you grant the above permissions? [y/N] y
latest: Pulling from weaveworks/net-plugin
2b84216b4cae: Download complete
Digest: sha256:f71186a3a406f939bb21df0ce12656b671932e4360ee0e37438a87cc7a3b160e
Status: Downloaded newer image for weaveworks/net-plugin:latest
Installed plugin weaveworks/net-plugin:latest
root@bryan-dev2:~# docker network create --driver weaveworks/net-plugin:latest my-net
rh2jzxdo1bdcr38jufd5el2dx
Please open an issue at https://github.com/weaveworks/weave if you are still having trouble.
@bboreham I just found out that one can't issue docker network create --driver weaveworks/net-plugin:latest from just any swarm manager node. The manager node has to be the leader.
@azaars I don't recognize that - do you get an error message? Otherwise what are the symptoms?
@bboreham I'm using the latest version of docker listed below, trying to use this plugin(weaveworks/net-plugin:latest) to do a tomcat session replication, I created a manager node with docker swarm(not a legacy standalone Swarm) and deployed 2 tomcat services with this network driver, but seems it doesn't works, I'm not sure what's wrong with the configuration, any idea?
Here's some configuration:
version: "3.1"
services:
nginx:
image: nginx:1.10.3
ports:
- 82:80
volumes:
- "/home/nginx.conf:/etc/nginx/nginx.conf"
depends_on:
- tomcat1
- tomcat2
networks:
- my-tomcat
tomcat1:
image: tomcat:7.0.77
ports:
- 8888:8080
volumes:
- "/mnt/tomcat1/server.xml:/usr/local/tomcat/conf/server.xml"
- "/mnt/tomcat1/webapps:/usr/local/tomcat/webapps"
networks:
- my-tomcat
tomcat2:
image: tomcat:7.0.77
ports:
- 8889:8080
volumes:
- "/mnt/tomcat2/server.xml:/usr/local/tomcat/conf/server.xml"
- "/mnt/tomcat2/webapps:/usr/local/tomcat/webapps"
networks:
- my-tomcat
networks:
my-tomcat:
driver: weaveworks/net-plugin:latest
Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:05:44 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:05:44 2017
OS/Arch: linux/amd64
Experimental: false
+1 request from ADP
+1
+1 It is Crazy this hasnt been implemented
+1
+1
+1
+1
+1, this issue is the only thing blocking me from using docker in my infrastructure
+1
I use macvlan network for now because of this (macvlan is supported in swarm since version 17.06), but it's clearly less convenient.
@mavenugo is there any plan or tips on how this feature could/should be designed. What would be the starting point for getting it implemented.
I'm guessing the code goes somewhere in here:
https://github.com/docker/libnetwork/tree/master/drivers/overlay
Does the driver contain a list or method of fetching all the IP's within the network. Could it watch for a multicast then replicate it over all the IP's individually. Would this work or be a performance hit?
+1
I've been relying on Hazelcast's multicast discovery until I found out that overlay network doesn't support multicast. An external network definition with macvlan driver (swarm scoped) seems to be working although it cannot be defined inside the compose file. (as part of the stack) There is an issue already filed for this one as well: docker/cli#410
+1
+1
@tunix right now I try to install a Hazelcast Cluster (Running on payara full Profile Server) on different nodes via docker service...and run into the same issue. Could you Please describe your macvlan workaround? This issue seems to be a along lasting One...
There are 2 solutions to this problem afaik:
- Using hazelcast-discovery-spi plugin for Docker
- Using macvlan network driver
It's been a while since my last trial on this but just creating a macvlan network and using it (as an external network) should be sufficient.
$ docker network create -d macvlan --attachable my-network
$ docker service create ... --network my-network ...
@tunix creating a macvlan in a Swarm scope doesn't seem to Work. The Container starts, but it cannot reach any ipadress. Running with Overlay it work ( but then multicast is not available).
Any ideas?
` docker network create --driver macvlan --scope swarm sa_hazel
docker service create --network sa_hazel ...foo
`
+1
@deratzmann Using a network like yours, I can ping containers on remote hosts but multicast still doesn't work.
+1
Found another tradeoff when using macvlan driver in swarm.
The macvlan driver driver does not support port mappings, which prevents from using "mode=host" published ports as described in https://docs.docker.com/engine/swarm/services/#publish-a-services-ports-directly-on-the-swarm-node
Just asking: Is there any progress on this?
Is there any chance, that this will be implemented sometime?
For the time being I suggest everyone to use Weave Net which works flawlessly in my setup.
+1. From my own testing, multicast does not work with the bridge driver either. I'm not talking about routing multicast between the host's network and the containers' NAT'ed network -- I'm talking about two containers deployed side-by-side (same host) using the default bridge or a user-defined bridge network. Said containers cannot communicate via multicast. IMO getting multicast working within the bridge network would be a logical first step before moving on to the overlay network.
Also, the suggestion to use Weave Net will only work with Linux hosts.
I wish I knew earlier that containers cannot use multicast.
Edit: I know multicast should work with "net=host" but, aside from that not being an ideal solution in any sense, it does not work with Windows hosts.
Any update? Want to run discovery services in docker setup, because our images are automagically pulled by a provisioning service (IoT-Edge). I can't update any binaries outside the docker system... or it would be great hackery.
@KylePreuss it is possible to setup side-by-side (same host) multicast traffic between two or more containers using virtual ethernet veth with macvlan or ipvlan drivers.
Create the veth1 device:
sudo nmcli connection add type veth con-name veth1 ifname veth1 peer veth2 ipv4.method manual ipv4.addresses 192.168.128.1/23
Next the veth2 device:
sudo nmcli connection add type veth con-name veth2 ifname veth2 peer veth1 ipv4.method manual ipv4.addresses 192.168.129.1/23
Bring created veth1 connection up:
sudo nmcli connection up veth1
Bring created veth2 connection up:
sudo nmcli connection up veth2
Create Docker network configuration for created veth1 device:
docker network create --config-only --gateway 192.168.128.1 --subnet 192.168.128.0/23 --ip-range 192.168.128.2/24 --opt parent=veth1 veth1-config
Create Docker network configuration for created veth2 device:
docker network create --config-only --gateway 192.168.129.1 --subnet 192.168.128.0/23 --ip-range 192.168.129.2/24 --opt parent=veth2 veth2-config
Create Docker Swarm network veth1:
docker network create --scope swarm --driver macvlan --config-from veth1-config veth1
Create Docker Swarm network veth2:
docker network create --scope swarm --driver macvlan --config-from veth2-config veth2
Use it in Docker compose:
services:
multicast-sender:
networks:
- veth1
multicast-receiver:
networks:
- veth2
networks:
veth1:
external: true
veth2:
external: trueThe solution to use weave net seems to work on linux hosts, but is there a way to achieve this on windows hosts? I would like to have linux host as the manager and windows host as a worker. I can install the weave plugin, but multicast communication between the hosts does not work