Rabbitmq AutoCluster using AWS autoscale group backend is unable to connect to other nodes
zoran3 opened this issue · 8 comments
When autocluster plugin interrogates aws autoscale group, it retrieves the value of private dns field something like ip-192-168-x-xx. The hostname of box is hostA for example. Then it tries to connect to ip-192-168-x-xx which won't resolve. Running manually rabbitmqctl join_cluster ip-192-168-x-xx fails. However, running manually rabbitmqctl join_cluster hostA it connects. We don't seem to be able to change ip-192-168-x-xx as that is something that amazon sets internally.
Is it possible to interrogate another field from the autoscale group so that it returns the actual hostname of box instead of amazon given invalid thing. All of these cluster nodes are on an internal private subnet.
I don't think so, see what the AWS backend does.
RabbitMQ clustering requires hostname resolution for all peers on each node, with this plugin or without.
We're experiencing this issue as well - we set our hostnames to be easily identifiable, and they differ from the private_dns hostname which EC2 sets - which means that the hosts aren't picked up by the plugin - is there any way to use the IP address instead? I guess not, since the clustering requires the hostname to be set
Anyone get this to work? I run rabbitmq in containers and the nodes appear as something like rabbit@3635fea3764b.
I thought the whole point of the plugin is that it figures all this connection info out based on the autoscaling group.
I have the IAM policy set, the three nodes in three servers in elastic beanstalk work, but they dont see each other to cluster. I have a shared erlang.cookie as well.
I've set the autoclustering log level to debug, but the plugin seems happy although it only sees itself
Also having this issue. Tried looking up the default hostname through the AWS provided DNS server at x.x.x.2 and it returned nxdomain on the AWS provided DNS name.
Making progress. I have custom DNS servers and, as such, they are not able to register the EC2 server's default names. Once I fix the below, I will retest and see what happens.
- In the VPC, ensure that DNS Resolution and DNS hostnames are enabled.
- Find your AWS provided DNS server IP. If your VPC is 10.0.0.0/8, the IP will be 10.0.0.2. It is always the 2nd IP of the network you specify.
- Ensure your systems are looking up hostname.ec2.local. This will mean setting up the default domain on the linux system to be ec2.local.
Alternatively, you can set up a conditional forwarder on your regular DNS servers for the EC2.local domain to 10.0.0.2.
Ah, looks like docker allows you to set this on the container as well. https://docs.docker.com/engine/userguide/networking/default_network/configure-dns/
--dns=[dns server IP] --dns-search=ec2.local
There is now an option to use private IP addresses in RabbitMQ's fork: rabbitmq#14.
This plugin was forked by the RabbitMQ team and is now part of RabbitMQ. More information can be found @ https://github.com/rabbitmq/rabbitmq-autocluster