Bound to wrong ip address
Closed this issue · 2 comments
I am trying to run px-dev on a CoreOS system that has DC/OS V1.7.2 installed. DC/OS V1.7.2 introduces a DNS forwarder called 'Spartan'; the way it works is to create a new virtual network interface called 'Spartan' which has it's own IP address; all the entries in /etc/resolv.conf are then pointed at that IP address range. (See below for what ifconfig and /etc/resolv.conf look like in this environment.)
This confuses portworx; when it starts up it seems to think that it should use this interface for communication. This leads to errors:
2016/07/18 21:24:14 [WARN] memberlist 198.51.100.1: Refuting a suspect message (from: df125a7a-e77b-4e5c-a29f-f16f9cba4624)
2016/07/18 21:24:16 [WARN] memberlist: Was able to reach e57572a7-d55f-48eb-89f3-f18ad0d27433 via TCP but not UDP, network may be misconfigured and not allowing bidirectional UDP
2016/07/18 21:24:21 [WARN] memberlist: Was able to reach fa21d6ef-48dc-43b7-84ff-67eaebb00c05 via TCP but not UDP, network may be misconfigured and not allowing bidirectional UDP
2016/07/18 21:24:26 [WARN] memberlist: Was able to reach e57572a7-d55f-48eb-89f3-f18ad0d27433 via TCP but not UDP, network may be misconfigured and not allowing bidirectional UDP
You can also see below that it is using the ip address in the pxctl output.
I had this working before the DC/OS upgrade added this new interface, but now it's stuck in a weird state. Is there any way to tell the container which interface it should bind to?
Thanks,
Craig
sudo /opt/pwx/bin/pxctl status
Status: PX is operational
Node ID: df125a7a-e77b-4e5c-a29f-f16f9cba4624
IP: 198.51.100.1
Local Storage Pool: 14 devices
Device Path Caching Tier Size Used Last-Scan
1 /dev/sdo 5.5 TiB 751 MiB 18 Jul 16 21:15 UTC
10 /dev/sdf 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
11 /dev/sdj 5.5 TiB 1.7 GiB 18 Jul 16 21:15 UTC
12 /dev/sdl 5.5 TiB 1.7 GiB 18 Jul 16 21:15 UTC
13 /dev/sdb 5.5 TiB 739 MiB 18 Jul 16 21:15 UTC
14 /dev/sdg 5.5 TiB 739 MiB 18 Jul 16 21:15 UTC
2 /dev/sda 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
3 /dev/sdh 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
4 /dev/sde 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
5 /dev/sdi 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
6 /dev/sdk 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
7 /dev/sdc 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
8 /dev/sdd 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
9 /dev/sdm 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
total - 76 TiB 12 GiB
Cluster Summary
ID: 5ac2ed6f-7e4e-4e1d-8e8c-3a6df1fb7801
IP: 198.51.100.1 - Capacity: 77 TiB/19 MiB OK (This node)
IP: 10.1.20.1 - Capacity: 77 TiB/19 MiB OK
IP: 10.1.19.1 - Capacity: 77 TiB/173 MiB OK
IP: 10.1.18.1 - Capacity: 77 TiB/19 MiB OK
Global Storage Pool
Total Capacity : 308 TiB
Total Used : 230 MiB
Craig
montana@node-1-17 ~ $ ifconfig spartan
spartan: flags=195<UP,BROADCAST,RUNNING,NOARP> mtu 1500
inet 198.51.100.1 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::48e4:78ff:fef6:bfb prefixlen 64 scopeid 0x20
ether 4a:e4:78:f6:0b:fb txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 21972 bytes 7314572 (6.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
montana@node-1-17 ~ $ cat /etc/resolv.conf
Generated by gen_resolvconf.py. Do not edit.
Change configuration options by changing DCOS cluster configuration.
This file must be overwritten regularly for proper cluster operation around
master failure.
options timeout:1
options attempts:3
nameserver 198.51.100.1
nameserver 198.51.100.2
nameserver 198.51.100.3
Yes, you can specify which network interface to use. In you PX configuration file at /etc/pwx/config.json add the following two fields.
"mgtiface":
"dataiface": .
Your config file will look something like this if the interface is "eth0" :
{
"clusterid": "make this unique in your k/v store",
"mgtiface": "eth0",
"dataiface":"eth0",
"kvdb": [
"etcd:https://[username]:[password]@[string].dblayer.com:[port]"
],
"storage": {
"devices": [
"/dev/xvdb",
"/dev/xvdc"
]
}
}
Let us know if you face any problems.
Assuming this is resolved?