Registering private IPs thru mesos-consul
Lax77 opened this issue · 18 comments
I am provisioning containers on mesos framework using Calico (multi host networking solution). Calico assigns a separate IP for each of the container that I provision thru Calico from my defined IP pool.
Issue is when my container got registered thru mesos-consul, in my Consul I see the HOST IP of the container. Instead is there a way I can register container's IP thru mesos-consul and have Consul report Container's IP instead of Host IP?
Thanks
Lax
You can try changing the IP order via the --mesos-ip-order option. The default is netinfo,mesos,host.
Ok. I was using mesos,host. Would preceding the order with docker would help?
You can try docker. netinfo might also work as well.
Ok. Some how when I gave in like "--mesos-ip-order=netinfo,mesos,host" didnt work. Will try just having either of docker or netinfo. Thanks
seeing weird behavior. When I change mesos-ip-order and re-deploy mesos-consul, I keep seeing the error 'Timed out waiting for initial ZK detection'. This causes the mesos-consul to keep re-deploying every 2 mins.
I tried pinging & telneting the ZK IP and port, that seems to work fine. Also I do see 'json.info_0000000002' file when 'ls /mesos' on ZKCli. Is there anything I am missing?
Can you post your mesos-consul config?
{
"args": [
"--zk=zk://zookeeper.service.local:2181/mesos",
"--mesos-ip-order=docker,mesos,host"
],
"container": {
"type": "DOCKER",
"docker": {
"network": "HOST",
"image": "ciscocloud/mesos-consul"
}
},
"id": "mesos-consul",
"instances": 1,
"cpus": 0.1,
"mem": 128,
"labels": {
"task_type": "infra"
}
}
That looks good. Hmm. Does zookeeper.service.local resolve properly? The IP order processing occurs after the initial zookeeper connection so that shouldn't be an issue...
Ok, It does resolve properly. I am re-spinning my cluster again just in case if it helps issue go away
On re-spun system with the above app definition, it came up fine this time. But when I deploy a container in consul I still see Host IP though
That's not good. Can you post the task from Mesos's state.json? I'm wondering if the container IP is even in the state file...
you mean content of json.info_0000000002 file?
No. curl http://mesos-master:5050/master/state.json. That's where mesos-consul gets its data from. I want to see if the container IP is in the mesos state data.
Here is my task data seen in the state.json file
{
"id": "redis2.92a2aeca-194d-11e6-8438-467bb8096b30",
"name": "redis2",
"framework_id": "fa6b2f1e-0483-4dad-9867-d0794f6daf40-0000",
"executor_id": "",
"slave_id": "48c73e8e-81a4-423f-ae84-9a6265e6264c-S2",
"state": "TASK_RUNNING",
"resources": {
"cpus": 0.2,
"disk": 0,
"mem": 64,
"ports": "[4740-4740]"
},
"statuses": [
{
"state": "TASK_RUNNING",
"timestamp": 1463173182.03674,
"labels": [
{
"key": "Docker.NetworkSettings.IPAddress",
"value": ""
}
],
"container_status": {
"network_infos": [
{
"ip_address": "",
"ip_addresses": [
{
"ip_address": ""
}
]
}
]
}
}
],
"discovery": {
"visibility": "FRAMEWORK",
"name": "redis2",
"ports": {
"ports": [
{
"number": 4740,
"protocol": "tcp"
}
]
}
},
"container": {
"type": "DOCKER",
"docker": {
"image": "redis",
"network": "HOST",
"privileged": false,
"parameters": [
{
"key": "net",
"value": "infrastructure"
}
],
"force_pull_image": false
}
}
},
Inside container I do see it has assigned IP such as 192.168.0.0, 192.168.0.64.
In the above definition ' "value": "infrastructure"' tells mesos to use custom network called 'infrastructure' that was created with Calico.
In the case of calico provisioned container, as you seen above container ip is not available in mesos state data. But the container internally does have an IP, is there a way we can ask mesos-consul to pull IP from inside the container and have it registered?
Not really. mesos-consul runs on one node and would have to communicate with docker on each follower node.
Finally able to get Calico's IP registered thru mesos-consul. Apparently version 0.28 of Mesos has a fix which makes calico's Ip available. Earlier I was running version 0.27.
Now one final question, since my current mesos-ip-order is docker,mesos,host. I end up seeing consul list internal IP for both Calico provisioned and Bridge provisioned. Is there any way can I have mesos-consul to register internal Ip for a specific set of containers and host ips for other set of containers?