Wrong elastic server address

[measwel]

Dear Masum,

I have abandoned the idea of implementing geo search myself and want to use elassandra instead. Unfortunately elastic is throwing this error:

Elasticsearch ERROR: 2018-01-11T20:16:48Z
  Error: Request error, retrying
  HEAD http://172.18.0.6:9200/xx => socket hang up

The driver is configured to query at http://0.0.0.0:9200, so I do not understand where the address http://172.18.0.6:9200 is coming from? Perhaps you have an idea?

[masumsoft]

I don't quite get it. Why the driver would be configured to query at 0.0.0.0? In the context of a route entry, It usually means the default route. What address is elasticsearch listening to?

0.0.0.0 is a non-routable meta-address used to designate an invalid, unknown, or non-applicable target (a ‘no particular address’ place holder). In the context of a route entry, it usually means the default route.

In the context of servers, 0.0.0.0 means all IPv4 addresses on the local machine. If a host has two IP addresses, 192.168.1.1 and 10.1.2.1, and a server running on the host listens on 0.0.0.0, it will be reachable at both of those IPs.

[measwel]

Dear Masum,

The setup: elassandra runs in a docker container, which runs inside DockerToolbox ( virtual box machine ) which in turn runs inside mac osx - the host. Inside DockerToolbox, the container is reachable on address 0.0.0.0:PORT.

Elastic search listens on 0.0.0.0:9200
Cassandra listens on 0.0.0.0:9042

In the firewall between the host and DockerToolbox, these ports are open. I can query both without problems from the host.

from the host: curl 0.0.0.0:9200 gives:

{
  "name" : "172.18.0.3",
  "cluster_name" : "arbotti_cluster",
  "cluster_uuid" : "a0aa99db-c6a0-4823-b3de-f6e729a98e5c",
  "version" : {
    "number" : "5.5.0",
    "build_hash" : "37f8692",
    "build_date" : "2018-01-06T20:52:07.026Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}

Inside DockerToolbox: docker ps produces:
e93d0a8e2891 strapdata/elassandra "/docker-entrypoin..." 21 hours ago Up 21 hours 7000-7001/tcp, 7199/tcp, 0.0.0.0:9042->9042/tcp, 9160/tcp, 9300/tcp, 0.0.0.0:9200->9200/tcp a_DC1_CASSANDRA_SEED_1

In the driver settings I have:

      elasticsearch: {
        apiVersion: '5.5',
        sniffOnStart: true,
        host: 'http://0.0.0.0:9200',
        requestTimeout: 60000
      }

However, the application (running on host) tries to reach elastic by the IP 172.18.0.3 and produces the error:

Elasticsearch ERROR: 2018-01-12T17:27:26Z
Error: Request error, retrying
HEAD http://172.18.0.3:9200/arbotti => socket hang up

It seems as if the driver ignores the elastic search IP setting and gets it from elastic instead. Elastic mentions: "name" : "172.18.0.3" in its response. Maybe I need to force elastic to listen on 0.0.0.0 ?

PS The error only happens when I set a mapping for a table. There is no error when creating the elastic search index itself. It gets created fine.

[measwel]

The bottom line is, that the cassandra-express driver tries to connect to elastic search over the IP address 172.18.0.3 ( which is the IP of the docker container running elassandra ), while in the driver settings is stated:

  elasticsearch: {
    apiVersion: '5.5',
    sniffOnStart: true,
    host: 'http://0.0.0.0:9200',
    requestTimeout: 60000
  }

I have pinned down the problem. It is not a problem with the cassandra-express driver. It is caused by sniffOnStart in the elastic driver. This function gets the IP of the local elastic node ( which is the IP of the container running elassandra ) and overrides the 0.0.0.0 address set in config. For some reason elastic is only reachable via 0.0.0.0 but not via the IP of the container.

In any case thank you for the help Masum and for the excellent cassandra driver!

[masumsoft]

Aha! It was sniffOnStart that caused this. Okay, closing this since it's not directly an issue with express-cassandra. Thanks for reporting.

[measwel]

I still need to understand why sniffOnStart updates the connection pool and forgets the initial IP setting and how this should work in a distributed setting...

[masumsoft]

AFAIK sniffOnStart connects to the node you described in the connection setting and gets the cluster state and available node addresses in the cluster from the node it connected to. So it attempts to connect to one of the node addresses returned from the initial sniff. This is done in a distributed setup to avoid single point of failure. For example the hosts you've connected to might become unavailable, but other nodes in the cluster may be still available to respond to queries. So the elasticsearch driver updates it's initial hosts list with the alive host list returned from the cluster. Cassandra driver also does similar node discovery to avoid single point of failure.

I guess in your case, the elasticsearch cluster inside your docker container is returning the cluster host addresses that are valid in it's internal network which is not accessible from your elasticsearch client.