fabric8io/elasticsearch-cloud-kubernetes

This plugin is no longer under active development and can be replaced by utilizing a headless service as noted below

Kubernetes Cloud Plugin for Elasticsearch:

The Kubernetes Cloud plugin allows to use Kubernetes API for the unicast discovery mechanism. Alternatively, you can utilize headless services:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
  name: elasticsearch-cluster
spec:
  clusterIP: None
  ports:
  - port: 9300
    protocol: TCP
    targetPort: 9300
  publishNotReadyAddresses: true
  selector:
    component: es
  sessionAffinity: None
  type: ClusterIP

elasticsearch.yml:

...
    discovery.zen:
      minimum_master_nodes: ${NODE_QUORUM}
      ping.unicast.hosts: elasticsearch-cluster
...

NOTE: You are highly encouraged to use headless services as the they obsolete this plugin.

Installation

elasticsearch-plugin install io.fabric8:elasticsearch-cloud-kubernetes:6.2.3.2

Versions available

As of 2.0.0, each released version of this plugin is compatible with the equal version of Elasticsearch, e.g. 2.2.0 of this plugin is compatible with 2.2.0 of Elasticsearch.

Older versions:

• 1.3.x for Elasticsearch 1.6 and 1.7

Kubernetes Pod Discovery

Kubernetes Pod discovery allows to use the kubernetes APIs to perform automatic discovery. Here is a simple sample configuration:

cloud:
  kubernetes:
    service: ${SERVICE}
    namespace: ${NAMESPACE}
discovery.zen.hosts_provider: kubernetes

path:
  data: /data/data
  logs: /data/log

Kubernetes auth

The preferred way to authenticate to Kubernetes is to use service accounts. This is fully supported by this Elasticsearch Kubernetes plugin as it uses the Fabric8 Kubernetes API client.

As an example, to create a service account do something like:

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elasticsearch
EOF

This creates a service account called elasticsearch. To use this service account, you need to add the service account to your pod spec or your replication controller pod template spec like this:

apiVersion: v1
kind: Pod
metadata:
  name: elasticsearch
  <SNIP>
spec:
  serviceAccountName: elasticsearch
  <SNIP>

This will mount the service account token at /var/run/secrets/kubernetes.io/servicaccount/token & will be automatically read by the fabric8 kubernetes client when the request is made to the API server.

Kubernetes example

In this example, we're going to use a headless service to look up the Elasticsearch cluster nodes to join.

The following manifest uses 3 replication controllers to created Elasticsearch pods in 3 different modes:

• master
• data
• coordinating-only

We use 2 services as well:

• One to target the coordinating-only pods so that all requests to the cluster go through the coordinating-only nodes
• A headless service to list all cluster endpoints managed by all 3 RCs.

kind: "List"
apiVersion: "v1"
items:
- apiVersion: "v1"
  kind: "Service"
  metadata:
    name: "elasticsearch"
  spec:
    ports:
      - port: 9200
        targetPort: "http"
    selector:
      component: "elasticsearch"
      type: "coordinating-only"
      provider: "fabric8"
    type: "LoadBalancer"
- apiVersion: "v1"
  kind: "Service"
  metadata:
    name: "elasticsearch-cluster"
  spec:
    clusterIP: "None"
    ports:
      - port: 9300
        targetPort: 9300
    selector:
      provider: "fabric8"
      component: "elasticsearch"
- apiVersion: "v1"
  kind: "ReplicationController"
  metadata:
    name: "elasticsearch-data"
  spec:
    replicas: 1
    selector:
      component: "elasticsearch"
      type: "data"
      provider: "fabric8"
    template:
      metadata:
        labels:
          component: "elasticsearch"
          type: "data"
          provider: "fabric8"
      spec:
        serviceAccount: elasticsearch
        serviceAccountName: elasticsearch
        containers:
          - env:
              - name: "SERVICE"
                value: "elasticsearch-cluster"
              - name: "KUBERNETES_NAMESPACE"
                valueFrom:
                  fieldRef:
                    fieldPath: "metadata.namespace"
              - name: "NODE_MASTER"
                value: "false"

            image: "fabric8/elasticsearch-k8s:6.2.3"
            name: "elasticsearch"
            ports:
              - containerPort: 9300
                name: "transport"
            volumeMounts:
              - mountPath: "/usr/share/elasticsearch/data"
                name: "elasticsearch-data"
                readOnly: false
        volumes:
          - emptyDir:
              medium: ""
            name: "elasticsearch-data"
- apiVersion: "v1"
  kind: "ReplicationController"
  metadata:
    name: "elasticsearch-master"
  spec:
    replicas: 1
    selector:
      component: "elasticsearch"
      type: "master"
      provider: "fabric8"
    template:
      metadata:
        labels:
          component: "elasticsearch"
          type: "master"
          provider: "fabric8"
      spec:
        serviceAccount: elasticsearch
        serviceAccountName: elasticsearch
        containers:
          - env:
              - name: "SERVICE"
                value: "elasticsearch-cluster"
              - name: "KUBERNETES_NAMESPACE"
                valueFrom:
                  fieldRef:
                    fieldPath: "metadata.namespace"
              - name: "NODE_DATA"
                value: "false"

            image: "fabric8/elasticsearch-k8s:6.2.3"
            name: "elasticsearch"
            ports:
              - containerPort: 9300
                name: "transport"
- apiVersion: "v1"
  kind: "ReplicationController"
  metadata:
    name: "elasticsearch-coordinating-only"
  spec:
    replicas: 1
    selector:
      component: "elasticsearch"
      type: "coordinating-only"
      provider: "fabric8"
    template:
      metadata:
        labels:
          component: "elasticsearch"
          type: "coordinating-only"
          provider: "fabric8"
      spec:
        serviceAccount: elasticsearch
        serviceAccountName: elasticsearch
        containers:
          - env:
              - name: "SERVICE"
                value: "elasticsearch-cluster"
              - name: "KUBERNETES_NAMESPACE"
                valueFrom:
                  fieldRef:
                    fieldPath: "metadata.namespace"
              - name: "NODE_DATA"
                value: "false"
              - name: "NODE_MASTER"
                value: "false"
            image: "fabric8/elasticsearch-k8s:6.2.3"
            name: "elasticsearch"
            ports:
              - containerPort: 9200
                name: "http"
              - containerPort: 9300
                name: "transport"