/terraform-aws-druid

Terraform module to deploy Apache Druid in Kubernetes

Primary LanguageHCLApache License 2.0Apache-2.0

Lint

Apache Druid terraform

This is a Terraform module for installing Druid on your Kubernetes cluster. This modules uses normal Kubernetes definitions files instead of the Helm Chart. Despite the Helm chart would make this module way more smaller, we think that for faster testing and deployment, it would have been simpler using multiple yaml files rather then templating.

Currently the this module works only with S3. We ❤️ contributions! Open a PR to support multiple cloud providers.

Build the docker image

To build the Druid docker image, follow the steps below:

  1. cd docker
  2. docker build -t your-repo/druid:latest .
  3. docker push your-repo/druid:latest

Remeber to use your own repository

Deploy the module

Once the image is built and pushed to the registry, you can install the module in your cluster.

It will take few minutes before it gets everything up and running. Once it's ready, you should be able to port-forward towards the Druid UI. To do so, run kubectl port-forward --namespace druid svc/router-cs 8888:8888, open your browser at http://localhost:8888/unified-console.html# and you should see the UI running. If you see 500 Errors within the boxes of the services, it means that it's not ready yet. Wait a little longer and then refresh. If it stays that way, you need to check where the error is.

If you are able to see all the services in the Druid UI it means that your cluster is ready to be used.

Affinity and AntiAffinity

In this module, we followed what the production documentation explains. The location/colocation of the services are done by associating the nodes according to the master, data and query services.

Postsgres resource

If you want to deploy Postgres without using different values for Database name, user and password, mind that you need to keep the same namespace as the druid resources. Those values are defined in a configmap and you cannot share configmaps across namespaces.

Requirements

Name Version
terraform >= 0.12.9
kubernetes >= 1.11.1

Providers

Name Version
kubernetes >= 1.11.1

Inputs

Name Description Type Default Required
aws_access_key AWS Access Key value. Permissions needed for S3 string n/a yes
aws_bucket_index S3 bucket for storing the indexes string n/a yes
aws_bucket_storage S3 bucket for storing the segments string n/a yes
aws_region AWS region string n/a yes
aws_secret_key AWS Secret Key value. Permissions needed for S3 string n/a yes
broker_replicas Number of replicas for the Broker service number 3 no
coordinator_replicas Number of replicas for the Coordinator service number 1 no
create_postgres Controls if Postgres database resources should be created (it affects almost all resources) bool true no
create_zookeeper Controls if Zookeeper resources should be created (it affects almost all resources) bool true no
druid_image_registry Docker registry used to fetch the Apache Druid image string "davideberdin" no
druid_image_repository Docker image of Apache Druid compatible for this module string "apache-druid" no
druid_image_tag Docker image tag string "0.18.1" no
historical_replicas Number of replicas for the Historical service number 1 no
middlemanager_replicas Number of replicas for the Middlemanager service number 1 no
namespace Namespace where Druid will be deployed string "druid" no
overlord_replicas Number of replicas for the Overlord service number 1 no
postgres_db Postgress Database name for Druid string "druid" no
postgres_host Postgress Database hostname for Druid string "postgres-cs.druid.svc.cluster.local" no
postgres_namespace namespace where to deploy the postgres resource string "druid" no
postgres_password Postgress Password of the user string "druid" no
postgres_port Postgress Database port for Druid string "5432" no
postgres_user Postgres username for accessing the DB string "druid" no
router_replicas Number of replicas for the Router service number 1 no
tolerations_broker toleration to apply to the deployment of the broker 
list(object({
    effect             = string
    key                = string
    operator           = string
    toleration_seconds = string
    value              = string
  }))
 [] no
tolerations_coordinator toleration to apply to the deployment of the coordinator 
list(object({
    effect             = string
    key                = string
    operator           = string
    toleration_seconds = string
    value              = string
  }))
 [] no
tolerations_historical toleration to apply to the deployment of the historical 
list(object({
    effect             = string
    key                = string
    operator           = string
    toleration_seconds = string
    value              = string
  }))
 [] no
tolerations_middlemanager toleration to apply to the deployment of the middlemanager 
list(object({
    effect             = string
    key                = string
    operator           = string
    toleration_seconds = string
    value              = string
  }))
 [] no
tolerations_overlord toleration to apply to the deployment of the overlord 
list(object({
    effect             = string
    key                = string
    operator           = string
    toleration_seconds = string
    value              = string
  }))
 [] no
tolerations_router toleration to apply to the deployment of the router 
list(object({
    effect             = string
    key                = string
    operator           = string
    toleration_seconds = string
    value              = string
  }))
 [] no
zookeeper_host Zookeeper hostname for Druid string "zk-cs.zk-druid.svc.cluster.local" no
zookeeper_namespace namespace where to deploy the zookeeper resource string "zk-druid" no
zookeeper_replicas Number of replicas for the Zookeeper service number 3 no

Outputs

No output.