/kafka-cluster-ansible

Fully Managed Apache Kafka Cluster with Ansible.

Primary LanguageShell

Apache Kafka Ansible

It is group of playbooks to manage apache kafka. It is also 100% compliant with ansible-lint rules.

Requirements

  • Download Apache Kafka Tar on Ansible Server ( Mandatory )
  • vagrant (optional)
  • Any OS with SystemD
  • Ansible

Notes*

1. It assumes you have zookeeper already running in localmode or cluster mode.
2. All tasks like broker/jvm/logging/downgrade/removeOldVersion will be done in serial order.

Extra

Production Environment Setup

Apache Kafka Playbooks

To start new cluster

  • Update Required vars in inventory/<environment>/group_vars/all.yml .
  • Update Required vars in inventory/<environment>/cluster.ini .

ansible-playbook -i inventory/<environment>/cluster.ini -u <user eg ec2-user > --private-key <key path eg: ~/Documents/ec2.key> clusterSetup.yml

To add new broker to cluster

  • Update Required vars in inventory/<environment>/group_vars/all.yml .
  • Update Required vars in inventory/<environment>/cluster.ini .

ansible-playbook -i inventory/<environment>/cluster.ini -u <user eg ec2-user > --private-key <key path eg: ~/Documents/ec2.key> clusterAddNodes.yml

To Upgrade Kafka Version

  • download kafka tar in your local ansible box.
  • Update Required vars in inventory/<environment>/group_vars/all.yml .
  • Update Required vars in inventory/<environment>/cluster.ini .

ansible-playbook -i inventory/<environment>/cluster.ini -u <user eg ec2-user > --private-key <key path eg: ~/Documents/ec2.key> clusterUpgrade.yml

Rolling restart cluster

ansible-playbook -i inventory/<environment>/cluster.ini -u <user eg ec2-user > --private-key <key path eg: ~/Documents/ec2.key> clusterRollingRestart.yml

To add newrelic monitoring to cluster

  • add newrelic rpms to files folder.

ansible-playbook -i inventory/<environment>/cluster.ini -u <user eg ec2-user > --private-key <key path eg: ~/Documents/ec2.key> clusterNewRelicSetup.yml

To update broker settings only of cluster

  • Update Required vars in inventory/<environment>/group_vars/all.yml .

ansible-playbook -i inventory/<environment>/cluster.ini -u <user eg ec2-user > --private-key <key path eg: ~/Documents/ec2.key> clusterBrokerProperties.yml

To update jvm settings only of cluster

  • Update Required vars in inventory/<environment>/group_vars/all.yml .

ansible-playbook -i inventory/<environment>/cluster.ini -u <user eg ec2-user > --private-key <key path eg: ~/Documents/ec2.key> clusterJvmConfigs.yml

To update logging settings only of cluster

  • Update Required vars in inventory/<environment>/group_vars/all.yml .

ansible-playbook -i inventory/<environment>/cluster.ini -u <user eg ec2-user > --private-key <key path eg: ~/Documents/ec2.key> clusterLogging.yml

To upgrade java version of cluster

  • Update Required vars in inventory/<environment>/group_vars/all.yml .

ansible-playbook -i inventory/<environment>/cluster.ini clusterJava.yml

To remove old version files of kafka from cluster

  • Update Required vars in inventory/<environment>/group_vars/all.yml .

ansible-playbook -i inventory/<environment>/cluster.ini -u <user eg ec2-user > --private-key <key path eg: ~/Documents/ec2.key> clusterRemoveOldVersion.yml

To remove kafka broker from cluster

  • Update Required vars in inventory/<environment>/group_vars/all.yml .

ansible-playbook -i inventory/<environment>/cluster.ini -u <user eg ec2-user > --private-key <key path eg: ~/Documents/ec2.key> clusterRemoveNodes.yml

Apache Kafka Mirror Maker Playbooks

To start new cluster

  • Update Required vars in inventory/<environment>/group_vars/kafka-mirror-maker.yml .
  • Update Required vars in inventory/<environment>/mirror-maker.ini .

ansible-playbook -i inventory/<environment>/mirror-maker.ini -u <user eg ec2-user > --private-key <key path eg: ~/Documents/ec2.key> clusterKafkaMirrorMaker.yml

To Remove nodes from cluster

  • Update Required vars in inventory/<environment>/group_vars/kafka-mirror-maker.yml .
  • Update Required vars in inventory/<environment>/mirror-maker.ini .

ansible-playbook -i inventory/<environment>/mirror-maker.ini -u <user eg ec2-user > --private-key <key path eg: ~/Documents/ec2.key> clusterKafkaMirrorMakerRemoveNodes.yml

Rolling restart cluster

ansible-playbook -i inventory/<environment>/mirror-maker.ini -u <user eg ec2-user > --private-key <key path eg: ~/Documents/ec2.key> clusterKafkaMirrorMakerRollingRestart.yml

Tested OS

  • CentOS 7
  • RedHat 7

Tested Ansible Version

ansible 2.8.1
  config file = None
  configured module search path = ['/home/davinderpal/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0]