saltstack-formulas/nifi-formula

nifi-formula

A salt formula to deploy Apache NiFi

Configure Apache NiFi in 3 different configurations:

• Standalone
• Cluster with embedded Zookeeper
• Cluster with external Zookeeper

Configures systemd and sysctl limits automatically. See the pillar.example for detailed list of pillars.

Table of Contents

General notes

See the full SaltStack Formulas installation and usage instructions.

If you are interested in writing or contributing to formulas, please pay attention to the Writing Formula Section.

If you want to use this formula, please pay attention to the FORMULA file and/or git tag, which contains the currently released version. This formula is versioned according to Semantic Versioning.

See Formula Versioning Section for more details.

If you need (non-default) configuration, please pay attention to the pillar.example file and/or Special notes section.

Contributing to this repo

Commit message formatting is significant!!

Please see How to contribute for more details.

Special notes

Requirements

• CentOS, Ubuntu, Debian
• Cluster configurations require hosts setup with FQDN and DNS.

Sample Pillars

Pillars - Standalone Node

By Default all of the settings will configure a standalone NiFi Node

nifi:
  pkg:
    name: nifi
    downloadurl: https://mirror.csclub.uwaterloo.ca/apache/nifi/1.12.1/nifi-1.12.1-bin.tar.gz
    version: 1.12.1
    installdir: /opt
    # JDK Package to install. Leave empty to not install.
    javajdk: java-1.8.0-openjdk
  nifi:
    cluster.is.node: 'false'
  systemdconfig:
    user: root
    group: root
    limitnofile: 50000
    limitnproc: 10000

Pillars - 3 Node Cluster with Embedded Zookeeper

nifi:
  pkg:
    name: nifi
    downloadurl: https://mirror.csclub.uwaterloo.ca/apache/nifi/1.12.1/nifi-1.12.1-bin.tar.gz
    version: 1.12.1
    installdir: /opt
    # JDK Package to install. Leave empty to not install.
    javajdk: java-1.8.0-openjdk
  systemdconfig:
    user: root
    group: root
    limitnofile: 50000
    limitnproc: 10000
  nifi:
    # cluster node properties (only configure for cluster nodes) #
    cluster.is.node: 'true'
    cluster.node.address: {{ grains['fqdn'] }}
    cluster.node.protocol.port: '1111'
    cluster.flow.election.max.wait.time: '1 mins'
    zookeeper.connect.string: 'nifi-1.localdomain:2181,nifi-2.localdomain:2181,nifi-3.localdomain:2181'
    state.management.embedded.zookeeper.start: 'true'
    web.http.host: {{ grains['fqdn'] }}
  zookeeperproperties:
    # To configure Zookeeper.properties set 'state.management.embedded.zookeeper.start' to 'true' above, and then define your Embedded Zookeeper servers here.
    customservers:
      Node1:
        hostname: nifi-1.localdomain
        zookeeper_myid: 1
        zookeeper_clientPort: 2181
        zookeeper_peerPorts: '2888:3888'
      Node2:
        hostname: nifi-2.localdomain
        zookeeper_myid: 2
        zookeeper_clientPort: 2181
        zookeeper_peerPorts: '2888:3888'
      Node3:
        hostname: nifi-3.localdomain
        zookeeper_myid: 3
        zookeeper_clientPort: 2181
        zookeeper_peerPorts: '2888:3888'

Pillars - 3 Node Cluster with External Zookeeper Servers

nifi:
  pkg:
    name: nifi
    downloadurl: https://mirror.csclub.uwaterloo.ca/apache/nifi/1.12.1/nifi-1.12.1-bin.tar.gz
    version: 1.12.1
    installdir: /opt
    # JDK Package to install. Leave empty to not install.
    javajdk: java-1.8.0-openjdk
  systemdconfig:
    user: root
    group: root
    limitnofile: 50000
    limitnproc: 10000
  nifi:
    # cluster node properties (only configure for cluster nodes) #
    cluster.is.node: 'true'
    cluster.node.address: {{ grains['fqdn'] }}
    cluster.node.protocol.port: '1111'
    cluster.flow.election.max.wait.time: '1 mins'
    zookeeper.connect.string: 'nifi-1.localdomain:2181,nifi-2.localdomain:2181,nifi-3.localdomain:2181'
    state.management.embedded.zookeeper.start: 'true'
    web.http.host: {{ grains['fqdn'] }}

Available states

nifi

Meta-state (This is a state that includes other states).

This installs the nifi package, manages the nifi configuration file and then starts the associated nifi service.

nifi.package

This state will install the nifi package only. This downloads the tar.gz file from the downloadurl and deploys it to servers.

nifi.config

This state will configure the nifi service and has a dependency on nifi.install via include list.

nifi.service

This state will start the nifi service and has a dependency on nifi.config via include list.

Testing

Linux testing is done with kitchen-salt.

Requirements

• Ruby
• Docker

$ gem install bundler
$ bundle install
$ bin/kitchen test [platform]

Where [platform] is the platform name defined in kitchen.yml, e.g. debian-9-2019-2-py3.

bin/kitchen converge

Creates the docker instance and runs the nifi main state, ready for testing.

bin/kitchen verify

Runs the inspec tests on the actual instance.

bin/kitchen destroy

Removes the docker instance.

bin/kitchen test

Runs all of the stages above in one go: i.e. destroy + converge + verify + destroy.

bin/kitchen login

Gives you SSH access to the instance for manual testing.