/helm-nifi

Helm Chart for Apache Nifi

Primary LanguageJavaScriptApache License 2.0Apache-2.0

Helm Chart for Apache NiFi

CircleCI License version test

Introduction

This Helm chart installs Apache NiFi 1.16.3 in a Kubernetes cluster.

Prerequisites

  • Kubernetes cluster 1.10+
  • Helm 3.0.0+
  • Persistent Volumes (PV) provisioner support in the underlying infrastructure.

Installation

Add Helm repository

helm repo add cetic https://cetic.github.io/helm-charts
helm repo update

Configure the chart

The following items can be set via --set flag during installation or configured by editing the values.yaml file directly (need to download the chart first).

Configure how to expose nifi service

  • Ingress: The ingress controller must be installed in the Kubernetes cluster.
  • ClusterIP: Exposes the service on a cluster-internal IP. Choosing this value makes the service only reachable from within the cluster.
  • NodePort: Exposes the service on each Node’s IP at a static port (the NodePort). You’ll be able to contact the NodePort service, from outside the cluster, by requesting NodeIP:NodePort.
  • LoadBalancer: Exposes the service externally using a cloud provider’s load balancer.

Configure how to persist data

  • Disable: The data does not survive the termination of a pod.
  • Persistent Volume Claim(default): A default StorageClass is needed in the Kubernetes cluster to dynamically provision the volumes. Specify another StorageClass in the storageClass or set existingClaim if you have already existing persistent volumes to use.

Configure authentication

  • By default, the authentication is a Single-User authentication. You can optionally enable ldap or oidc to provide an external authentication. See the configuration section or doc folder for more details.

Use custom processors

To add custom processors, the values.yaml file nifi section should contain the following options, where CUSTOM_LIB_FOLDER should be replaced by the path where the libs are:

  extraVolumeMounts:
    - name: mycustomlibs
      mountPath: /opt/configuration_resources/custom_lib
  extraVolumes: # this will create the volume from the directory
    - name: mycustomlibs
      hostPath:
        path: "CUSTOM_LIB_FOLDER"
  properties:
    customLibPath: "/opt/configuration_resources/custom_lib"

Configure prometheus monitoring

  • You first need monitoring to be enabled which can be accomplished by enabling the appropriate metrics flag (metrics.prometheus.enabled to true). To enable the creation of prometheus metrics within Nifi we need to create a Reporting Task. Login to the Nifi UI and go to the Hamburger menu on the top right corner, click Controller Settings --> Reporting Tasks After that use the + icon to add a task. Click on the Reporting in the wordcloud on the left and select PrometheusReportingTask --> change Send JVM metrics to true and click on the play button to enable this task.

If you plan to use Grafana for the visualization of the metrics data the following dashboard is compatible with the exposed metrics.

Install the chart

Install the nifi helm chart with a release name my-release:

helm install my-release cetic/nifi

Install from local clone

You will find how to perform an installation from a local clone on this page.

Uninstallation

To uninstall/delete the my-release deployment:

helm uninstall my-release

Configuration

The following table lists the configurable parameters of the nifi chart and the default values.

Parameter Description Default
ReplicaCount
replicaCount Number of nifi nodes 1
Image
image.repository nifi Image name apache/nifi
image.tag nifi Image tag 1.16.3
image.pullPolicy nifi Image pull policy IfNotPresent
image.pullSecret nifi Image pull secret nil
SecurityContext
securityContext.runAsUser nifi Docker User 1000
securityContext.fsGroup nifi Docker Group 1000
sts
sts.useHostNetwork If true, use the host's network nil
sts.serviceAccount.create If true, a service account will be created and used by the statefulset false
sts.serviceAccount.name When set, the set name will be used as the service account name. If a value is not provided a name will be generated based on Chart options nil
sts.serviceAccount.annotations Service account annotations {}
sts.podManagementPolicy Parallel podManagementPolicy Parallel
sts.AntiAffinity Affinity for pod assignment soft
sts.pod.annotations Pod template annotations security.alpha.kubernetes.io/sysctls: net.ipv4.ip_local_port_range=10000 65000
sts.hostAliases Add entries to Pod /etc/hosts []
sts.startupProbe.enabled enable Startup Probe on Nifi server container false
sts.startupProbe.failureThreshold sets Startup Probe failureThreshold field value 60
sts.startupProbe.periodSeconds sets Startup Probe periodSeconds field value 10
secrets
secrets Pass any secrets to the nifi pods. The secret can also be mounted to a specific path if required. nil
configmaps
configmaps Pass any configmaps to the nifi pods. The configmap can also be mounted to a specific path if required. nil
nifi properties
properties.algorithm Encryption method NIFI_PBKDF2_AES_GCM_256
properties.sensitiveKey Encryption password (at least 12 characters) changeMechangeMe
properties.sensitiveKeySetFile Update Sensitive Properties Key if this file does not exist, and then create it. nil
properties.sensitiveKeyPrior Prior sensitiveKey when updating via sensitiveKeySetFile mechanism nil
properties.externalSecure externalSecure for when inbound SSL false
properties.isNode cluster node properties (only configure for cluster nodes) false
properties.httpPort web properties HTTP port 8080
properties.httpsPort web properties HTTPS port null
properties.clusterPort cluster node port 6007
properties.provenanceStorage nifi provenance repository max storage size 8 GB
properties.siteToSite.secure Site to Site properties Secure mode false
properties.siteToSite.port Site to Site properties Secure port 10000
properties.safetyValve Map of explicit 'property: value' pairs that overwrite other configuration nil
properties.customLibPath Path of the custom libraries folder nil
properties.webProxyHost Proxy to access to Nifi through the cluster ip address Port:30236
Authentication
Single-user authentication Automatically disabled if Client Certificate, OIDC, or LDAP enabled
auth. admin Default admin identity. It will overwrite the LDAP Bind DN for this purpose, when both is filled CN=admin, OU=NIFI
auth.singleUser.username Single user identity username
auth.singleUser.password Single user password changemechangeme
Client Certificate authentication
auth.clientAuth.enabled Enable User auth via Client Certificates false
Ldap authentication
auth.ldap.admin Default admin identity and LDAP Bind DN
auth.ldap.enabled Enable User auth via ldap false
auth.ldap.host ldap hostname ldap://<hostname>:<port>
auth.ldap.searchBase ldap searchBase CN=Users,DC=example,DC=com
auth.ldap.searchFilter ldap searchFilter CN=john
auth.ldap.userSearchScope ldap userSearchScope ONE_LEVEL
auth.ldap.groupSearchScope ldap groupSearchScope ONE_LEVEL
Oidc authentication
auth.oidc.enabled Enable User auth via oidc false
auth.oidc.discoveryUrl oidc discover url https://<provider>/.well-known/openid-configuration
auth.oidc.clientId oidc clientId nil
auth.oidc.clientSecret oidc clientSecret nil
auth.oidc.claimIdentifyingUser oidc claimIdentifyingUser email
auth.oidc.admin Default OIDC admin identity nifi@example.com
Note that OIDC authentication to a multi-NiFi-node cluster requires Ingress sticky sessions See background Also how
postStart
postStart Include additional libraries in the Nifi containers by using the postStart handler nil
Headless Service
headless.type Type of the headless service for nifi ClusterIP
headless.annotations Headless Service annotations service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
UI Service
service.type Type of the UI service for nifi NodePort
service.httpPort Port to expose service 8080
service.httpsPort Port to expose service in tls 443
service.annotations Service annotations {}
service.loadBalancerIP LoadBalancerIP if service type is LoadBalancer nil
service.loadBalancerSourceRanges Address that are allowed when svc is LoadBalancer []
service.processors.enabled Enables additional port/ports to nifi service for internal processors false
service.processors.ports Specify "name/port/targetPort/nodePort" for processors sockets []
Ingress
ingress.enabled Enables Ingress false
ingress.className Ingress controller Class nginx
ingress.annotations Ingress annotations {}
ingress.path Path to access frontend (See issue #22) /
ingress.hosts Ingress hosts []
ingress.tls Ingress TLS configuration []
Persistence
persistence.enabled Use persistent volume to store data false
persistence.storageClass Storage class name of PVCs (use the default type if unset) nil
persistence.accessMode ReadWriteOnce or ReadOnly [ReadWriteOnce]
persistence.configStorage.size Size of persistent volume claim 100Mi
persistence.authconfStorage.size Size of persistent volume claim 100Mi
persistence.dataStorage.size Size of persistent volume claim 1Gi
persistence.flowfileRepoStorage.size Size of persistent volume claim 10Gi
persistence.contentRepoStorage.size Size of persistent volume claim 10Gi
persistence.provenanceRepoStorage.size Size of persistent volume claim 10Gi
persistence.logStorage.size Size of persistent volume claim 5Gi
persistence.existingClaim Use an existing PVC to persist data nil
jvmMemory
jvmMemory bootstrap jvm size 2g
SideCar
sidecar.image Separate image for tailing each log separately and checking zookeeper connectivity busybox
sidecar.tag Image tag 1.32.0
sidecar.imagePullPolicy Image imagePullPolicy IfNotPresent
Resources
resources Pod resource requests and limits for logs {}
logResources
logresources. Pod resource requests and limits {}
affinity
affinity Pod affinity scheduling rules {}
nodeSelector
nodeSelector Node labels for pod assignment {}
terminationGracePeriodSeconds
terminationGracePeriodSeconds Number of seconds the pod needs to terminate gracefully. For clean scale down of the nifi-cluster the default is set to 60, opposed to k8s-default 30. 60
tolerations
tolerations Tolerations for pod assignment []
initContainers
initContainers Container definition that will be added to the pod as initContainers []
extraVolumes
extraVolumes Additional Volumes available within the pod (see spec for format) []
extraVolumeMounts
extraVolumeMounts VolumeMounts for the nifi-server container (see spec for details) []
env
env Additional environment variables for the nifi-container (see spec for details) []
envFrom Additional environment variables for the nifi-container from config-maps or secrets (see spec for details) []
extraContainers
extraContainers Additional container-specifications that should run within the pod (see spec for details) []
extraLabels
extraLabels Additional labels for the nifi pod nil
openshift
openshift.scc.enabled If true, a openshift security context will be created permitting to run the statefulset as AnyUID false
openshift.route.enabled If true, a openshift route will be created. This option cannot be used together with Ingress as a route object replaces the Ingress. The property properties.externalSecure will configure the route in edge termination mode, the default is passthrough. The property properties.httpsPort has to be set if the cluster is intended to work with SSL termination false
openshift.route.host The hostname intended to be used in order to access NiFi web interface nil
openshift.route.path Path to access frontend, works the same way as the ingress path option nil
zookeeper
zookeeper.enabled If true, deploy Zookeeper true
zookeeper.url If the Zookeeper Chart is disabled a URL and port are required to connect nil
zookeeper.port If the Zookeeper Chart is disabled a URL and port are required to connect 2181
registry
registry.enabled If true, deploy Nifi Registry false
registry.url If the Nifi Registry Chart is disabled a URL and port are required to connect nil
registry.port If the Nifi Registry Chart is disabled a URL and port are required to connect 80
ca
ca.enabled If true, deploy Nifi Toolkit as CA false
ca.server CA server dns name nil
ca.port CA server port number 9090
ca.token The token to use to prevent MITM 80
ca.admin.cn CN for admin certificate admin
ca.serviceAccount.create If true, a service account will be created and used by the deployment false
ca.serviceAccount.name When set, the set name will be used as the service account name. If a value is not provided a name will be generated based on Chart options nil
ca.openshift.scc.enabled If true, an openshift security context will be created permitting to run the deployment as AnyUID false
certManager
certManager.enabled If true, use cert-manager to create and rotate intra-NiFi-cluster TLS keys (note that cert-manager is a Kubernetes cluster-wide resource, so is not installed automatically by this chart) false
certManager.clusterDomain Kubernetes cluster top level domain, to generate fully qualified domain names for certificate Common Names cluster.local
certManager.keystorePasswd Java Key Store password for NiFi keystore changeme
certManager.truststorePasswd Java Key Store password for NiFi truststore changeme
certManager.additionalDnsNames Additional DNS names to incorporate into TLS certificates (e.g. where users point browsers to access the NiFi UI) [ localhost ]
certManager.caSecrets Names of Kubernetes secrets containing ca.crt keys to add to the NiFi truststore [ ]
certManager.refreshSeconds How often the sidecar refreshes the NiFi keystore (not truststore) from the cert-manager Kubernetes secrets 300
certManager.resources Memory and CPU resources for the node certificate refresh sidecar 100m CPU, 128MiB RAM
certManager.replaceDefaultTrustStore Use the certManager truststore, not the default Java trusted CA collection (for [e.g.] private OIDC provider) false
certManager.certDuration NiFi node certificate lifetime (90 days) 2160h
certManager.caDuration Certificate Authority certificate lifetime (10 years) 87660h
metrics
metrics.prometheus.enabled Enable prometheus to access nifi metrics endpoint false
metrics.prometheus.port Port where Nifi server will expose Prometheus metrics 9092
metrics.prometheus.serviceMonitor.enabled If true, creates a Prometheus Operator ServiceMonitor (also requires metrics.prometheus.enabled to be true) false
metrics.prometheus.serviceMonitor.namespace In which namespace the ServiceMonitor should be created
metrics.prometheus.serviceMonitor.labels Additional labels for the ServiceMonitor nil
customFlow
customFlow Use this file (uncompressed XML; possibly from a configmap) as the Flow definition nil

Troubleshooting

Before filing a bug report, you may want to:

  • check the FAQ
  • check that persistent storage is configured on your cluster
  • keep in mind that a first installation may take a significant amount of time on a home internet connection
  • check if a pod is in error:
kubectl get pod
NAME                  READY   STATUS    RESTARTS   AGE
myrelease-nifi-0             3/4     Failed   1          56m
myrelease-nifi-registry-0    1/1     Running   0          56m
myrelease-nifi-zookeeper-0   1/1     Running   0          56m
myrelease-nifi-zookeeper-1   1/1     Running   0          56m
myrelease-nifi-zookeeper-2   1/1     Running   0          56m

Inspect the pod, check the "Events" section at the end for anything suspicious.

kubectl describe pod myrelease-nifi-0

Get logs on a failed container inside the pod (here the server one):

kubectl logs myrelease-nifi-0 server

Credits

Initially inspired from https://github.com/YolandaMDavis/apache-nifi.

TLS work/inspiration from https://github.com/sushilkm/nifi-chart.git.

Contributing

Feel free to contribute by making a pull request.

Please read the official Helm Contribution Guide from Helm for more information on how you can contribute to this Chart.

License

Apache License 2.0