It is a repository with Helm charts for Apache NiFi and Apache NiFi Registry.
Blog Post about NiFi on Kubernetes.
This repo is constantly being improved.
You need the following components to use these Helm charts:
- Kubernetes cluster (1.15 and newer)
- Helm 3
Tested with the following Ingress:
- NGINX Ingress Controller Here we use featured SSL Passthrough.
You can find CI pipelines for two CICD tools: GitLab CI and GitHub Actions.
- GitLab CI: .gitlab-ci.yml
- GitHub Actions: directory .github/workflows
Create your values file or use the default one.
Apache NiFi:
helm --namespace nifi nifi upgrade --install ./apache-nifi/chart -f ./apache-nifi/chart/values.yamlApache NiFi Registry:
helm --namespace nifireg nifi upgrade --install ./apache-nifi-registry/chart -f ./apache-nifi-registry/chart/values.yamlThe following table lists the configurable parameters of the Apache NiFi chart and the default values.
| Parameter | Description | Default |
|---|---|---|
| ReplicaCount | ||
replicaCount |
Number of nifi nodes | 1 |
| Image | ||
image.repository |
nifi Image name | apache/nifi |
image.tag |
nifi Image tag | 1.11.4 |
image.pullPolicy |
nifi Image pull policy | IfNotPresent |
image.pullSecret |
nifi Image pull secret | nil |
| SecurityContext | ||
securityContext.runAsUser |
nifi Docker User | 1000 |
securityContext.fsGroup |
nifi Docker Group | 1000 |
| sts | ||
sts.podManagementPolicy |
Parallel podManagementPolicy | Parallel |
sts.AntiAffinity |
Affinity for pod assignment | soft |
sts.pod.annotations |
Pod template annotations | security.alpha.kubernetes.io/sysctls: net.ipv4.ip_local_port_range=10000 65000 |
| secrets | ||
secrets |
Pass any secrets to the nifi pods. The secret can also be mounted to a specific path if required. | nil |
| configmaps | ||
configmaps |
Pass any configmaps to the nifi pods. The configmap can also be mounted to a specific path if required. | nil |
| nifi properties | ||
properties.externalSecure |
externalSecure for when inbound SSL | false |
properties.isNode |
cluster node properties (only configure for cluster nodes) | true |
properties.httpPort |
web properties HTTP port | 8080 |
properties.httpsPort |
web properties HTTPS port | null |
properties.clusterPort |
cluster node port | 6007 |
properties.clusterSecure |
cluster nodes secure mode | false |
properties.needClientAuth |
nifi security client auth | false |
properties.provenanceStorage |
nifi provenance repository max storage size | 8 GB |
properties.siteToSite.secure |
Site to Site properties Secure mode | false |
properties.siteToSite.port |
Site to Site properties Secure port | 10000 |
properties.siteToSite.authorizer |
managed-authorizer |
|
properties.safetyValve |
Map of explicit 'property: value' pairs that overwrite other configuration | nil |
| nifi user authentication | ||
auth.ldap.enabled |
Enable User auth via ldap | false |
auth.ldap.host |
ldap hostname | ldap://<hostname>:<port> |
auth.ldap.searchBase |
ldap searchBase | CN=Users,DC=example,DC=com |
auth.ldap.searchFilter |
ldap searchFilter | CN=john |
| postStart | ||
postStart |
Include additional libraries in the Nifi containers by using the postStart handler | nil |
| Headless Service | ||
headless.type |
Type of the headless service for nifi | ClusterIP |
headless.annotations |
Headless Service annotations | service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" |
| Ingress | ||
ingress.enabled |
Enables Ingress | false |
ingress.annotations |
Ingress annotations | {} |
ingress.path |
Path to access frontend (See issue #22) | / |
ingress.hosts |
Ingress hosts | [] |
ingress.tls |
Ingress TLS configuration | [] |
| Persistence | ||
persistence.enabled |
Use persistent volume to store data | false |
persistence.storageClass |
Storage class name of PVCs (use the default type if unset) | nil |
persistence.accessMode |
ReadWriteOnce or ReadOnly | [ReadWriteOnce] |
persistence.dataStorage.size |
Size of persistent volume claim | 1Gi |
persistence.flowfileRepoStorage.size |
Size of persistent volume claim | 10Gi |
persistence.contentRepoStorage.size |
Size of persistent volume claim | 10Gi |
persistence.provenanceRepoStorage.size |
Size of persistent volume claim | 10Gi |
persistence.logStorage.size |
Size of persistent volume claim | 5Gi |
persistence.existingClaim |
Use an existing PVC to persist data | nil |
| jvmMemory | ||
jvmMemory |
bootstrap jvm size | 2g |
| SideCar | ||
sidecar.image |
Separate image for tailing each log separately | ez123/alpine-tini |
sidecar.tag |
Image tag | latest |
| BusyBox | ||
busybox.image |
Separate image for initContainer that verifies zookeeper is accessible | busybox |
busybox.tag |
Image tag | latest |
| Resources | ||
resources |
Pod resource requests and limits for logs | {} |
| logResources | ||
logresources. |
Pod resource requests and limits | {} |
| nodeSelector | ||
nodeSelector |
Node labels for pod assignment | {} |
| terminationGracePeriodSeconds | ||
terminationGracePeriodSeconds |
Number of seconds the pod needs to terminate gracefully. For clean scale down of the nifi-cluster the default is set to 60, opposed to k8s-default 30. | 60 |
| tolerations | ||
tolerations |
Tolerations for pod assignment | [] |
| initContainers | ||
initContainers |
Container definition that will be added to the pod as initContainers | [] |
| extraVolumes | ||
extraVolumes |
Additional Volumes available within the pod (see spec for format) | [] |
| extraVolumeMounts | ||
extraVolumeMounts |
VolumeMounts for the nifi-server container (see spec for details) | [] |
| env | ||
env |
Additional environment variables for the nifi-container (see spec for details) | [] |
| extraContainers | ||
extraContainers |
Additional container-specifications that should run within the pod (see spec for details) | [] |
The following table lists the configurable parameters of the Apache NiFi Registry chart and the default values.
| Parameter | Description | Default |
|---|---|---|
| ReplicaCount | ||
replicaCount |
Number of nifi nodes | 1 |
| Image | ||
image.repository |
nifi Image name | apache/nifi |
image.tag |
nifi Image tag | 1.11.4 |
image.pullPolicy |
nifi Image pull policy | IfNotPresent |
image.pullSecret |
nifi Image pull secret | nil |
| SecurityContext | ||
securityContext.runAsUser |
nifi Docker User | 1000 |
securityContext.fsGroup |
nifi Docker Group | 1000 |
| sts | ||
sts.podManagementPolicy |
Parallel podManagementPolicy | Parallel |
sts.AntiAffinity |
Affinity for pod assignment | soft |
sts.pod.annotations |
Pod template annotations | security.alpha.kubernetes.io/sysctls: net.ipv4.ip_local_port_range=10000 65000 |
| secrets | ||
secrets |
Pass any secrets to the nifi pods. The secret can also be mounted to a specific path if required. | nil |
| configmaps | ||
configmaps |
Pass any configmaps to the nifi pods. The configmap can also be mounted to a specific path if required. | nil |
| nifi properties | ||
properties.externalSecure |
externalSecure for when inbound SSL | false |
properties.isNode |
cluster node properties (only configure for cluster nodes) | true |
properties.httpPort |
web properties HTTP port | 8080 |
properties.httpsPort |
web properties HTTPS port | null |
properties.clusterPort |
cluster node port | 6007 |
properties.clusterSecure |
cluster nodes secure mode | false |
properties.needClientAuth |
nifi security client auth | false |
properties.provenanceStorage |
nifi provenance repository max storage size | 8 GB |
properties.siteToSite.secure |
Site to Site properties Secure mode | false |
properties.siteToSite.port |
Site to Site properties Secure port | 10000 |
properties.siteToSite.authorizer |
managed-authorizer |
|
properties.safetyValve |
Map of explicit 'property: value' pairs that overwrite other configuration | nil |
| nifi user authentication | ||
auth.ldap.enabled |
Enable User auth via ldap | false |
auth.ldap.host |
ldap hostname | ldap://<hostname>:<port> |
auth.ldap.searchBase |
ldap searchBase | CN=Users,DC=example,DC=com |
auth.ldap.searchFilter |
ldap searchFilter | CN=john |
| postStart | ||
postStart |
Include additional libraries in the Nifi containers by using the postStart handler | nil |
| Headless Service | ||
headless.type |
Type of the headless service for nifi | ClusterIP |
headless.annotations |
Headless Service annotations | service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" |
| Ingress | ||
ingress.enabled |
Enables Ingress | false |
ingress.annotations |
Ingress annotations | {} |
ingress.path |
Path to access frontend (See issue #22) | / |
ingress.hosts |
Ingress hosts | [] |
ingress.tls |
Ingress TLS configuration | [] |
| Persistence | ||
persistence.enabled |
Use persistent volume to store data | false |
persistence.storageClass |
Storage class name of PVCs (use the default type if unset) | nil |
persistence.accessMode |
ReadWriteOnce or ReadOnly | [ReadWriteOnce] |
persistence.dataStorage.size |
Size of persistent volume claim | 1Gi |
persistence.flowfileRepoStorage.size |
Size of persistent volume claim | 10Gi |
persistence.contentRepoStorage.size |
Size of persistent volume claim | 10Gi |
persistence.provenanceRepoStorage.size |
Size of persistent volume claim | 10Gi |
persistence.logStorage.size |
Size of persistent volume claim | 5Gi |
persistence.existingClaim |
Use an existing PVC to persist data | nil |
| jvmMemory | ||
jvmMemory |
bootstrap jvm size | 2g |
| SideCar | ||
sidecar.image |
Separate image for tailing each log separately | ez123/alpine-tini |
sidecar.tag |
Image tag | latest |
| BusyBox | ||
busybox.image |
Separate image for initContainer that verifies zookeeper is accessible | busybox |
busybox.tag |
Image tag | latest |
| Resources | ||
resources |
Pod resource requests and limits for logs | {} |
| logResources | ||
logresources. |
Pod resource requests and limits | {} |
| nodeSelector | ||
nodeSelector |
Node labels for pod assignment | {} |
| terminationGracePeriodSeconds | ||
terminationGracePeriodSeconds |
Number of seconds the pod needs to terminate gracefully. For clean scale down of the nifi-cluster the default is set to 60, opposed to k8s-default 30. | 60 |
| tolerations | ||
tolerations |
Tolerations for pod assignment | [] |
| initContainers | ||
initContainers |
Container definition that will be added to the pod as initContainers | [] |
| extraVolumes | ||
extraVolumes |
Additional Volumes available within the pod (see spec for format) | [] |
| extraVolumeMounts | ||
extraVolumeMounts |
VolumeMounts for the nifi-server container (see spec for details) | [] |
| env | ||
env |
Additional environment variables for the nifi-container (see spec for details) | [] |
| extraContainers | ||
extraContainers |
Additional container-specifications that should run within the pod (see spec for details) | [] |
Inspired from https://github.com/cetic/helm-nifi