Longhorn

Status

Engine:
Manager:
UI:
Test:

Longhorn is a distributed block storage system for Kubernetes.

Longhorn is lightweight, reliable, and powerful. You can install Longhorn on an existing Kubernetes cluster with one kubectl apply command or using Helm charts. Once Longhorn is installed, it adds persistent volume support to the Kubernetes cluster.

Longhorn implements distributed block storage using containers and microservices. Longhorn creates a dedicated storage controller for each block device volume and synchronously replicates the volume across multiple replicas stored on multiple nodes. The storage controller and replicas are themselves orchestrated using Kubernetes. Here are some notable features of Longhorn:

Enterprise-grade distributed storage with no single point of failure
Incremental snapshot of block storage
Backup to secondary storage (NFS or S3-compatible object storage) built on efficient change block detection
Recurring snapshot and backup
Automated non-disruptive upgrade. You can upgrade the entire Longhorn software stack without disrupting running volumes!
Intuitive GUI dashboard

You can read more technical details of Longhorn here.

Current status

Longhorn is beta-quality software. We appreciate your willingness to deploy Longhorn and provide feedback.

The latest release of Longhorn is v0.7.0.

Source code

Longhorn is 100% open source software. Project source code is spread across a number of repos:

Longhorn engine -- Core controller/replica logic https://github.com/longhorn/longhorn-engine
Longhorn manager -- Longhorn orchestration, includes Flexvolume driver for Kubernetes https://github.com/longhorn/longhorn-manager
Longhorn UI -- Dashboard https://github.com/longhorn/longhorn-ui

Requirements

Docker v1.13+
Kubernetes v1.14+.
open-iscsi has been installed on all the nodes of the Kubernetes cluster.
1. For GKE, recommended Ubuntu as guest OS image since it contains open-iscsi already.
2. For Debian/Ubuntu, use apt-get install open-iscsi to install.
3. For RHEL/CentOS, use yum install iscsi-initiator-utils to install.
A host filesystem supports file extents feature on the nodes to store the data. Currently we support:
1. ext4
2. XFS

Install

On Kubernetes clusters Managed by Rancher 2.1 or newer

The easiest way to install Longhorn is to deploy Longhorn from Rancher Catalog.

On Rancher UI, select the cluster and project you want to install Longhorn. We recommended to create a new project e.g. Storage for Longhorn.
Navigate to the Catalog Apps screen. Select Launch, find Longhorn in the list. Select View Details, then click Launch. Longhorn will be installed in the longhorn-system namespace.

After Longhorn has been successfully installed, you can access the Longhorn UI by navigating to the Catalog Apps screen.

One benefit of installing Longhorn through Rancher catalog is Rancher provides authentication to Longhorn UI.

If there is a new version of Longhorn available, you will see an Upgrade Available sign on the Catalog Apps screen. You can click Upgrade button to upgrade Longhorn manager. See more about upgrade here.

On any Kubernetes cluster

Install Longhorn with kubectl

You can install Longhorn on any Kubernetes cluster using following command:

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml

Google Kubernetes Engine (GKE) requires additional setup in order for Longhorn to function properly. If your are a GKE user, read this page before proceeding.

Install Longhorn with Helm

First, you need to initialize Helm locally and install Tiller into your Kubernetes cluster with RBAC.

Then download Longhorn repository:

git clone https://github.com/longhorn/longhorn.git

Now using following command to install Longhorn:

helm install ./longhorn/chart --name longhorn --namespace longhorn-system

Longhorn will be installed in the namespace longhorn-system

One of the two available drivers (CSI and Flexvolume) would be chosen automatically based on the version of Kubernetes you use. See here for details.

A successful CSI-based deployment looks like this:

# kubectl -n longhorn-system get pod
NAME                                        READY     STATUS    RESTARTS   AGE
csi-attacher-0                              1/1       Running   0          6h
csi-provisioner-0                           1/1       Running   0          6h
engine-image-ei-57b85e25-8v65d              1/1       Running   0          7d
engine-image-ei-57b85e25-gjjs6              1/1       Running   0          7d
engine-image-ei-57b85e25-t2787              1/1       Running   0          7d
longhorn-csi-plugin-4cpk2                   2/2       Running   0          6h
longhorn-csi-plugin-ll6mq                   2/2       Running   0          6h
longhorn-csi-plugin-smlsh                   2/2       Running   0          6h
longhorn-driver-deployer-7b5bdcccc8-fbncl   1/1       Running   0          6h
longhorn-manager-7x8x8                      1/1       Running   0          6h
longhorn-manager-8kqf4                      1/1       Running   0          6h
longhorn-manager-kln4h                      1/1       Running   0          6h
longhorn-ui-f849dcd85-cgkgg                 1/1       Running   0          5d

Accessing the UI

You can run kubectl -n longhorn-system get svc to get the external service IP for UI:

NAME                TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)        AGE
longhorn-backend    ClusterIP      10.20.248.250   <none>           9500/TCP       58m
longhorn-frontend   LoadBalancer   10.20.245.110   100.200.200.123  80:30697/TCP   58m

If the Kubernetes Cluster supports creating LoadBalancer, you can use EXTERNAL-IP(100.200.200.123 in the case above) of longhorn-frontend to access the Longhorn UI. Otherwise you can use <node_ip>:<port> (port is 30697in the case above) to access the UI.

Noted that the UI is unauthenticated when you installed Longhorn using YAML file.

Upgrade

See here for details.

Upgrade Longhorn manager

On Kubernetes clusters Managed by Rancher 2.1 or newer

Follow the same steps for installation to upgrade Longhorn manager

Using kubectl

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml

Using Helm

helm upgrade longhorn ./longhorn/chart

Upgrade Longhorn engine

After Longhorn Manager was upgraded, Longhorn Engine also need to be upgraded using Longhorn UI. See here for details.

Create Longhorn Volumes

Before you create Kubernetes volumes, you must first create a storage class. Use following command to create a StorageClass called longhorn.

kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/master/examples/storageclass.yaml

Now you can create a pod using Longhorn like this:

kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/master/examples/pvc.yaml

The above yaml file contains two parts:

Create a PVC using Longhorn StorageClass.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-volv-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 2Gi

Use it in the a Pod as a persistent volume:

apiVersion: v1
kind: Pod
metadata:
  name: volume-test
  namespace: default
spec:
  containers:
  - name: volume-test
    image: nginx:stable-alpine
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: volv
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: volv
    persistentVolumeClaim:
      claimName: longhorn-volv-pvc

More examples are available at ./examples/

Documentation

Snapshot and Backup

Volume operations

Settings

Multiple disks

iSCSI

Kubernetes workload in Longhorn UI

Storage Tags

Customized default setting

Taint Toleration

Restoring Stateful Set volumes

Google Kubernetes Engine

Deal with Kubernetes node failure

Use CSI driver on RancherOS/CoreOS + RKE or K3S

Restore a backup to an image file

Disaster Recovery Volume

Recover volume after unexpected detachment

Troubleshooting

You can click Generate Support Bundle link at the bottom of the UI to download a zip file contains Longhorn related configuration and logs.

See here for the troubleshooting guide.

Uninstall Longhorn

Using kubectl

To prevent damaging the Kubernetes cluster, we recommend deleting all Kubernetes workloads using Longhorn volumes (PersistentVolume, PersistentVolumeClaim, StorageClass, Deployment, StatefulSet, DaemonSet, etc) first.
Create the uninstallation job to clean up CRDs from the system and wait for success:

kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/master/uninstall/uninstall.yaml
kubectl get job/longhorn-uninstall -w

Example output:

$ kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/master/uninstall/uninstall.yaml
serviceaccount/longhorn-uninstall-service-account created
clusterrole.rbac.authorization.k8s.io/longhorn-uninstall-role created
clusterrolebinding.rbac.authorization.k8s.io/longhorn-uninstall-bind created
job.batch/longhorn-uninstall created

$ kubectl get job/longhorn-uninstall -w
NAME                 COMPLETIONS   DURATION   AGE
longhorn-uninstall   0/1           3s         3s
longhorn-uninstall   1/1           20s        20s
^C

Remove remaining components:

kubectl delete -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml
kubectl delete -f https://raw.githubusercontent.com/longhorn/longhorn/master/uninstall/uninstall.yaml

Tip: If you try kubectl delete -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml first and get stuck there, pressing Ctrl C then running kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/master/uninstall/uninstall.yaml can also help you remove Longhorn. Finally, don't forget to cleanup remaining components.

Using Helm

helm delete longhorn --purge

Community

Longhorn is an open source software, so contribution are greatly welcome. Please read Code of Conduct and Contributing Guideline before contributing.

Contributing code is not the only way of contributing. We value feedbacks very much and many of the Longhorn features are originated from users' feedback. If you have any feedbacks, feel free to file an issue and talk to the developers at the CNCF #longhorn slack channel.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Jamiejoin/longhorn

Longhorn

Status

Current status

Source code

Requirements

Install

On Kubernetes clusters Managed by Rancher 2.1 or newer

On any Kubernetes cluster

Install Longhorn with kubectl

Install Longhorn with Helm

Accessing the UI

Upgrade

Upgrade Longhorn manager

On Kubernetes clusters Managed by Rancher 2.1 or newer

Using kubectl

Using Helm

Upgrade Longhorn engine

Create Longhorn Volumes

Documentation

Troubleshooting

Uninstall Longhorn

Using kubectl

Using Helm

Community

License

Longhorn is a CNCF Sandbox Project