ltsu

Long Term Storage Uploader.

CLI for uploading very large single files (up to 40 TB) to "cold" long-term cloud storage services for archival/backup purposes.

Currently supports AWS S3 Glacier and Backblaze B2.

Features

Smart resilience Handles service errors gracefully and automatically retries with exponential backoff.	Efficient transfer Uploads in parts simultaneously to balance performance, rate limits, and impact of errors.
Interruption safe Saves progress during upload to support resuming and out-of-order uploading.	Memory optimised Uses file streams and custom hash tree builder for low memory usage, even for large files.

Requirements

ltsu is written in TypeScript and requires Node.js version 10 or greater, available on Windows, macOS, and Linux.

npm is the default package manager that comes with Node.js and is used to install ltsu or run it on demand using npx.

Setup

To install it as a CLI tool:

npm i -g ltsu

This creates an executable in the global npm bin folder, so ensure that it's in the PATH environment variable. This folder can be found by running npm bin -g.

To use it directly without installing:

npx ltsu --file file --work workdir [...]

Usage

Common arguments

Name	Value	Description
`--file`	Required	Path to the file to upload.
`--work`	Required	Path to the directory that is used to hold state, such as information about resuming uploads. This folder will be filled with many files, so write permissions are required, existing files might be overwritten, and it may get cluttered. It's best to provide an empty directory for the exclusive use of one upload.
`--concurrency`	Default: 3	How many parts to upload at the same time. A high value might cause rate limiting, increased errors, and degraded performance. A low value might result in very slow total upload times.
`--quiet`	Optional	Hide the progress bar. This option can be used in tandem with `verbose`.
`--verbose`	Optional	Log whenever a part has been successfully uploaded. This option can be used in tandem with `quiet`.
`--force`	Optional	Always resume existing session, even if file is not the same or has changed since.
`--service`	Required, one of: `aws`, `b2`	Which cloud service to use.

Resuming

To resume an upload, simply run ltsu with the same --file and --work arguments.

If the file has changed since the last upload session, ltsu will refuse to run. Use --force to override this behaviour.

AWS S3 Glacier

Name	Description
`--region`	Region containing vault.
`--access`	Access key ID.
`--secret`	Secret access key.
`--vault`	Vault name.

The account owning the vault must be the same as the account associated with the credentials. For more information on AWS credentials and S3 Glacier, see here.

Example:

ltsu \
  --file /home/user/Archive.img \
  --work /tmp/archive-backup-2019-03-20 \
  --service aws \
  --region us-east-1 \
  --access AKIAIOSFODNN7EXAMPLE \
  --secret  wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
  --vault backups

Backblaze B2

An application key is required. Generate one from here. Both master and non-master keys can be used.

A key has two fields of interest: keyID and applicationKey. These need to be provided to ltsu as arguments.

Name	Description
`--account`	Key ID.
`--key`	Application key.
`--bucket`	Bucket ID (not name).

Full command format:

ltsu \
  --file /path/to/file \
  --work /path/to/working/dir \
  --service b2 \
  --account B2_ACCOUNT_ID \
  --key B2_APPLICATION_KEY \
  --bucket B2_BUCKET_ID

wilsonzlin/ltsu