High Availability Postgres on Fly.io

This repo contains all the code and configuration necessary to run a highly available Postgres cluster in a Fly.io organization's private network. This source is packaged into Docker images which allow you to track and upgrade versions cleanly as new features are added.

If you just want to get a standard Postgres standalone or highly-available setup on Fly, check out the docs.

Customizing cluster behavior

Fly Postgres clusters are just regular Fly applications. If you need to customize Postgres in any way, you may fork this repo and deploy using normal Fly deployment procedures. You won't be able to use fly postgres commands with custom clusters. But it's a great way to experiment and potentially contribute back useful features!

Follow the rest of this README to run a customized setup.

Prepare a new application

You'll need a fresh Fly application in your preferred region to get started. Run these commands within the fork of this repository.

`fly launch --no-deploy`

This gets you started with a Fly application and an associated config file. Choose yes when asked whether to copy the existing configuration to the newly generated app.

Set secrets

This app requires a few secret environment variables. Generate a secure string for each, and save them.

SU_PASSWORD is the PostgreSQL super user password. The username is flypgadmin. Use these credentials to run high privilege administration tasks.

REPL_PASSWORD is used to replicate between instances.

OPERATOR_PASSWORD is the password for the standard user postgres. Use these credentials for connecting from your application.

fly secrets set SU_PASSWORD=<PASSWORD> REPL_PASSWORD=<PASSWORD> OPERATOR_PASSWORD=<PASSWORD>

Set the PRIMARY_REGION environment variable within your fly.toml

The PRIMARY_REGION value lets Stolon know which Postgres instances are eligible for election in the event of a failover. If this value is not set to the correct region, your cluster may not boot properly.

Deploy one instance

First, get one instance deployed in your preferred start region.

fly volumes create pg_data --region ord --size 10
fly deploy
fly status

Add a replica

Scaling up will automatically setup a replica for you. Do that now in the same region.

fly volumes create pg_data --region ord --size 10
fly scale count 2
fly status

Add a replica in another region

Scale to another region by creating a volume there. Now you should have a primary/replica pair in ord and a replica in syd.

fly volumes create pg_data --region syd --size 10
fly scale count 3
fly status

Connecting

Fly apps within the same organization can connect to your Postgres using the following URI:

postgres://postgres:<operator_password>@<postgres-app-name>.internal:5432/<database-name>

Connecting to Postgres from your local machine

Forward the server port to your local system with flyctl proxy:

flyctl proxy 5432 -a <postgres-app-name>

Postgres needs to be installed on your local machine.
Use psql to connect to your Postgres instance on the forwarded port.

psql postgres://postgres:<operator_password>@localhost:5432

Enabling TimescaleDB

This app includes the TimescaleDB extension. To enable TimescaleDB, take the following steps:

Ensure your Postgres app is running >= v0.0.28.

# View your image details
fly image show --app <app-name>

# Update to the latest ( Nomad-based apps )
fly image update --app <app-name>

Configure Postgres to preload the TimescaleDB library

fly pg config update --shared-preload-libraries timescaledb --app <app-name>

Restart Postgres

fly postgres restart --app <app-name>

Create the extension

# Connect to your target database
fly pg connect --app <app-name> --database <db-name>

# Create the extension
CREATE EXTENSION IF NOT EXISTS timescaledb;

Having trouble?

Create an issue or ask a question here: https://community.fly.io/

Contributing

If you're looking to get involved, fork the project and send pull requests.

attaryz/postgres-ha