Global NATS Cluster

Creating a global messaging cluster using NATS and Fly.

Rationale

NATS is an open source messaging backend you can use for everything from chat applications to infrastructure events.

This is an example application that runs multiple NATS servers on Fly.io. It creates a mesh of NATS servers that communicate over a private, encrypted IPv6 network.

This network is available to apps within the same 6PN network on Fly; part of the same organization

Setup

flyctl init --import example-fly.toml

You'll be prompted for an app name, hit return to let Fly generate an app name for you. The --import flag uses the given file as a template to configure.
flyctl deploy

This will configure a single node of a NATS cluster
Add more regions with flyctl regions add <region> or

For this demo, we set ord, syd, cdg regions.

fly regions set ord syd cdg

Scale the application so it can place nodes in the regions.

fly scale count 3

Then run flyctl logs and you'll see the virtual machines discover each other.

2020-11-17T17:31:07.664Z d1152f01 ord [info] [493] 2020/11/17 17:31:07.646272 [INF] [fdaa:0:1:a7b:abc:21de:af5f:2]:4248 - rid:1 - Route connection created
2020-11-17T17:31:07.713Z 21deaf5f cdg [info] [553] 2020/11/17 17:31:07.704807 [INF] [fdaa:0:1:a7b:81:d115:2f01:2]:34902 - rid:19 - Route connection created
2020-11-17T17:31:08.123Z 82fabc30 syd [info] [553] 2020/11/17 17:31:08.114852 [INF] [fdaa:0:1:a7b:81:d115:2f01:2]:4248 - rid:7 - Route connection created
2020-11-17T17:31:08.259Z d1152f01 ord [info] [493] 2020/11/17 17:31:08.241644 [INF] [fdaa:0:1:a7b:b92:82fa:bc30:2]:45684 - rid:2 - Route connection created

Testing the cluster

While the cluster is only accessible from inside the Fly network, you can use Fly's Wireguard support to create a VPN into your Fly organisation and private network.

Then you can use tools such as natscli to subscribe to topics, publish messages to topics and perform various tests on your NATS cluster. Install the tool first.

Once installed, create a context that points at your NATS cluster:

nats context add --server appname.internal:4222 --description "My Cluster" --select

You can subscribe to a topic with nats sub topicname:

nats sub fly.demo

And then, in another terminal sessions, we can use nats pub topicname to send either simple messages to that topic:

nats pub fly.demo "Hello World"

Or send multiple messages:

nats pub fly.demo "fly.demo says {{.Cnt}} @ {{.TimeStamp}}" --count=10

You're ready to start integrating NATS messaging into your other Fly applications.

What to try next

NATS streaming offers persistence features, you can create a NATS streaming app by modifying this demo and adding volumes: flyctl volume create
Create a NATS super cluster let you join multiple NATS clusters with gateways. If you want to run regional clusters, you can query the Fly DNS service to with <region>.<app-name>.internal to find server in specific regions.

Discuss