supabase/realtime

horizontal scaling for self-hosted realtime servers, for broadcast and presence features

menasheh opened this issue · 7 comments

menasheh commented

If we spin up a single realtime server, presence feature works beautifully for the most part, with the possible exception of long-opened tabs not counting as present after a while. However, if we spin up multiple parallel realtime servers and use a load balancer, we're seeing presence smaller than the number of tabs we have open. It seems some are connected to one server, and some to the other, so the presence becomes out of sync.

Is there a way to keep multiple realtime servers in sync for broadcast and presence features?

menasheh commented

As I'm researching I've found DNS_NODES param, not sure if this works off of FLY yet? Looking for something that works in kubernetes or docker swarm.

filipecabaco commented

Hi @menasheh

You might be having issues because you actually need to connect the erlang nodes. we're using libcluster postgres strategy to achieve it in Realtime.

Where are you deploying your system? In theory the only thing you will need is to be able to ping another machines host name and that should be enough for them to connect between them.

More details on the strategy: https://github.com/supabase/libcluster_postgres

menasheh commented

Thanks @filipecabaco. If I understood this correctly, the default cluster strategy is DNS, but I can set CLUSTER_STRATEGIES: "POSTGRES" to change this? So far this is not working.

We're deploying on linux servers in a company-owned private cloud environment, using either Docker or Kubernetes.

Attaching section of logs from one of the kubernetes pods:

18:23:35.074 [debug] QUERY OK db=42.6ms
grant all on table public.test_tenant to authenticated; []
18:23:35.160 [debug] QUERY OK db=43.0ms
create publication supabase_realtime for table public.test_tenant []
18:23:35.205 [debug] QUERY OK db=45.5ms
commit []
18:23:37.290 [notice]     :alarm_handler: {:set, {:system_memory_high_watermark, []}}
18:23:37.375 [info] Elixir.Realtime.SignalHandler is being initialized...
18:23:37.376 [notice] SYN[realtime@127.0.0.1] Adding node to scope <Elixir.Realtime.Tenants.Connect>
18:23:37.377 [notice] SYN[realtime@127.0.0.1] Creating tables for scope <Elixir.Realtime.Tenants.Connect>
18:23:37.377 [notice] SYN[realtime@127.0.0.1|registry<Elixir.Realtime.Tenants.Connect>] Discovering the cluster
18:23:37.377 [notice] SYN[realtime@127.0.0.1|pg<Elixir.Realtime.Tenants.Connect>] Discovering the cluster
18:23:37.377 [notice] SYN[realtime@127.0.0.1] Adding node to scope <users>
18:23:37.377 [notice] SYN[realtime@127.0.0.1] Creating tables for scope <users>
18:23:37.377 [notice] SYN[realtime@127.0.0.1|registry<users>] Discovering the cluster
18:23:37.377 [notice] SYN[realtime@127.0.0.1|pg<users>] Discovering the cluster
18:23:37.377 [notice] SYN[realtime@127.0.0.1] Adding node to scope <Elixir.RegionNodes>
18:23:37.377 [notice] SYN[realtime@127.0.0.1] Creating tables for scope <Elixir.RegionNodes>
18:23:37.377 [notice] SYN[realtime@127.0.0.1|registry<Elixir.RegionNodes>] Discovering the cluster
18:23:37.378 [notice] SYN[realtime@127.0.0.1|pg<Elixir.RegionNodes>] Discovering the cluster
18:23:37.378 [warning] Replica region not found, defaulting to Realtime.Repo
18:23:37.384 [info] Running RealtimeWeb.Endpoint with cowboy 2.10.0 at :::4000 (http)
18:23:37.384 [info] Access RealtimeWeb.Endpoint at http://realtime.fly.dev
18:23:37.385 [notice] SYN[realtime@127.0.0.1] Adding node to scope <Elixir.PostgresCdcStream>
18:23:37.386 [notice] SYN[realtime@127.0.0.1] Creating tables for scope <Elixir.PostgresCdcStream>
18:23:37.386 [notice] SYN[realtime@127.0.0.1|registry<Elixir.PostgresCdcStream>] Discovering the cluster
18:23:37.386 [notice] SYN[realtime@127.0.0.1|pg<Elixir.PostgresCdcStream>] Discovering the cluster
18:23:37.387 [notice] SYN[realtime@127.0.0.1] Adding node to scope <Elixir.Extensions.PostgresCdcRls>
18:23:37.387 [notice] SYN[realtime@127.0.0.1] Creating tables for scope <Elixir.Extensions.PostgresCdcRls>
18:23:37.387 [notice] SYN[realtime@127.0.0.1|registry<Elixir.Extensions.PostgresCdcRls>] Discovering the cluster
18:23:37.388 [notice] SYN[realtime@127.0.0.1|pg<Elixir.Extensions.PostgresCdcRls>] Discovering the cluster
18:23:37.617 [info] [libcluster:postgres] Connected to Postgres database
18:23:38.180 request_id=F6JOxT985rM4oT0AAAtB [info] GET /
18:23:38.180 request_id=F6JOxT985rM4oT0AAAtB [debug] Processing with RealtimeWeb.PageLive.Index.index/2
  Parameters: %{}
  Pipelines: [:browser]
18:23:38.187 request_id=F6JOxT985rM4oT0AAAtB [info] Sent 200 in 6ms
18:23:38.374 [warning] Replica region not found, defaulting to Realtime.Repo
menasheh commented

How does realtime get the hostname?

filipecabaco commented

that might be the issue. we're using a vpc to achieve it so the ips are discoverable between all nodes which means that it works for Postgres libcluster strategy to just do a quick node() ( see here )

are you able to ping the hostnames / ip's from one container to another?

Towerful commented

Hello,
I'm trying to get into this as well.
I'm using k8s to try and deploy supabase, with a cloudnative-pg database.
So far everything is working, with a bit of fiddling.
But deploying multiple realtime pods seems difficult.

I have 3 pods of realtime.
The SLOT_NAME_SUFFIX is configured to the pod's pod-index, so they are connecting on supabase_realtime_replication_slot_0, supabase_realtime_replication_slot_1 and supabase_realtime_replication_slot_2 with the wal2json plugin correctly assigned.
So realtime postgres changes are working fine.

Following this thread, I have configured the pod's hostnames to be their FQDN.
So realtime-dev-1 hostname (and I guess node() ) will return realtime-dev-1.realtime.default.svc.cluster.local
It can ping realtime-dev-0.realtime.default.svc.cluster.local and realtime-dev-2.realtime.default.svc.cluster.local

I have also configure the following envVars:

CLUSTER_STRATEGIES: POSTGRES
POSTGRES_CLUSTER_CHANNEL_NAME: realtime_broadcast_cluster

I have tried a psql> listen realtime_broadcast_cluster and I dont see any activity, even when deleting and recreating the pods.
I also see nothing in the logs along the lines of Trying to connect to node that this would suggest exists.
The only log entry that seems relevant is [libcluster:postgres] Connected to Postgres database
So it seems like libcluster is loading the correct module, and is connecting.
It's just not doing anything else

Do I need to manually create a tenant via a curl request to trigger realtime to set itself up properly?
I'm currently relying on the default realtime-dev tenant that is created

Edit:
Poking through the code, i wonder if setting an envVar to match "$AWS_EXECUTION_ENV" = "AWS_ECS_FARGATE" ( from here ) would prevent log entries like SYN[realtime@127.0.0.1|..., which might be tripping up node joining?

Final edit:
if those env.*.eex files are applicable, perhaps the /rel/env.*.eex file could test for the env vars RELEASE_DISTRIBUTION and RELEASE_NODE before overwriting them?
this would allow docker/k8s/BYOC users to be able to manuall configure them
I'm going to have a play around with them tomorrow. if the "$AWS_EXECUTION_ENV" = "AWS_ECS_FARGATE" hack changes the log outputs, then this seems like a simple fix for BYOC

Towerful commented

Ok, I've had a play.
I'd love to create a pull request, but as a solo dev I suck at github stuff.

I've tested this, and it seems to be working for the POSTGRES strategy.
the lines

if [ -z $ip ]; then
  ip=127.0.0.1
fi

were overwriting anything that might be able to bootstrap the cluster connectivity, unless the environments exactly match fly's 6pn thingy or AWS fargate environment.
So, I tweaked it to this, and added a few echos to help people debugging a deployment.

patching: https://github.com/supabase/realtime/blob/cd04f2f744834296b5a4b3e360e95c3fab5f9165/rel/env.sh.eex

#!/bin/sh

# Set the release to work across nodes. If using the long name format like
# the one below (my_app@127.0.0.1), you need to also uncomment the
# RELEASE_DISTRIBUTION variable below. Must be "sname", "name" or "none".

if [ -z $ip ]; then
  # for Fly.io
  ip=$(grep fly-local-6pn /etc/hosts | cut -f 1)
  echo "No EnvVar 'ip' set, trying fly-local-6pn '${ip}'"
else
  echo "ip set from EnvVar 'ip' '${ip}'"
fi

# for AWS ECS Fargate
if [ "$AWS_EXECUTION_ENV" = "AWS_ECS_FARGATE" ]; then
  ip=$(hostname -I | awk '{print $3}')
  echo "AWS_EXECUTION_ENV is AWS_ECS_FARGATE. Overriding 'ip' '${ip}'"
fi

# default to localhost
if [ -z $ip ]; then
  ip=127.0.0.1
  echo "No EnvVar 'ip' set and could not auto-configure, defaulting to '${ip}'"
fi

# assign the value of NODE_NAME if it exists, else assign the value of FLY_APP_NAME,
# and if that doesn't exist either, assign "realtime" to node_name
node_name="${NODE_NAME:=${FLY_APP_NAME:=realtime}}"

if [ -z $RELEASE_DISTRIBUTION ]; then
  export RELEASE_DISTRIBUTION=name
  echo "No RELEASE_DISTRIBUTION set. Using '${RELEASE_DISTRIBUTION}'"
fi
if [ -z $RELEASE_NODE ]; then
  export RELEASE_NODE=$node_name@$ip
  echo "No RELEASE_NODE set. Using '${RELEASE_NODE}'"
fi

I've tested this on my local k8s cluster with the following env vars set:

- name: ip
  valueFrom:
    fieldRef:
      fieldPath: status.podIP
- name: CLUSTER_STRATEGIES
  value: POSTGRES
- name: FLY_APP_NAME
  value: testing
- name: SLOT_NAME_SUFFIX
  valueFrom:
    fieldRef:
      fieldPath: metadata.labels['apps.kubernetes.io/pod-index']

and I am seeing goodies like these in the pod logs:

ip set from EnvVar 'ip' '10.244.2.115'
No RELEASE_DISTRIBUTION set. Using 'name'
No RELEASE_NODE set. Using 'testing@10.244.2.115'
14:36:21.327 [info] Elixir.Realtime.SignalHandler is being initialized...
14:36:21.328 [notice] SYN[testing@10.244.2.115] Adding node to scope <Elixir.Realtime.Tenants.Connect>
14:36:21.328 [notice] SYN[testing@10.244.2.115] Creating tables for scope <Elixir.Realtime.Tenants.Connect>
14:36:21.328 [notice] SYN[testing@10.244.2.115|registry<Elixir.Realtime.Tenants.Connect>] Discovering the cluster
14:36:21.328 [notice] SYN[testing@10.244.2.115|pg<Elixir.Realtime.Tenants.Connect>] Discovering the cluster
14:36:21.328 [notice] SYN[testing@10.244.2.115] Adding node to scope <users>
14:36:21.328 [notice] SYN[testing@10.244.2.115] Creating tables for scope <users>
14:36:21.328 [notice] SYN[testing@10.244.2.115|registry<users>] Discovering the cluster
14:36:21.328 [notice] SYN[testing@10.244.2.115|pg<users>] Discovering the cluster
14:36:21.328 [notice] SYN[testing@10.244.2.115] Adding node to scope <Elixir.RegionNodes>
14:36:21.328 [notice] SYN[testing@10.244.2.115] Creating tables for scope <Elixir.RegionNodes>
14:36:21.328 [notice] SYN[testing@10.244.2.115|registry<Elixir.RegionNodes>] Discovering the cluster
14:36:21.328 [notice] SYN[testing@10.244.2.115|pg<Elixir.RegionNodes>] Discovering the cluster
14:36:21.328 [warning] Replica region not found, defaulting to Realtime.Repo
14:36:21.333 [info] Running RealtimeWeb.Endpoint with cowboy 2.10.0 at :::4000 (http)
14:36:21.333 [info] Access RealtimeWeb.Endpoint at http://testing.fly.dev
14:36:21.333 [notice] SYN[testing@10.244.2.115] Adding node to scope <Elixir.PostgresCdcStream>
14:36:21.333 [notice] SYN[testing@10.244.2.115] Creating tables for scope <Elixir.PostgresCdcStream>
14:36:21.333 [notice] SYN[testing@10.244.2.115|registry<Elixir.PostgresCdcStream>] Discovering the cluster
14:36:21.333 [notice] SYN[testing@10.244.2.115|pg<Elixir.PostgresCdcStream>] Discovering the cluster
14:36:21.334 [notice] SYN[testing@10.244.2.115] Adding node to scope <Elixir.Extensions.PostgresCdcRls>
14:36:21.334 [notice] SYN[testing@10.244.2.115] Creating tables for scope <Elixir.Extensions.PostgresCdcRls>
14:36:21.334 [notice] SYN[testing@10.244.2.115|registry<Elixir.Extensions.PostgresCdcRls>] Discovering the cluster
14:36:21.334 [notice] SYN[testing@10.244.2.115|pg<Elixir.Extensions.PostgresCdcRls>] Discovering the cluster
14:36:21.347 [info] [libcluster:postgres] Connected to Postgres database
14:36:21.349 [notice] SYN[testing@10.244.2.115|registry<Elixir.Realtime.Tenants.Connect>] Node testing@10.244.1.101 has joined the cluster, sending discover message
14:36:21.349 [notice] SYN[testing@10.244.2.115|registry<Elixir.Extensions.PostgresCdcRls>] Node testing@10.244.1.101 has joined the cluster, sending discover message
14:36:21.349 [notice] SYN[testing@10.244.2.115|pg<Elixir.Realtime.Tenants.Connect>] Node testing@10.244.1.101 has joined the cluster, sending discover message
14:36:21.349 [notice] SYN[testing@10.244.2.115|pg<Elixir.Extensions.PostgresCdcRls>] Node testing@10.244.1.101 has joined the cluster, sending discover message
14:36:21.349 [notice] SYN[testing@10.244.2.115|registry<users>] Node testing@10.244.1.101 has joined the cluster, sending discover message
14:36:21.349 [notice] SYN[testing@10.244.2.115|pg<users>] Node testing@10.244.1.101 has joined the cluster, sending discover message
14:36:21.349 [notice] SYN[testing@10.244.2.115|registry<Elixir.RegionNodes>] Node testing@10.244.1.101 has joined the cluster, sending discover message
14:36:21.349 [notice] SYN[testing@10.244.2.115|pg<Elixir.RegionNodes>] Node testing@10.244.1.101 has joined the cluster, sending discover message
14:36:21.349 [notice] SYN[testing@10.244.2.115|registry<Elixir.PostgresCdcStream>] Node testing@10.244.1.101 has joined the cluster, sending discover message
14:36:21.349 [notice] SYN[testing@10.244.2.115|pg<Elixir.PostgresCdcStream>] Node testing@10.244.1.101 has joined the cluster, sending discover message
14:36:21.349 [notice] SYN[testing@10.244.2.115|pg<Elixir.RegionNodes>] Received DISCOVER request from node testing@10.244.1.101
14:36:21.349 [notice] SYN[testing@10.244.2.115|registry<Elixir.Extensions.PostgresCdcRls>] Received DISCOVER request from node testing@10.244.1.101
14:36:21.349 [notice] SYN[testing@10.244.2.115|pg<Elixir.PostgresCdcStream>] Received DISCOVER request from node testing@10.244.1.101
14:36:21.349 [notice] SYN[testing@10.244.2.115|pg<users>] Received DISCOVER request from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<users>] Received DISCOVER request from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.RegionNodes>] Received DISCOVER request from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.PostgresCdcStream>] Received DISCOVER request from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.Extensions.PostgresCdcRls>] Received DISCOVER request from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.Realtime.Tenants.Connect>] Received DISCOVER request from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.Realtime.Tenants.Connect>] Received DISCOVER request from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<users>] Received ACK SYNC (0 entries) from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.Extensions.PostgresCdcRls>] Received ACK SYNC (0 entries) from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.Realtime.Tenants.Connect>] Received ACK SYNC (0 entries) from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.Extensions.PostgresCdcRls>] Received ACK SYNC (0 entries) from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.Realtime.Tenants.Connect>] Received ACK SYNC (0 entries) from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.RegionNodes>] Received ACK SYNC (0 entries) from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.Realtime.Tenants.Connect>] Node testing@10.244.0.127 has joined the cluster, sending discover message
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<users>] Node testing@10.244.0.127 has joined the cluster, sending discover message
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.RegionNodes>] Node testing@10.244.0.127 has joined the cluster, sending discover message
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<users>] Node testing@10.244.0.127 has joined the cluster, sending discover message
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<users>] Received ACK SYNC (0 entries) from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.RegionNodes>] Node testing@10.244.0.127 has joined the cluster, sending discover message
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.PostgresCdcStream>] Node testing@10.244.0.127 has joined the cluster, sending discover message
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.PostgresCdcStream>] Node testing@10.244.0.127 has joined the cluster, sending discover message
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.Extensions.PostgresCdcRls>] Node testing@10.244.0.127 has joined the cluster, sending discover message
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.Realtime.Tenants.Connect>] Node testing@10.244.0.127 has joined the cluster, sending discover message
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.Extensions.PostgresCdcRls>] Node testing@10.244.0.127 has joined the cluster, sending discover message
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.RegionNodes>] Received ACK SYNC (1 entries) from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.PostgresCdcStream>] Received ACK SYNC (0 entries) from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<users>] Received DISCOVER request from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.PostgresCdcStream>] Received ACK SYNC (0 entries) from node testing@10.244.1.101
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.Realtime.Tenants.Connect>] Received DISCOVER request from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.Realtime.Tenants.Connect>] Received DISCOVER request from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.RegionNodes>] Received DISCOVER request from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.Extensions.PostgresCdcRls>] Received DISCOVER request from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.Extensions.PostgresCdcRls>] Received DISCOVER request from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<users>] Received DISCOVER request from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.RegionNodes>] Received DISCOVER request from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.RegionNodes>] Received ACK SYNC (0 entries) from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.Extensions.PostgresCdcRls>] Received ACK SYNC (0 entries) from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<users>] Received ACK SYNC (0 entries) from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<users>] Received ACK SYNC (0 entries) from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.RegionNodes>] Received ACK SYNC (1 entries) from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.PostgresCdcStream>] Received DISCOVER request from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.Realtime.Tenants.Connect>] Received ACK SYNC (0 entries) from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.PostgresCdcStream>] Received ACK SYNC (0 entries) from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.Extensions.PostgresCdcRls>] Received ACK SYNC (0 entries) from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.PostgresCdcStream>] Received DISCOVER request from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|registry<Elixir.Realtime.Tenants.Connect>] Received ACK SYNC (0 entries) from node testing@10.244.0.127
14:36:21.350 [notice] SYN[testing@10.244.2.115|pg<Elixir.PostgresCdcStream>] Received ACK SYNC (0 entries) from node testing@10.244.0.127