closeio/sync-engine

Make sync processes compatible with autoscaling

Opened this issue · 0 comments

Modern cloud infrastructures can auto scale compute resources to reduce costs and reap other benefits.

Accounts are currently assigned to a specific host by setting its sync_host property to the hostname the process is running on. This makes it very difficult to change the underlying compute resource which is likely a Docker container in modern infrastructures. Ideally the sync process wouldn't have to know or care what host/container it was running on.

Two potential options:

  1. Leverage the existing .desired_sync_host (see this script) to reassign accounts after a scaling event.
  2. Remove the concept of .sync_host completely from accounts and have them dynamically assigned to running sync processes. Use something like a redis hashring to assign accounts in an orderly fashion.