Make sync processes compatible with autoscaling
Opened this issue · 0 comments
jkemp101 commented
Modern cloud infrastructures can auto scale compute resources to reduce costs and reap other benefits.
Accounts are currently assigned to a specific host by setting its sync_host
property to the hostname the process is running on. This makes it very difficult to change the underlying compute resource which is likely a Docker container in modern infrastructures. Ideally the sync process wouldn't have to know or care what host/container it was running on.
Two potential options:
- Leverage the existing
.desired_sync_host
(see this script) to reassign accounts after a scaling event. - Remove the concept of
.sync_host
completely from accounts and have them dynamically assigned to running sync processes. Use something like a redis hashring to assign accounts in an orderly fashion.