Ghost does not support clustering or multiple dynos (CDN/HTTP-cache required to scale)
kevinansfield opened this issue · 16 comments
Hey @cobyism 👋 I've had a look at this Ghost/Heroku setup and unfortunately it promotes some practices that will result in broken Ghost installations.
Ghost does not support clustering in any form due to in-memory caches that will result in each instance getting out of sync with each other - there can only be 1 instance of Ghost for each site.
Can you please remove any mention of clustering, scaling up of dynos (unless advising not to do so!), and the use of node-cluster?
Ghost does not support clustering in any form due to in-memory caches that will result in each instance getting out of sync with each other - there can only be 1 instance of Ghost for each site.
Yikes, this is good to know!
Once a day in Heroku (and probably many other PaaS too) the dynos cycle. Heroku deploys a new one and starts sending requests to it before tearing down the old one.
Is that likely to cause sync issues? Any ideas on whether it would be possible to make this a safe process?
I started using ghost-on-heroku because it's more convenient for me than running a cloud server or docker, though I'd like to make sure it's going to be stable.
(P.S. @cobyism I reached out about helping with maintenance. Hit me up!)
Once a day in Heroku (and probably many other PaaS too) the dynos cycle. Heroku deploys a new one and starts sending requests to it before tearing down the old one.
Is that likely to cause sync issues?
It is possible for sync to be introduced if the admin is being actively used whilst the cycle is in progress. One example - Ghost's routing map is stored in an in-memory cache which gets updated when changes are made to published resources via an internal events system, if you publish a post at the moment a new instance is being booted up but the PUT request hits the hits the old instance then the new instance will return a 404 for that post once requests start to be sent to it.
Huh. I wonder if there could be a solution. Would it be possible for the server to respond to a signal (or to an API endpoint) to trigger a rebuild of that cache? It could be the deployer's responsibility to fire the signal after an appropriate interval. It's not perfect, because things could still look out of sync during that interval, but it's much better than staying out of sync for the life of the process.
Even if clustering isn't on the immediate roadmap for self-hosting users, it's useful to have graceful failover so the single-node option is stable.
@kevinansfield, are you really saying that Ghost cannot scale beyond a single, single-threaded Node.js process?
That's exactly what he said 😉
I found this related thread: https://forum.ghost.org/t/external-in-memory-cache-option/1140
Given that I don't think multi-dyno can be reliably supported.
I'm interested in trying to find a stable workaround for a single-dyno install. I'm using Ghost for the admin interface, with the API / Gatsby, so avoiding corruption or bugginess in the admin interface is important; HA / scaling out, not so much.
are you really saying that Ghost cannot scale beyond a single, single-threaded Node.js process?
@mars A single, single-threaded Node.js process has been more than sufficient to serve 1-5 of HN's frontpage stories and every day for the last 6 years alongside literally tens of billions of requests without ever falling over as a result of load.
Does your code not scale properly without lots of dynos?
Thanks for your reponse @JohnONolan
I’m very happy to hear that Ghost can squeeze so much performance out of a single thread of javascript. That is clearly a testament to Ghost’s engineering practices and history to have so much confidence about this architecture.
Heroku’s dyno scaling is not just for “scaling code” but also to support rolling deployments without downtime and to load balance across healthy dynos if a server/dyno suffers hardware failure.
I always prefer operating a service that can scale horizontally, separate from whether or not the “code scales properly”.
How do you deploy new versions of a Ghost server without downtime or deal with hardware outages if Ghost does not support multiple servers?
In that forum thread (https://forum.ghost.org/t/external-in-memory-cache-option/1140) Hannah says Ghost is expected to be run:
with a cache or multiple load balanced caches in front of it to handle scaling
That is a very different story than saying a single Node process serves all the traffic.
@paulmelnikow it sounds like the right solution given this new information would be to run a CDN like Fastly in front of a single Ghost process.
It would be great to see a PR removing node-cluster, putting the Fastly addon in app.json, and updating the README with the “do not scale beyond one dyno” guidance.
I've taken a look at solving this problem, but I see Ghost no longer documents the basis for this implementation, starting Ghost from its npm module. There used to be docs for this at https://docs.ghost.org/v1.0.0/docs/using-ghost-as-an-npm-module but cannot find that anymore.
Without someone actually putting effort into remaking this Heroku deployment for state-of-the-art Ghost (currently version 2.14), I don't feel that it's healthy or considerate us to keep this project available @cobyism. It's leading folks to problems 😔 when they could instead use the Ghost platform and have a great experience.
I just removed node-cluster, added a "do not scale dynos" warning, and bumped up to the most recent 1.x release.
Are there other problems using the npm module? The changes in #157 are working fine for me, so it seems the npm module is still being published, even if it's not still recommended.
Glad to hear it still works @paulmelnikow. This is the second time the Ghost folks have dismissed this project (first-time), not wanting to support this kind of third party usage. Does not feel like a fruitful relationship anymore.
Gatsby is much better for running your own blog 👍 these days.
To clarify, @mars, this issue was opened by way of an olive branch as an offer of help and guidance. We had both discussed and intended to offer to take over the project and have it officially maintained by the Ghore core team.
Unfortunately the petty hostility in this thread put an end to that idea.
All the best
I did my best to understand the problem and contributed my time & energy making the changes you Ghost folks requested.
The thread speaks for itself 💔
@JohnONolan @kevinansfield, I'm jumping in here as a bystander / user of Ghost. I'm sorry to see this thread end poorly when it started with a really helpful suggestion from the Ghost team. I for one would love to see a mending of wounds and a re-engagement between this project and the Ghost team.