Support Nixery deployments inside of Kubernetes clusters
tazjin opened this issue · 5 comments
Several open questions around caching, internal addressing etc. remain here - more information coming soon.
There does not seem to be a clean way of doing this that works across all Kubernetes clusters using something like NodePort.
Some discussions with people revealed that there's also no good definition of what "clean" means in this context.
Here's a very raw list of issues:
- using a Kubernetes-internal
Servicedoes not work in most cases becausekube-dnsand cluster-internal routing is not available to nodes NodePortworks, but it has a limited port range (30000-31000 by default) and Docker requires TLS certificates on registries - in a simple internal case, one might end up pulling images from a registry named something likelocalhost:30822which is not pleasant- GCP-specific tools that make this cleanly achievable (e.g. internal zones in Cloud DNS) aren't necessarily available elsewhere
I will set up guides and examples for how to do this that focus specifically on GKE. Other users might want to contribute equivalent guides for other Kubernetes hosters.
Exciting times!
My personal infrastructure repository (tazjin/depot) now features a Nixery deployment inside of Kubernetes (see here). A similar setup to this should be documented in the Nixery docs for people to experiment with.
The network setup basically involves a private DNS zone for the GCP VPC (in which the cluster pool(s) run) that points nixery.local towards an internal LB which directs traffic to Nixery. There's some room for improvement here (using "real" domains with certificates for instance) still.
Exciting times!
The link seems to be broken. I’d be really interested in learning how to get nixery running on GKE. Thanks!