Feature Request: Launch Kubernetes Jobs instead of local containers
Closed this issue · 6 comments
I hope I am asking in the right place. When one is running kubernetes, one already has a container orchestrator. It would be great to delegate running the containers that concourse needs to kubernetes. That way, concourse (running on kubernetes) could be better contained (not run as (docker) PID 0 with --privileged). Kubernetes also knows more about the cluster and could scale/move containers better.
I hope, that it's feasible to have a garden plugin that will launch kube jobs instead. We are willing in dedicating some effort to make the story above a reality. Please do advise on how to do that.
Hi there!
We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.
The current status is as follows:
- #138873035 Feature Request: Launch Kubernetes Jobs instead of local containers
This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.
Garden is just an API, there are existing multiple backends - including for windows and for linux using the open-standard runc container tech. If you wanted your own backend, for example a garden-kubernetes implementation, you would just need to implement the garden.Backend interface in this repo and then use server.New(type, path, gracetime, YOUR_BACKEND, logger) to create a server.
You could then just point concourse at that server and it will interact with it as any other worker. You can implement the backend however you like, including spawning kube jobs.
Of course, if you do this concourse will no longer be able to ensure that jobs are scheduled near to where their data is, because you will be hiding the orchestration, so things might not perform very well, and may not work at all (I think concourse assumes, for example, that it will be able to co-locate a baggage claim on the same machine as your jobs, which probably won't work right if the co-located garden is actually scheduling the container on a different host! @vito may be able to comment more on this though..
I can see two approaches. I think the first might be more feasible, but the second would be the better outcome. The second is more vague though.
A kubernetes worker could mount a hostPath (directly or via hostMount pvc) for any baggage claim and instruct kubernetes to schedule the containers on the same node (and also mount the hostPath). When scaling to more than one node, one would have to scale the kubernetes worker.
Or, kubernetes could be more tightly integrated into concourse so that containers accessing the same baggage claim get scheduled in the same pod. That would require that the kubernetes integration knows (up-front) which containers will be needed. Scaling to more than one node would be automatic (done by kubernetes itself). Rate limiting (so that the kube cluster does not get overloaded) has to be configured in kubernetes (with resource limits).
It's pretty late for me, so I hope what I wrote makes at least a little sense.
Do you have any advice?
This isn't really a question for Garden. If you want to change how concourse orchestrates containers I'd suggest chatting with the concourse folks about that. If you need any help writing a custom garden backend we're happy to help with that, feel free to come visit our slack channel (http://slack.cloudfoundry.org #garden).
Thank you. I'll ask around on concourse.