Ticket handling access privileges

[celskeggs]

When we eventually have maintainers of the Hyades cluster who are not root admins, they will need a way to inspect user namespaces to help answer tickets.

We could just give them read access on everything in every user namespace except secrets, but that seems like a suboptimal solution. We should consider putting together something more secure.

For example, we could have a tool that could integrate with the ticketing system and the authentication system; when a maintainer submitted a form with a ticket number from a user, it would automatically grant read-only access to non-secret kubernetes objects in that user's namespace to said maintainer. It would send an email to the ticket, maybe including the user or maybe just as a comment, and would automatically revoke the permissions some N days later. Higher levels of permission, like read access to secrets and read-write access to the namespace, might also be facilitated through the same tool (and record the access grant in the same way) but require approval by a root admin.

We might also want to make a "segmented privilege" model for root admins, so that they have the option of using system:masters group access, but by default have restricted access so that they can inspect state of kube-system and related namespaces, but can't modify anything or touch user namespaces. They'd have to either "break glass" and use system:masters or go through the same process that regular maintainers have to go through to gain access.

To be clear -- I'm not sure exactly what the right solution is here, so these are just initial thoughts. It's also the case that we're a long way off from having anyone helping with Hyades tickets who isn't trustworthy enough to also get root, so this is not remotely urgent. (And if we need to stop-gap this, we can easily grant a blanket "read everything but secrets" permission to our maintainers.)