Following kubernetes logs for all containers in all pods of a deployment.

Running ./follow-kube-logs.y -n my-namespace -p my-deployment -d logdir will have the following effect.

Will create directory logdir under the current directory
It will create a subdirectory in logdir for each pod running in the deployment my-deployment in namepace my-namespace - this directory has the name of the running pod and contains all the log files per followed container.
Get the label set that identifies the pods that are running in this deployment; spawns a process that follows the logs of each of the containers for each pod of the deployment; this process gathers the logs for that container into a file in the log directory of the pod, while the script is running.
The script then waits and asks for the user to press enter, whereas it will kill the spawned processes and stop the logging.
While the logging is proceeding: once a second the program scans the deployment for new pods and stopped pods. For new pods we also create a subdirectory with the name of the pod in logdir. A log file for each container will be created that follows the logs of the container.
Events like starting/stopping of a pod are displayed on standard output.

The purpose of this script is to be a more lightweight solution then to use prometheus/graphana for viewing your deployment logs, as it is sometimes easier to grep through the logs, as compared to writing elaborate prometheus queries. Also by following the logs for a set time period you will have all of them, and you will not have to deal with log rotation.

Following the logs for all containers of a statefull set

Running ./follow-kube-logs.y -n my-namespace -s my-statefull-set -d logdir will have the following effect.

Follows all containers of all pods in statefull set my-statefull-set in namespace my-namespace, all this is done by the same process as described above.

Following the logs for all containers of a replica set

Running ./follow-kube-logs.y -n my-namespace -r my-replica-set -d logdir will have the following effect.

Follows all containers of all pods in replicaset set my-replica-set in namespace my-namespace, all this is done by the same process as described above.

Installation.

Download the script by following this link and make it executable; you need to have python3 on the system.
You can optionally install argument autocompletion for the bash shell by running the following command: follow-kube-logs.py -b >>$HOME/.bashrc

help file of the script

The scripts help text:

usage: follow-kube-logs.py [-h] [--namespace NAMESPACE]
                           [--deployment DEPLOYMENT] [--stset STATEFULSET]
                           [--rset REPLICASET] [--out OUTDIR]
                           [--kubectl KUBECMD] [--trace] [--complete-bash]
                           [--complete]

This program starts to follow the logs of containers in all pods of a
kubernetes deployment/replicaset/statefulset. The output is written to a file
per container. The script then waits for user input, logging is stopped once
the user has pressed enter.

optional arguments:
  -h, --help            show this help message and exit

log  pods/containers in either one of deployment/replicaset/statefuleset:
  --namespace NAMESPACE, -n NAMESPACE
                        optional: specify namespace of deployment (default: )
  --deployment DEPLOYMENT, -d DEPLOYMENT
                        name of deployment (default: )
  --stset STATEFULSET, -s STATEFULSET
                        name of statefull set (default: )
  --rset REPLICASET, -r REPLICASET
                        name of replica set (default: )
  --out OUTDIR, -o OUTDIR
                        mandatory: name of output directory (default: )
  --kubectl KUBECMD, -k KUBECMD
                        optional: name of kubectl command (default: kubectl)
  --trace, -x           optional: enable tracing (default: False)

suport for bash autocompletion of command line arguments:
  --complete-bash, -b   show bash source of completion function (default:
                        False)
  --complete, -c        internal: used during code completion (default: False)
  --kubectl KUBECMD, -k KUBECMD
                        optional: name of kubectl command (default: kubectl)

What I learned from this

It is best to extract some items from the kubectl with in-built json path querries, shorter and more robust than grepping for the output or running jq to filter stuff out.
For deployments: the deployment controller adjusts the set of running pods based on the pods labels; i am using the following jsonpath expression to extract the labels kubectl -n NAMESPACE get deployment DEPLOYMENT_NAME -o jsonpath='{.spec.selector.matchLabels}
Using the following command to list all the pods of a deployment, and to make a listing that includes the pod name and all of its containers; kubectl -n NAMSPACE get pods -l POD_SELECTORS -o jsonpath="{range .items[*]}{' '}{.metadata.name}{range .spec.containers[*]}{' '}{.name}{end}{'\\n'}{end}"

Similar projects

It turns out there is Stern - but that one is dumping all logs to the same terminal.

Also, maybe i should have put some more effort into learning this prometheus query stuff, instead of writing this scripts; however right now it's enough for me, and it pays my bills ;-)