Support artifact uploads from lava jobs
Opened this issue · 3 comments
for some lava tests it would be really nice to also attach artifacts; Think of e.g. seperate (bigger log files), image captures being made or really anything else that's not suitable to stick into just plain text output.
Lava does not have inbuild functionality for this; For reasonable reasons; The suggested solution in the lava documentation is to have a seperate service running and upload to it using curl with some token; Which is not a bad suggestion per se, but does require a seperate service while we already have gitlab being able to do artifact management :).
So a suggested solution for this would be to have the gitlab runner also be a http service that can receive such files, caches them locally while the job is running and when finishing pushes them into the general gitlab artifacts.
To both authenticate and identify which job an upload is for the runner could generate a random dynamic endpoint for each job to receive uploads which is only valid during a job run. This endpoint can be communicate to lava via the templating mechanism we already have available, leaving it up to job writer on how to cooperate this into the yaml job.
Implementation detail wise; we could simply integrate axum as a http server (to stay nicely in the hyper universe) and for simplicity we can assume there is a frontend server that does SSL termination so there is no need to manage ssl certificates. Similarly the base URL for where to reach a given runner should be provided by runner configuration.
From an operational point of view exposing a http enpoint is quite annoying.
The runner at the moment is a headless service, so on k8s no need to deal with services, ingresses, certificates, etc. That alone would be annoying but not a big deal. What I am actually worried about is how to map the incoming upload to the correct runner replica when there are multiple instances. At the moment we run the runner with replicas > 1 deployment, this may require some sort of custom proxy that forwards a request for /lavarunner/6f95bcdd66-4mdlx to the lava-gitlab-runner-6f95bcdd66-4mdlx pod, as I am not aware of any native kubernetes solution.
The other annoyance is that at the moment the DUT only need to talk to the dispatcher, while this would likely require some more global network routing.
Perhaps the mechanism to upload files from DUT could be handled in the lava job itself & uploaded to the lava job ? Then you can retrieve it from LAVA server using the gitlab-runner ?
Perhaps the mechanism to upload files from DUT could be handled in the lava job itself & uploaded to the lava job ? Then you can retrieve it from LAVA server using the gitlab-runner ?
Lava doesn't have a mechanism for dealing with that; as per https://docs.lavasoftware.org/lava/publishing-artifacts.html the suggested method for publishing files externally is to use curl to do a POST to an external service
Having it go to the lava server is annoying as well because then we have to deal with more GC and artifacts there. So it just ends up complicating things more