thought-machine/falco-probes

Determine whether a given Kernel Package name has been uploaded to the repository already

Closed this issue · 2 comments

VJftw commented

Our initial implementation of checking if a kernel probe is already uploaded is performed quite late in the build process as we have to obtain the KernelPackage including its sources in order to get the kernel probe name.
This doesn't really save us much time and means our builds are still quite resource intensive and long as they download a KernelPackage's sources each time.

Ideally, we can move this earlier in the process, to once we've obtained the list of KernelPackage names (operatingsystem.OperatingSystem.GetKernelPackageNames()) we can verify if a given KernelPackage name is already uploaded to the repository. In order to do this, we could store a mapping of Kernel Package Name -> Kernel Probe name in the release notes. e.g.

Kernel Package Probe
4.14.101-91.76.amzn2 falco_amazonlinux2_4.14.101-91.76.amzn2.x86_64_1.o

We could then use this mapping in our tooling to verify if we already have a Release asset of Probe for the given Kernel Package.

sHesl commented

Happy to take a look at this if no-one else is picking this up.

Guessing we want something like:

  • a RELEASE_NOTES.md generation tool (first slice can pretty much just be the table above, maybe with driver ver added as well)
  • a step for parsing the rows in the RELEASE_NOTES.md to determine if probes are already compiled/released so we can avoid recompiling them (reading from the release notes in the checked-out source code rather than from the API)
  • a step for appending newly compiled probes into the releases notes (either done in a 'smart' way by appending rows to the table when new probes are compiled, or a 'stupid' way where we just re-run the full RELEASE_NOTES.md generation tool after every job).
VJftw commented

Yes, please! I hadn't put much thought into it, but what you've suggested sounds 💯 . I'm beginning to lean a bit more towards the full generation way to start with in order to reduce our requests to the GitHub APIs as we're hitting limits at the moment (1000/hour).