prometheus/client_golang

gocollector tests: Adjust CI to tell us when generated files were manually changed + refactor?

ArthurSens opened this issue · 7 comments

While working on #1559, we noticed that the generated files for the go collector can be manually changed and CI won't block the merge.

Let's investigate this and adjust :)

Yup, CI is NOT running this generative script.

We only check if metrics given by the runtime is what we expect in those generated _test files and then complain with instruction to potentially rerun that script. This helps us to figure out what metrics changed across Go version. That's the main goal of those files.

You might be right some CI check that would rerun this script and point mismatch on TOP of our existing test would let us know if someone shoveled some custom manual code to those files.

Do you mind adding clear acceptance criteria to description?

This might need proper refactor e.g. have one generated set in internals and separate logic to filter those for test purposes (it's just regex)

Ideally I would also have only one script that generates both package files 🤔

Acceptance Criteria

  • One script to generate test helpers for two places, we can move it to scripts dir or so.
  • Make sure our CI job for periodic script run and check actually works for BOTH files - it looks like it ran only one #1597 recently
  • Fix conflicts (visible in #1597) as we added manual changes to generated files in #1559 - those manual changed have to be either moved somewhere else or moved to generated content. This might need some test refactor
  • Have another CI job that checks for manual changes on those files.