github/octocatalog-diff

exported resources mocking

b4ldr opened this issue · 3 comments

b4ldr commented

I wonder if it would be possible and if so where would one start, to try and mock, in some way the puppetdb api specificity for exported resources and possibly queries issued with dalen/puppet-puppetdbquery.

The scenario i have in mind is being able to use octocatalog-diff in CI pipeline which would preform the following steps

  • install puppet
  • download the current main/production/master branch in one dir
  • download the change branch/commit into another dir
  • run octocatalog-diff on the two branches to report on diffs.

This* mostly works unless a manifest needs to retrieve data from puppet i.e. exported_resources. for the purposes of testing we generally dont care what the content of the exported resources are we just want need something stable to be available at run time so we can compile the catalogue consistently without error. As such i wonder if there is currently a way, oir if you have considered implementing some way to mock the puppet api endpoint and fake the response to exported resources quires.

Im not sure how this might be implemented but it could be as simple as creating a yaml file with a bunch of valid resources to uses for puppetdb queries or dynamicly scan the puppet code base and generate exported resources on the fly based on some sane defaults.

*we currently uses a home grown tool to preform a similar function to octocatalog-diff however we also need to make changes to support puppet 6 and as such are considering switching to this tool. in our current set up we just spin up an empty puppetdb instance and populate it with some junk data

ahayworth commented

That's a good set of questions, really...

So at GitHub, we run jobs on every PR that (basically) do the following:

  • ensure a representative set of catalogs compile (via octocatalog-diff)
  • run a bunch of rspec-puppet tests (and we use octofacts to use realistic facts in those tests)
  • run integration tests that build shared OS base images
  • run linters

We don't actually use the output of octocatalog-diff as a CI job unless you ask for it (as an alternative to running a puppet noop). If you do ask for it, we kick off a CI job that does essentially what you describe (and there are options to do it against a single host, or multiple, etc).

I think we might have just retired some last vestiges of exported resources, but we do call out to puppetdb as part of those optional CI jobs to report on diffs. And more or less everything works fine - we don't have a general problem with the "stability" of the data in-between runs.

So I suspect I don't understand fully what use-case you're talking about? There may be some magic happening in our test setup to keep the stability that you're referencing - there's a lot of shell scripts and I stopped spelunking after awhile 😆.

Can you give me another example of what you're talking about? Maybe we didn't use exported resources in the same way that you do, so maybe I don't know what problems one might run into? 😆

ahayworth commented

Also - if you do decide to update your tool... the bulk of what needs to be done for puppet 6 support is here: https://github.com/github/octocatalog-diff/pull/226/files#diff-2f75d312af42343fc9eefc3b6295eb78cfad93005ca0c7ebfe77015e775d5e6bR61-R122 - the kicker, of course, is that you have to upgrade to at least Puppet 6.5.0. The rest of the PR is mostly test changes, and supporting another resource type that I think was introduced with Puppet 6. So the work to update your tool might be simpler with Puppet 6.5.0.

Of course we'd love for octocatalog-diff to work for what you need, though! 😄

b4ldr commented

Thanks for the response. What we have sounds vary similar to what you have described. including the fact that

  • our tool is also only run on demand, you can see examples via the Jenkins job.
  • we don't need puppetdb for facts as we use yaml files for the pcc run (rspec-puppet-facts for the rspec test)

however we do use exported resources as well as dalen/puppet-puppetdbquery query_* functions (which may be more of an issues then exported resources).

Currently the images that run theses jobs have a standalone puppet master configured with a local puppetdb instance. the puppetdb instance is populated as CI jobs are scheduled i.e. the first job to run, runs against an empty puppetdb. As such any catalogue compilations which require information from puppetdb can fail or have nondeterministic affects leading to strange issues and hacky workarounds

We can fix our manifest so that the are resilient to puppet queries returning no data however this can have significant implications to the final compiled catalogue meaning not all code paths get tested.

As such it would be nice to have something like rspec-puppet-facts/octofacts. but for exported resources or dummy resources which could be queried via dalen/puppet-puppetdbquery

I appreciate this is an edge case and writing it out makes me thing its more a problem with dalen/puppet-puppetdbquery and not exported_ resources. Thus making it even more edge however the machination of others who may have consiedred this are most welcome even if the bug is determined invalid/wont fix :)

As to further development, first thanks for the link, however for us moving to puppet 6.5 is probably some time away that said my current thinging is that when we do octocatalog-diff will be the preferred candidate to replace are internal tooling