Implement Snapshot Testing/Something Similar to ensure either nothing is changing or changes are intended (phew!)

Question

Implement Snapshot Testing/Something Similar to ensure either nothing is changing or changes are intended (phew!)

Closed this issue 6 months ago · 7 comments

This is really moving!!

We have a lot of cooks in the kitchen now and maybe we should be building in some sort of testing on the final outputs before merging into main? This way as we make changes the Pull Requests can help us catch little updates that might change the data examples in untoward ways??

I'm not sure if it is possible to set up snapshot testing here to check that outputs match previous outputs. Or if there is some other method to help us capture the new static outputs and compare against the old outputs.

@ddsjoberg @cicdguy I think this is possible just never done anything like this outside of an R package.

Answer 1 · 2023-12-18T20:00:17.000Z

hey hey @bms63 ! Which final outputs are you referring to?

Answer 2 · 2023-12-18T21:17:35.000Z

I think the final dataset in each example or dataset used to display the output.

For example in https://pharmaverse.github.io/examples/adam/adpc.html

just checking that nothing changed at the end for adpc_xpt when a new change is applied in the repo.

Answer 3 · 2024-03-07T15:04:57.000Z

@bms63 i'm hesitant to close another issue 😬 but i'm thinking this one might be overkill for an examples site?? i'd get such testing for something like the admiral templates, but these are just examples for people to play around with so the main CI check i care about each time would be with the latest CRAN packages does the example fail or fall over. what do you think? sorry, i'm being a bit strict today trying to channel my inner product owner!

Answer 4 · 2024-03-07T20:29:21.000Z

all good to close!

My only hesitation is we don't have any detection for things being deprecated in our examples. So functions could be outputting warnings without us knowing??

Answer 5 · 2024-03-08T08:46:29.000Z

fair point! i'll try and keep an eye as will be checking periodically for any new releases as need to maintain the posit cloud package installations anyway, so if there are i could always do a quick test run when i make that update.

Answer 6 · 2024-03-08T13:02:44.000Z

Maybe a Pull Request Template could be created to help give us some reminders on things to check??

Answer 7 · 2024-03-08T13:13:10.000Z

Good idea - feel free to make an issue for this (and PR of course if you had time, but no pressure!)