Validate the test data version as part of a travis build
Closed this issue · 20 comments
Thanks to the work of @rbasso (see exercism/problem-specifications#602, exercism/problem-specifications#625, exercism/problem-specifications#673, etc) and many more, we now not only have a schema but also a test data version.
In exercism/problem-specifications#524 @robkeim started to think about how to propagate changes from x-common to the tracks. In this issue @ErikSchierboom wrote:
Yes, sending an update each time something changes in x-common can be tiresome, but the alternative (manually keeping track of x-common) is not a whole lot better.
So, I thought about a way that doesn't generate a lot of mails, issues or notifications, but makes sure that at least every time someone actually is working on a test file it is up to date.
I started to work on a script (check-testversion.py) for the python track that checks the versions for all exercises that are part of a travis build against the newest version in the x-common repo, a bit like a linter or stylechecker. It still needs a lot of cleanup and testing, but it at least proves that the basic idea is working.
The basic concept is ...
- get a list of changed files
git diff --name-only $TRAVIS_COMMIT_RANGE - get the newest version from
canonical-data.jsonthroughhttps://api.github.com/repos/... - get the actual version from a comment in the test file
# test @ version: 1.0.0
(maybe it should be excluded into a separate JSON file as part of the exercise directory) - compare the versions and output some infos
As a test I changed the anagram version back from 1.0.1 to 1.0.0 in this commit: behrtam/python@3dc7acd
Which resulted in a failed build as expected: https://travis-ci.org/behrtam/xpython/jobs/219306945#L163
I don't know if this will finally be integrated into the python track or if this is a good idea at all, but I thought maybe some parts of this could also be interesting for other tracks.
I like this idea a lot. I'd love to hear if any other tracks try it out and what you think.
Nice. I have https://github.com/petertseng/x-common/tree/up-to-date going but haven't gone to the length of putting it in the CI for the respective tracks yet (probably shouldn't, since why would Ruby code go in non-Ruby tracks?)
That gives colour coded output of whether things are up to date, like (sorry, can't show the colour coding on Github markdown)
bracket-push: 1.0.0 -> 1.1.0
hamming: 1.0.0 == 1.0.0
anagram: 1.0.1 == 1.0.1
largest-series-product: 1.0.0 == 1.0.0
sieve: 1.0.0 == 1.0.0
react: 1.0.0 == 1.0.0
leap: 1.0.0 == 1.0.0
xruby is working towards something similar. We now have the test data version in our generated test files:
xocaml is also doing this for generated tests, with a comment at the top of each test.ml file:
(* Test/exercise version: "1.0.0" *)
I'm trying to figure out the best cadence for running a script that ensures exercises are up to date. I thought of a few choices:
- The maintainers simply have to remember to run the script on a manual but regular basis. This of course adds no complexity to the CI. The obvious disadvantage is the human element - what if someone forgets?
- CI checks changed files only. This leaves open the possibility that a file will go unchanged and therefore unchecked for a long time. How to deal with this? Combine it with the above strategy?
- CI checks all files. A big disadvantage here is that a previously-passing build may turn into failing even though nothing has changed in the repo being checked (only x-common).
I would be interested to hear any of y'all's experiences on that.
I would not want to enable the 3rd option, it would take the focus off the PR I'm working on.
Would it be possible for a central script to raise /update a single issue in each stream repo for version mismatches?
@stevejb71 we have a script that can do that and we discussed enabling something like that here, although there was concern about the additional burden for track maintainers.
Can it be done on a per track basis? I'd be fine with it for the ocaml track.
I wouldn't mind issues being created / updated in the Delphi track as long as the issue pertained to an exercise my track currently offers. Tracks should be able to opt out of the feature.
If exercism/meta#97 comes to fruition, I would prefer that solution to CI checks. While both could be done together, to me using both might feel redundant.
I agree that the bot would be a better solution—it will not slow down CI or break the build.
Not sure about having the test version in a comment, why not in the config.json for each exercise?
How about putting the test version inside a file in the .meta dir?
If test version lived in a separate file in .meta, I'd find it pretty easy to accidentally forget to update it when updating an exercise. A comment near the top of the file containing the tests has the property of showing up as soon as I open that file in a text editor.
That would be more visible, but also would require changes to existing files in every exercise in most tracks.... :-\
@coriolinus if you automate the change, then it wouldn't matter where it is, right?
@kytrinyx I was talking about automating the change to insert the version comment without considering maintenance. You're quite right: if you insert the version string automatically, it doesn't matter where it is. However, I believe that my earlier point still stands: after the big update to insert tracking version strings, it does still matter where it lives; putting it near the top of the file most likely to be edited makes it more likely to be remembered than putting it in an unrelated file.
@petertseng also brought up an interesting point in a different discussion: there may be track-specific canonical locations where version strings are expected. Rust, for example, expects to find version info in Cargo.toml in the project root; it would not surprise me if other tracks now or in the future also had project-specific canonical version locations. Adding versioning information elsewhere would require either keeping the two version strings synchronized, or depreciating the track-specific location in favor of the exercism-specific one.
The utility of having a bot such as proposed in exercism/meta#97 would outweigh the hassle of taking either of those courses, but a really good bot might be configurable to look in the track-specific location anyway.
@coriolinus Thanks for the clarification.
I think the bot is the best first step, and having the bot look for version information in a configurable location would be a nice add-on.
We've imported this issue to the https://github.com/exercism/exercism.io repository.