Ensure canonical-data.json updates are propagated to all tracks
robkeim opened this issue · 10 comments
There's been a lot of great work done recently to add the extract/update/create canonical-data.json files for all of the exercises.
Currently, the updates don't create any corresponding link in the different language repositories to let the maintainers know that they should update their tests to correspond to the standard.
Is there some sort of way that we could automatically create an issue in all of the corresponding language tracks each time the canonical data changes for one of the exercises?
As a track maintainer that would let me track which exercises I need to update without having to try to keep my own todo list or manually open issues for each updated exercise.
I was thinking something like what was done with ensuring every track had a hello world exercise:
https://github.com/exercism/todo/issues/12
There was a bit of previous discussion on when we should emblazon tracks at exercism/discussions#104 (comment). It would be useful to get clarity on the threshold at which we consider a change significant enough to warrant it. How worried are we about issue fatigue?
I imagine that we would consider a new canonical-data.json file to be significant enough to warrant emblazoning all the tracks, right?
Potential counterpoints:
- But if the canonical-data.json file matches all tracks' existing tests anyway, why bother?
- But it's not always the case! Remember I said in #510 (comment) I think we want the canonical tests to represent what we want the tests to be, which is not necessarily what they already are
- And if track maintainers don't get a notification on a new canonical-data.json file, how will I tell the difference between "there was no significant change" versus "somebody just forgot to emblazon"?
The below is no solution, but at some point in the past I needed to check whether various tracks are up-to-date - master...petertseng:up-to-date has scripts which compare the last modify time of select tracks' test files against the last modify time in x-common. I say this is no solution because the track maintainer still has to remember to run it from time to time.
I'm still against bulk blazoning.
- Actively maintained tracks can watch the progress of things through x-common.
- Active tracks can create scripts to check for canonical data updates. (I'm sure someone could come up with a git one-liner for this.)
- Inactive tracks will just have dozens(hundreds?) of issues piling up in them.
Track specific opt-in blazoning could be OK I guess, I don't want to stop tracks that want automatic issues created from getting them.
Except x-common can be overwhelming. I got about 50 emails from just 3 exercism repos today, mostly x-common. I'd rather have an issue show up in my track, where I'm far more likely to pay attention to it.
I'm with @IanWhitney and @robkeim. Yes, sending an update each time something changes in x-common can be tiresome, but the alternative (manually keeping track of x-common) is not a whole lot better. This is partially due to the fact that canonical data updates are mixed with other types of updates, like description modifications and such.
I was thinking that perhaps I could create a website that could keep track of changes to canonical-data files automatically? With potentially a diff view to see what changed.
@ErikSchierboom and @IanWhitney I think that @Insti's suggestion of having an opt-in blazon could be a good compromise. It would have to be automated to ensure that checkins aren't missed though. In terms of minimizing the amount of issues that would get created, is there a way to programatically determine which changes are "important" vs description modifications?
@Insti if I play devil's advocate for a minute, what's the purpose of creating and maintaining the canonical data if we're not ensuring it gets integrated in the different tracks? My understanding was that this canonical data is used to ensure there's cross track consistency.
I do get an influx of e-mail (but I use the terminal, so it is a few keypresses to processes the hundreds of repository related email every day. But to coordinate I use the web based notifications and zenhub's todo (That last link will work properly if you use zenhub, otherwise it is "ignored".)
... This is partially due to the fact that canonical data updates are mixed with other types of updates, like description modifications and such...
Semantic versioning of test data would be great, and could also be used in a track-specific solution to match its versions against x-common.
Semantic versioning of test data would be great, and could also be used in a track-specific solution to match its versions against x-common.
Absolutely! Great idea.