Semantic Versioning of canonical data.
rbasso opened this issue · 13 comments
As questioned in #671, we still don't have defined how to version the canonical-data.json files in the new schema. I think we should strive to get something like SemVer.
Edit: Below we well keep updated the proposed semantic versioning resulting from the discussion.
Current proposal
Adapting from SemVer 2.0.0 ...
Given a version number MAJOR.MINOR.PATCH, increment the:
- MAJOR version when you make incompatible API changes
Only changes in the test suite in a fundamentally incompatible way should
change the MAJOR version:
- A new "property" (test type).
- Renaming a "property".
- Insertion, deletion or renaming of keys in the test data object.
- Changing the type of one of the test data keys.
MAJOR changes would break even well-behaved test generators, that are not affected by test grouping/nesting.
- MINOR version when you add functionality in a backwards-compatible manner, and
Non-breaking changes that alter the meaning of the test suite, making previously
passing solutions possibly fail, or failing solutions fail at a different spot, should bump the MINOR version:
- Adding or deleting test cases.
- Changing the test cases inputs and/or outputs.
- Changing the test cases ordering.
MINOR changes would never break well-designed test generators, because the test-generation logic remains exactly the same.
- PATCH version when you make backwards-compatible bug fixes.
Anything else that do not change the meaning of the tests should bump the
PATCH version:
- Regrouping/"Renesting" test cases without changing test cases ordering.
- Changing descriptions or comments.
- Changing keys' ordering or formatting (would result in an equivalent JSON file).
Maybe ..
Given a version number MAJOR.MINOR.PATCH, increment the:
MAJOR version when you add new test cases or change input/output values,
MINOR version when you change descriptions or other descriptive fields, and
PATCH version when you fix small typos or other trivial things.
The MAJOR number could be reserved for changes in the structure of the tests, things that would probably break any well designed test generator.
Good point. Than we move the new test cases and input/output changes to MINOR and put the cosmetic changes (rewrite/typos/etc) of descriptive fields into PATCH .
I'm writing a proposal so that we can discuss the details. There are a lot of corner cases...
This is what I got until now, @behrtam . I tried do adapt SemVer to the canonical data.:
Given a version number MAJOR.MINOR.PATCH, increment the:
- MAJOR version when you make incompatible API changes
Only changes in the test suite in a fundamentally incompatible way should
change the MAJOR version:
- A new "property" to be tested.
- Insertion, deletion or renaming of keys in the test data object.
- Changing the type of one of the test data keys.
These are the changes that would break even a well behaved test generator.
- MINOR version when you add functionality in a backwards-compatible manner, and
Non-breaking changes that alter the meaning of the tests, making previously
passing solutions possibly fail, should bump the MINOR version:
- Adding or deleting test cases.
- Changing the test cases inputs and/or outputs.
- PATCH version when you make backwards-compatible bug fixes.
Anything else that do not change the meaning of the tests should bump the
PATCH version:
- Reordering, regrouping, renesting test cases. (not sure if this should be MINOR)
- Changing descriptions and comments.
- Changind key orders and formatting (would result in an equivalent JSON file).
What do you think about it?
I just updated the proposal, @behrtam, because test cases' ordering seems to be important in Exercism, so reordering would change the meaning of the test suite, implying a bump in the MINOR version.
I wondered if I might have summarised it as:
- Major: You might need to rewrite your test generator.
- Minor: You will definitely not need to rewrite your test generator. You probably want to rerun your test generator though.
- Patch: Rerunning the test generator is probably not necessary.
Under this thought, I would put a reordering in minor.
Then again, some tracks might want to rerun their generator even on patch-level changes, so maybe I have the demarcation between minor and patch incorrect.
Great summarization, @petertseng.
Under this thought, I would put a reordering in minor.
I completely agree!
Then again, some tracks might want to rerun their generator even on patch-level changes, so maybe I have the demarcation between minor and patch incorrect.
What matter here is conveying information to the tracks about what changed:
- MAJOR - Test data structure changed. (API break)
- MINOR - Test data changed / test cases reordered. (test content changed)
- PATCH - Non-API-breaking, non-test-content-changing modifications (test organization, cosmetic changes)
- PATCH - json-content-irrelevant changes (json-irrelevant changes)
The only drawback here is being forced to bump the PATCH on json-irrelevant changes, but they are so uncommon that I don't see the need to add one more level of versioning just for that, which would make everything even more confusing.
@rbasso @petertseng I agree with the proposed setup. Looks great.
@rbasso Have we documented the proposed semantic versioning scheme somewhere?
I don't think we did, @ErikSchierboom. I just added the policy label and updated the first comment to reflect what was discussed.
Hmmm, I would to see it documented somewhere, or at least a link to this document be added in a logical place (but where). What do you think?
Considering that the documentation for the new schema is in the README.md, it should probably be inserted there. Maybe a reference in the CONTRIBUTING.md would be necessary...not sure.