asciidoc-py/asciidoc-py2

testasciidoc could compare output with a reference snapshot

Closed this issue · 8 comments

florentx commented

(from #25)

I discovered this limitation: doing tests/testasciidoc.py update then tests/testasciidoc.py run with the same revision is slow and dumb: it controls that the output is identical when running asciidoc twice.
This is the intended behavior as explained here: http://asciidoc.org/testasciidoc.html#_rationale
It is acceptable when running the tests manually while developing new features but it's not compatible with the continuous integration approach.
We need to think more about this.

I see 2 solutions so far:

  • running tests/testasciidoc.py update and committing the result
  • or computing an MD5SUMS file of the expected output, storing it in the tests/data directory, and implementing a new flag tests/testasciidoc.py run --compare-md5 which could be used for Travis-CI.
florentx commented

I've measured the test output, it's 196 html and xml files with a total size of 3 MB.
I would be in favor of storing this expected output with the source code.

The solution based on MD5 sums is limited: it saves some space on your hard disk, but when the test fails you don't have a diff with the expected output.

mojavelinux commented

A possible solution here is to use the "release" attachments on GitHub to store a snapshot that is tagged to a specific AsciiDoc version. That solves the large file problem and it clearly correlates the expected output with a git revision. You can add attachments to any previous release, so this is something we could do retroactively.

elextr commented

Other than the mechanics I agree that storing the results is the way to go.

It would be preferable to not store it in the standard repository. Even a separate repository under the asciidoc organisation would be ok if Travis can still access it.

@mojavelinux can github tag the snapshot to a commit? otherwise I think travis would then fail up until the next release every time a change really needed to have a new set of test data.

mojavelinux commented

Another repository might be a better alternative (e.g., asciidoc-doctest-output). The releases can only be associated with real git tags, as far as I know. What we could do is keep the results in a separate git repository, but then attach the results to the git tag in the main repository when the release is cut. That way, the results are archived along with the AsciiDoc release (but you can always get the results from the canonical results repository).

elextr commented

Thinking about it some more, I don't think that a separate repository will work unless we can link individual commits on the main repo to test result sets on the test repo.

Any change that needs new tests (because of a new feature or because of a bug fix) will fail in CI because it doesn't have new results to test against, unless it can be associated with a changed set of test results. And so the whole of asciidoc would fail CI from that commit until release, which is unacceptable.

Also there would be no way of associating pull requests with changed test data since its in another repo, and so pull requests that needed new test results would always fail CI. And its the ones that make changes that are significant enough to need new tests that we really want to see pass.

Considering that git compresses file contents, and, if I understand correctly, packs changes when pushed, it may not be such a major issue to have it in the main repo. Being mostly a text file, the test results are likely to compress well, and only have small changes each time.

This way the test results are also part of pull requests that need them, so they should be able to show that they pass tests and make the CI more useful.

So on further consideration I would say lets put test results in the asciidoc repo.

florentx commented

Definitely, I agree that it's much simpler to store them in the repository.
The output is 196 text files (html and xml), with an average of 16 kB per files.

 kB    file pattern
====  ==============
 368  tests/data/*-docbook.xml
 304  tests/data/*-html4.html
1148  tests/data/*-html5.html
1144  tests/data/*-xhtml11.html
 132  tests/data/slidy-example-slidy.html

They should not vary a lot, after PR #31 is applied.

aerostitch commented

Hi,

The python 2 implementation of asciidoc (which is what this repo is about) is deprecated.
Please move your issue to the python 3 implementation at https://github.com/asciidoc/asciidoc-py3 if still relevant on that newer implementation and close this issue.

Thanks for your help,
Joseph