Support docs tests
Closed this issue · 12 comments
In #20 I've reworked the "test" workflow to be a chain of containers rather than a single preparer container execution, but to keep scope under control I limited it to "preparer, then submitter." Just the same, the next natural step is to support an arbitrary number of doctest containers that run between the two.
Implementation
The general design is that 0..n doctest containers will mount the workspace volume and perform an arbitrary test on the source markup at CONTENT_ROOT or on the JSON envelopes at ENVELOPE_DIR. The doctest containers used for each build will need to be whitelisted in the system install. The doctests that run on a specific build will be configured within content-repositories.json. If the doctest completes without finding any issues, the doctest container produces no output and exits with a status of 0. If the doctest finds critical issues, it reports diagnostic messages to stderr and exits with a status of 1. If it finds nonfatal warnings instead, it reports diagnostic messages to stderr and exits with a status of 2.
Configuration
Configuring which doctests execute on which builds should be done within content-repositories.json. I'm guessing that we're likely to have sets of doctests that are almost always run in tandem, so it may be useful to have a way to configure them in groups, as well.
Here's a quick sketch of what the schema might look like:
{
"builds": [
{ "kind": "github", "project": "getcarina/getcarina.com", "audits": [ "common", "quay.io/deconst/doctest-containerscontainerscontainers" ] },
{ "kind": "github", "project": "rackerlabs/docs-developer-blog", "audits": [ "common" ] }
],
"doctestGroups": {
"common": ["quay.io/deconst/doctest-spellcheck", "quay.io/deconst/doctest-jekyll-categories"]
}
}Doctest Ideas
I'm going to start by porting @ktbartholomew's jekyll-audit script to a doctest container that we can run on the devblog. That'll let me work out any kinks in the protocol before we have any new doctest logic itself to debug too.
I'd ❤️ to hear any ideas for other doctests from the writers! I will note that this is an area that can very quickly oscillate between "completely trivial" and "would require strong AI," so I can't promise that I can make everything work, and I certainly won't be replacing the need for a human editor. Robots can only get us so far, but hopefully I can take care of some of the more mechanical, tedious stuff.
- Spell checking, with specific additions for Rackspace document vocab and terminology find-and-replace. Difficulty estimate: low
- Service name capitalization. Difficulty estimate: trivial
- Double words. Difficulty estimate: trivial
- Link validity check. Difficulty estimate: trivial
- Active voice. Difficulty estimate: high
- Sentence case for chapter and section headings. Difficulty estimate: trivial
- Present tense predominantly. Difficulty estimate: high
- Unclear pronoun antecedents: avoid "it", "this", "that", and "there" as sentence subjects. Difficulty estimate: high
/cc @meker12, @kmsholcomb
@smashwilson The InfoDev Quality team has created a quality checklist here: https://github.com/rackerlabs/docs-workstream/blob/master/quality/quality-checklist.md. We've discussed adding tests to check for issues mentioned in the checklist.
Was just talking to @RobinJoe about coming up with a list of requirements. Will checkout the Jekyll audit, and then our team can give you a list of desired tests for consideration/discussion. Will work to get this to you by next week.
Related issue: https://github.com/rackerlabs/docs-workstream/issues/14 (spell check, link check, content requirement check) -- Guessing that the jekyll audit might include link check and spell check.
The Jekyll audit only does basic checking on the categories of blog posts. I believe there is a link checker that we run against the entire site periodically. Offhand, I'd think that we'd want link checking done both at prepare-time and periodically for existing content.
Thanks for the links, that's a good place for me to start 🙇 No rush on the ideas; I wanted to give you a chance to figure out what would be most useful while I'm still working on build issues.
Hi @smashwilson, So the TL;DR for automation: spell checking specific for Rackspace document vocab, service name capitalization, and double words.
The Spell Check rationale - we want to save some time on quality checking and pull request reviews. I've made pull requests with typos, and reviews for finding and correcting those takes a bit of time out of the review cycle. With the Quality checklists that the docs team has introduced, I'm concerned that this will take more time to complete and manage. Spell checks performed during build testing would help compensate by automating some of the spell checking.
Specific words is one part of the check, but we would also like to include errors such as double words ('the the' 'to to' 'is is') which can appear in writing, and specifying a check for lower case project names (neutron, nova, swift, cinder, for example).
Brian Moss @kallimachos has started work adding a spell checker as an extension of the sphinx build we use in the Private Cloud docs repo. There is a forthcoming pull request for this addition.
On the time point is it possible also to record as a metric time between Pull Request appear on github and merging?
Fantastic, thank you!
I've read through your comments and the quality checklist and edited a checklist of ideas into the issue itself, along with a back-of-the-envelope estimate of the complexity involved with each. I'm guessing that we can find quite a few more in the style guide (... and also I definitely need to spend some time reading it for myself 😉)
Brian Moss @kallimachos has started work adding a spell checker as an extension of the sphinx build we use in the Private Cloud docs repo. There is a forthcoming pull request for this addition.
Well, remember that we have significant amounts of content that isn't in Sphinx. Part of the point of Deconst is that it's agnostic about the native format of the content repository. As much as possible, doctests will run against the final output of the content. Otherwise, we'd need to implement all of these at least three times.
Still: if you have work toward a spellchecker already done, I'd love to see it! It could likely be adapted to run as a general Deconst doctest without much trouble.
Recently attended a presentation about MapBox documentation. They talked quite a bit about their content testing and have also open-sourced their content testing tool for automatically reviewing their technical documentation: https://www.mapbox.com/blog/retext-mapbox-standard/
Perhaps we can use or learn from their implementation.
@smashwilson Please add Kelly to this repo so she can follow along on the testing conversation. Thanks! Well she can follow along but I don't seem to be able to @ mention her.
Recently attended a presentation about MapBox documentation. They talked quite a bit about their content testing and have also open-sourced their content testing tool for automatically reviewing their technical documentation: https://www.mapbox.com/blog/retext-mapbox-standard/
Perhaps we can use or learn from their implementation.
👍 Quite possible. I'll have to look at how strongly they're coupled to Jekyll and Markdown.
@smashwilson Please add Kelly to this repo so she can follow along on the testing conversation. Thanks!
I've sent her an invite to the Deconst org, but because this is a public repository she should be able to read, comment, and subscribe to issues regardless (and the @-mention I did in the issue itself should have sent this to her GitHub notifications.)
I have been getting the notifications, thanks!
I read the MapBox article, and from that and my own previous experience at "programming" and testing content checkers, I can say a couple of things. I agree that focusing on vocabulary is probably the easiest automated checking, but context and exceptions still make it tricky! @smashwilson, I can supply you with a comprehensive list of words to look for and suggested substitutions (by combining all the various lists and guidelines in the style guide and checklists). I can also look through the artifacts from a past project that I did at my last company and suggest some additional tests we might want to try. Thanks for including me! I'd love to completely do away with some of the tedious things I mark all the time!
On the time point: is it possible also to record as a metric time between Pull Request appear on github and merging?
@RobinJoe: Yes, but a doctest isn't the right place to do this. You'd want to pull that as a report from the GitHub API. Is there a requirement for this? It'd probably be its own issue in rackerlabs/dxeng.
@smashwilson Okay, the rational behind my thoughts there was to determine if the added quality checklist slows review time. It's not a priority, though, and I'm not experienced enough working with APIs. Might be a good place to dive in though.
@smashwilson, @meker12, @RobinJoe: I've created a spreadsheet that could be used as the basis for a doc test. The first column is terms to flag, and the second column contains the terms to use instead, or guidance. Tests like this can return too many "false positives" (or would it be "false negatives"?) to be helpful, so I tried to limit the first column to terms that should never be used or that are very often misused (with some abbreviations and punctuation thrown in for good measure). The list tops out at 700, though, and I have no idea how much of a drag that would be on build processing. I could pare down the list if needed. Also, I don't know how the substitutions or guidance would be presented to the writer, but I tried to keep the entries in the second column fairly succinct. Again, if necessary, I could change that. However, I wanted to get a list to you to show the kind of terms (and how many) a test could look for.
Please review the spreadsheet (at https://docs.google.com/a/gapps.rackspace.com/spreadsheets/d/1T5SFguVZY4zd4Jw5DberTYShsIbyrXcKhcx90XCkGGk/edit?usp=sharing) and let me know if you have any feedback. Thanks!