Figure out automated testing of Sphinx code snippets in SymbiFlow tutorials

Question

Figure out automated testing of Sphinx code snippets in SymbiFlow tutorials

Closed this issue 5 years ago · 3 comments

CI testing of SymbiFlow tutorials

Brief explanation

Find a way to ensure the tutorials for SymbiFlow are tested in CI.

Expected results

There exists a relatively easy way to tag code snippets to be executed in order, possibly a way to supply some extra commands on top of what's exposed to the user. Might be needed to find a way to substitute variables.
Alternative (less nice) - in every repo with a tutorial there is an easily executable script (or scripts) mirroring the commands in the tutorial, co-maintained with the repo. A clear methodology is devised to lead the way.

Detailed Explanation

The problem with tutorials in fast moving projects is that they became outdated very easily. We need to figure out a way to test them so that we can keep the tutorials up to date.

Knowledge Prerequisites

Sphinx
Python
Possibly bash and knowledge about CI

Answer 1 · 2019-04-23T22:30:41.000Z

Kind of like http://www.sphinx-doc.org/en/master/usage/extensions/doctest.html

sphinx.ext.doctest – Test snippets in the documentation — Sphinx 3.0.0+/275745845 documentation

Answer 2 · 2019-04-23T22:36:27.000Z

Although the aim of the above is different. For our needs, it should be possible to extract code snippets from the docs programmatically using docutils. Need to figure out:

if there is a way to add totally custom attributes we could abuse (exec-before, exec-after (to run stuff before/after/in-between snippets), name (includes order I think, e.g. name-order_of_snippets_in_series) , subst-variables (to substitute variables in code)
how hard it is to extract specific code blocks that are somehow marked
once we have that, write a simplest possible proof of concept with say 3 snippets executed with something

Answer 3 · 2019-05-06T07:14:36.000Z

OK, did it the other way. Used .. literalinclude:: to fetch content from automated scripts in the RISC-V Getting Started Guide (see https://github.com/riscv/risc-v-getting-started-guide/blob/master/source/getting-zephyr.rst). This seems like a good option, so far it's working fine. This will allow us to write CI scripts which run the scripts that are being used in the docs.

GitHub
riscv/risc-v-getting-started-guide
The official RISC-V getting started guide. Contribute to riscv/risc-v-getting-started-guide development by creating an account on GitHub.