Parse and export directive options, too
sscherfke opened this issue · 18 comments
Status quo
ReST/MyST directives can have options, e.g.:
.. code-block:: python
:caption: example.py
print("Hello, world!")'``{code-block} python
:caption: example.py
print("Hello, world!")
'``The options are already part of the directive regex (rest, myst (although myst use the --- form and not the double colon form which seems to be recommended by MyST).
These options are not part of a capture group, though, and can thus not be used for more advanced stuff.
Use case
I have examples that show a config file, a python script that loads and prints the settings and a console code block that invokes the script and displays its output (similarly to doctests, but for bash). Here is an example: https://typed-settings.readthedocs.io/en/latest/#example
I use the :caption: filename.ext option for directives to indicate that the contents of a code block should be written to that file instead of being executed (that option also has the nice side-effect of being displayed in the docs :)).
For this to work I needed to copy and adjust lexers and parsers from Sybil. This is generally okay for me, but maybe parts of that functionality (or all of it) might be a nice addition to Sybil.
Here is the current code for this: https://gitlab.com/sscherfke/typed-settings/-/blob/main/conftest.py
Proposed changes
- Modify the code-block regexes to also capture the directive parameters/options.
- Modify the MyST code-block regex to use the double-colon format (because that is what seems to be recommended) (in addition or instead of the
---format). - Modify the lexer(s) to process the options string and convert it to a dictionary (e.g. ,
options = {"caption": "example.py"}) and make them part ofLexedRegion.lexemes. - Maybe add an
optionsproperty toRegionwhich is a shortcut toregion.lexemes.get("options", {}). - Add a hook (or something) to
CodeBlockParserthat can be used to trigger user defined functionality (e.g., depending on the directive's parameters). The current behavior is the default and fallback behavior.
If you are interested in adding this to Sybil, I would start working on a PR.
I think I've got a reasonable amount of this done already on master...skip_improvements
Can you have a look and let me know what you think?
I skimmed through the changes.
They seem to fix my requirements 1–3 (parse :key: value options from myst/rest directive and add them to the lexemes).
The evaluator stack seems to solve my requirements for 5: I can instantiate a CodeBlockParser for console or toml (or the PythonCodeBlockParser) and push an evaluator that looks for :caption: file.ext and raises a NotEvaluated if it's not there, which would cause the default evaluators (e.g., doctest evaluator for PythonCodeBlock) to kick in. Is that correct?
So, coming back to this testfixtures supports what you're after but with different spelling. It's a great library, but then I'm biased since I'm its author ;-)
I think master supports what you're after, but I'm not sure I'd use the evaluator stack, just a specialised CodeBlockParser.
Can you have a look at current master and let me know if you still feel anything is missing?
@sscherfke - last call on this, I'm planning a 6.0.0 release sometime in the next day or few...
I’ll try to test the new changes and give you feedback, but had no time for this yet.
Wasn't there directive option parsing for the MyST lexers, too, in the feature branch skip_improvements? In the current master, it's only there for ReST.
It would be cool if that feature was also available for Myst. This is the regex I am currently using:
MYST_START_PATTERN = (
r"^(?P<prefix>[ \t]*)"
r"```\{(?P<directive>code-block)}\s*"
r"(?P<arguments>[\w-]+\b)$\n"
r"(?P<options>(?:\s*\:[\w-]+\:.*\n)*)?"
)No, but you're right that it's missing and I've implemented in 89d3eee
Could you have a look at master now and see if that meets all your needs?
@sscherfke - I'll take the thumbs up as "all good now", please let me know if you spot anything else and we can re-open this.
It's a thumbs up for "I’ll test it and report back" ;-)
Okay, I found two issues:
- I think the options need to be forwared to the
Regionso that they are accessible in a parser. It needs to be added here: https://github.com/simplistix/sybil/blob/master/sybil/parsers/abstract/codeblock.py#L56
Like this (but as an actual class attribute ;-)):r = Region(lexed.start, lexed.end, lexed.lexemes['source'], self.evaluate) r.options = lexed.lexemes.get("options", {}) yield r
- The options in myst blocks must be enclosed with
---, but the MyST docs seem recommend options without---: https://myst-parser.readthedocs.io/en/latest/syntax/roles-and-directives.html – I posted a regex that is able to parse this above.
I can provide a PR as base, if you like :)
- Both styles are already supported:
sybil/tests/samples/myst-lexing-directives.md
Lines 21 to 38 in 93eb470
- f6d8772 shows how I'd currently do the kind of thing you're after, but I'm now wondering in
LexedRegionactually needs to exist...
- I need to access the options in
self.evaluate(), so overridingself.__call__()as in that test example does not work for me, I guess: https://gitlab.com/sscherfke/typed-settings/-/blob/main/conftest.py?ref_type=heads#L213 - You're right. A mistake on my side :)
I'm not sure doing what you're doing in self.evaluate() is right here; you're essentially picking between two evaluators based on an option in the ReST or MyST directive, so you should code it as such rather than trying to do both in a since evaluate method.
However, as of 59045ff I think you can do either. Let me know what you think.
Hi Chris, I got sick last week. Will try your changes asap and provide feedback.
Very sorry to hear that, hope you're feeling better soon! 6.0.0 is out now with all the changes in it, so please give it a spin and it me know how it goes :-)
I am happy now (and healthy again). :-)
Thanks a lot for your efforts! <3