support markdown as data files

[vitiral]

probably make them the "default" in the future through art init setting them as the default in settings.toml

A data markdown file would look like:

# REQ-markdown
- partof:
  - REQ-0
  - REQ-[foo, bar]
- done: this is the optional `done` field
- plugin-attr: this entire section is actually parsed as yaml, so
    you can do pretty much whatever you want until two or more `#` on their own line.
##
The above `##` is what separates the "data" from the "text" fields. The nice thing is 
that markdown doesn't actually render an empty header!

Everything else (like this and the sentance above) is parsed as text.

## Subheader
You can have sub-headers in the text.

# SPC-markdown
partof:
- REQ-0

##
Here is some text about SPC-markdown

# REQ-top
If there are no attributes (partof, etc), you can ommit the `##` and just write text. The parser will not complain.

I've been thinking about this, and the parser for this is actually fairly trivial. Re-writing markdown should be relatively easy as well.

Rendered

REQ-markdown

• partof:
  • REQ-0
  • REQ-[foo, bar]
• done: this is the optional done field
• plugin-attr: this entire section is actually parsed as yaml, so
  you can do pretty much whatever you want until two or more # on their own line.

The above ## is what separates the "data" from the "text" fields. The nice thing is
that markdown doesn't actually render an empty header!

Everything else (like this and the sentance above) is parsed as text.

Subheader

You can have sub-headers in the text.

SPC-markdown

partof:

• REQ-0

Here is some text about SPC-markdown

REQ-top

If there are no attributes (partof, etc), you can ommit the ## and just write text. The parser will not complain.

[rubdos]

Hmm, I like it. Probably needs some testing to see whether it can contain all features you have in the current format, but I don't see that as a problem.

[vitiral]

I've changed the "end data" block from --- to ###. ### doesn't get rendered at all in the end output, but is also a nice separator.

I'll keep experimenting, but I think this is the right approach.

[vitiral]

One thing this breaks is that you won't be able to use a top header anywhere.

I.e. this will not be allowed:

# REQ-foo

# Some header for REQ-foo

It would be possible to support this, but honestly it would be cluttered. Note: this IS backwards compatible -- it would just mean that when a user tried to export to markdown it would throw an error (with a helpful message) and they would have to manually fix. Single headers would still be supported in toml.

[vitiral]

@rubdos other than not supporting single headers, I don't see there being any issue. Between the # REQ-name and ### there can exist any arbitrary yaml, so you can put literally anything there. Even better, yaml sees # as a comment, so it's not like ### would have ever been parsed as real data anyway.

Speaking of yaml... I should think about maybe using a different format. I like the fact that yaml renders in markdown quite well though

- name1: value
- name2:
    - value1: 
        This is some really long text. This is some really long text.
        This is some really long text. This is some really long text.
        If I write a very long line it gets wrapped nicely. If I write a very long line it gets wrapped nicely. If I write a very long line it gets wrapped nicely. If I write a very long line it gets wrapped nicely. If I write a very long line it gets wrapped nicely. If I write a very long line it gets wrapped nicely. 
    - value2: this is not as long

• name1: value
• name2:
  • value1:
    This is some really long text. This is some really long text.
    This is some really long text. This is some really long text.
    If I write a very long line it gets wrapped nicely. If I write a very long line it gets wrapped nicely. If I write a very long line it gets wrapped nicely. If I write a very long line it gets wrapped nicely. If I write a very long line it gets wrapped nicely. If I write a very long line it gets wrapped nicely.
  • value2: this is not as long

Hmm

I actually think github might render comments slightly different than .md files. I expected the text block to be merged (without newlines).

[vitiral]

On the other hand, maybe sticking with toml is the way to go. The stuff between # REQ-foo and ### is actually data, it's probably good for it to look like data!

REQ-foo

partof = [
"REQ-foo",
"REQ-bar",
"REQ-baz",
]
done = "this is done"

Here is some text

Hmm... I don't like that though, especially that the indents are deleted.

[einsweniger]

my two cents:
I like reStructuredText better, but markdown is more common.
I'm not sure if I'd describe markdown as trivially parseable, is there no crate for that?

toml isn't very pretty, and does not have many features, but it's easy to maintain and easy to parse.
md is pretty, but: there are several dialects and extensions around (eg. man 5 pandoc_markdown) and AFAIR allows to litter HTML throughout the whole document.

I'd vote to implement a very small and restrictive subset of md, to keep it maintainable.
Maybe allow for an external parser to generate html, that can be fed back into the system.

[vitiral]

I have quite a bit of negative feelings against reStructuredText. Markdown has it's faults, but reStructuredText has way too much syntax for my taste... it's not very "clean".

Interestingly, I actually like the fact that you can insert html in markdown -- it is a feature I use in the web-ui to great effect.

Also, I'm not suggesting I parse the markdown exactly... the basic design would be two regexes:

• The header line is (approximately):

#\s(REQ|TST|SPC)-\w+\s*

I would then pull lines into a buffer until I hit one of:

• the "end of attributes" line

###+\s*

• another header line
• the end of the file

If I hit another header line, the buffer==text. If I hit the "end of attributes" line then I parse until a header line and that is my text.

The parsed attributes would be desrialized using a yaml library, so I wouldn't have to do much there.

Re-writing the files would be almost as easy. I don't see a lot of technical risk here.