Non-ASCII characters (?) are not preserved by `cmark.exe`

[kenny-evitt]

I ran cmark Test.md --out Test.html where Test.md contains the following:

# Header – Something Else

Was the en dash character preserved or was it replaced with the Unicode replacement character?

In Test.html, the en dash character (in the header) is replaced with the Unicode replacement character (EF BF BD as hex).

[Knagis]

Does your file has UTF-8 preamble (byte order marks)? Without them I guess the encoding is detected as ascii and thus the dash char is not recognized.

[kenny-evitt]

@Knagis I am guessing it does not. Notepad++ seems to think it's encoded in "ANSI", tho it displays the en dash character fine.

I was going to ask "Is there is any reason not to just read everything as UTF-8 given that it's backwards compatible with ASCII/ANSI?" but I tested the relevant CommonMark.NET code in LINQPad and it already is (defaulting to) using UTF-8.

I figured out that this is failing because the file is ISO-8859-1 encoded, the Encoding.Default encoding on my computer.

I changed the encoding of my file to UTF-8 but in the HTML file created by cmark, the en dash is output as:

–

This seems to be because cmark doesn't output a full valid HTML document. After I added a character encoding declaration, it rendered fine.

Would you be interested in a PR to extend the console app to support outputting full valid HTML documents?

[Knagis]

Would you be interested in a PR to extend the console app to support outputting full valid HTML documents?

I think that it should already support this if you create two small .html files - one for the start of the file and one for the end and then run

cmark.exe header.html input.md footer.html --out result.html

[kenny-evitt]

@Knagis Ahhhhh – now it makes sense why it combines all of the output into a single file! HTML is interpreted 'raw' (unless some text was not inside an element) and so essentially just passed-thru. Thanks!