Encoding issues when using markdown

Question

Encoding issues when using markdown

dominicroye opened this issue 3 years ago · 16 comments

Hi!

I have found an encoding issue when using Spanish symbols like accents; for instance, "Desde el comité organizador" looks like "Desde el comitÃ© organizador" with a markdown document encoded in UTF-8. I could find the origin in readChar() from read_text() function. If I use readLines it works fine for me. I am using Windows.

Have you any idea how to get a workaround?

Thank you very much! Congratulations on the package!

Best,

Dominic

Answer 1 · 2021-12-01T19:06:38.000Z

Hi @dominicroye!

Let's fix this! I don't want Spanish users to have a poor experience. Can you please provide me with some data to work with. Maybe an example as a REPREX? Also can you please tell me what locale you are working with?

Thanks, Andrew.

Answer 2 · 2021-12-02T11:14:30.000Z

Hi @datawookie.

Thank you for your reply. I made a smaller example on my GitHub: https://github.com/dominicroye/emayili_congress_aec

I just got it right on RStudio Cloud. It seems to be an issue with Windows as always. My Locale is:

> Sys.getlocale()
[1] "LC_COLLATE=Spanish_Spain.1252;LC_CTYPE=Spanish_Spain.1252;LC_MONETARY=Spanish_Spain.1252;LC_NUMERIC=C;LC_TIME=Spanish_Spain.1252"

Answer 3 · 2021-12-08T05:06:53.000Z

Hi @dominicroye, that example link you posted doesn't seem to work anymore. I'm looking at this issue now and it would be helpful to have an example to focus on. Thanks, Andrew.

I tried to recreate the problem based on the information that you have provided and I'm having trouble replicating it (even in my default locale).

library(emayili)
> 
> Attaching package: 'emayili'
> The following object is masked from 'package:graphics':
> 
>     text
> The following objects are masked from 'package:base':
> 
>     local, raw
Sys.getlocale("LC_CTYPE")
> [1] "en_ZA.UTF-8"
envelope() %>% render("Desde el comité organizador") %>% as.character() %>% cat()
> Date:                        Wed, 08 Dec 2021 06:22:38 GMT
> X-Mailer:                    {emayili}-0.7.0
> MIME-Version:                1.0
> Content-Type:                text/html;
>                               charset=utf-8
> Content-Disposition:         inline
> 
> <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
> <html><body><p>Desde el comité organizador</p></body></html>

I also tried with a Spanish locale.

library(emayili)
> 
> Attaching package: 'emayili'
> The following object is masked from 'package:graphics':
> 
>     text
> The following objects are masked from 'package:base':
> 
>     local, raw
spanish <- Sys.setlocale("LC_ALL", "Spanish")
Sys.getlocale("LC_CTYPE")
> [1] "Spanish"
envelope() %>% render("Desde el comité organizador") %>% as.character() %>% cat()
> Date:                        Wed, 08 Dec 2021 06:24:09 GMT
> X-Mailer:                    {emayili}-0.7.0
> MIME-Version:                1.0
> Content-Type:                text/html;
>                               charset=utf-8
> Content-Disposition:         inline
> 
> <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
> <html><body><p>Desde el comité organizador</p></body></html>

And the es_ES.utf8 locale.

library(emayili)
> 
> Attaching package: 'emayili'
> The following object is masked from 'package:graphics':
> 
>     text
> The following objects are masked from 'package:base':
> 
>     local, raw
spanish <- Sys.setlocale("LC_ALL", "es_ES.utf8")
Sys.getlocale("LC_CTYPE")
> [1] "es_ES.utf8"
envelope() %>% render("Desde el comité organizador") %>% as.character() %>% cat()
> Date:                        Wed, 08 Dec 2021 06:25:06 GMT
> X-Mailer:                    {emayili}-0.7.0
> MIME-Version:                1.0
> Content-Type:                text/html;
>                               charset=utf-8
> Content-Disposition:         inline
> 
> <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
> <html><body><p>Desde el comité organizador</p></body></html>

Unfortunately, I could not reproduce your precise locale on my Ubuntu machine, but I don't see a problem for these other Spanish locales (granted I am not an expert in Spanish or locales!).

Answer 4 · 2021-12-08T10:52:21.000Z

It is a strange thing, since your code works for me too, but not when I use the external markdown document.

envelope() %>% render("Desde el comité organizador") %>% as.character() %>% cat()
>Date:                        Wed, 08 Dec 2021 10:45:55 GMT
>X-Mailer:                    {emayili}-0.6.9
>MIME-Version:                1.0
>Content-Type:                text/html; charset=utf-8
>Content-Disposition:         inline
>
> <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
> <html><body><p>Desde el comité organizador</p></body></html>

Here using the markdown document you can find on GitHub:

> nombre = "Heinz"
> envelope() %>% render("msg.md") %>% as.character() %>% cat()
Date:                        Wed, 08 Dec 2021 10:48:37 GMT
X-Mailer:                    {emayili}-0.6.9
MIME-Version:                1.0
Content-Type:                text/html; charset=utf-8
Content-Disposition:         inline

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body>
<img src="https://aec2022.uvigo.es/wp-content/uploads/2021/03/logo-Congreso-AEC-v2022.png" width="150" style="float:block"><p>Estimada/o Heinz,</p>
<p>Desde el comitÃ© organizador del XII Congreso AEC queremos hacer un recordatorio del calendario del congreso para todos los que, a comienzos de 2020, nos hicisteis llegar un resÃºmen de vuestro trabajo previsto.</p>
<ul>
<li>
<p><em>Fecha lÃmite para el envÃo de resÃºmenes:</em><strong>21/01/2022</strong></p>
</li>
<li>
<p><em>ComunicaciÃ³n de aceptaciÃ³n de resÃºmenes:</em><strong>7/02/2022</strong></p>
</li>
<li>
<p><em>Fecha lÃmite de envÃo de trabajos:</em><strong>15/04/2022</strong></p>
</li>
<li>
<p><em>Fecha lÃmite para el envÃo de los trabajos corregidos:</em><strong>17/06/2022</strong></p>
</li>
</ul>
<p>Aunque presentÃ©is el mismo resÃºmen, recordad que tendrÃ©is que entrar de nuevo en la plataforma diseÃ±ada para las inscripciones y el envÃo de resÃºmenes y trabajos <a href="https://aec2022.uvigo.es/">https://aec2022.uvigo.es/</a>, y enviarlo antes del <em>21 de enero de 2022</em>.</p>
<p>Para consultar informaciÃ³n sobre el congreso podÃ©is entrar en nuestra pÃ¡gina web que se halla instalada en la web de la <a href="http://aeclim.org/actividades/congresos-aec/">AEC</a>, o tambiÃ©n podÃ©is escribirnos a la secretarÃa cientÃfica: <a href="mailto:congreso.12aec@usc.es">congreso.12aec@usc.es</a></p>
<p>Esperamos que, esta vez sÃ, podamos encontrarnos todos en Santiago en octubre del prÃ³ximo aÃ±o.</p>
<p>Muchas gracias por vuestra comprensiÃ³n.</p>
<p>Recibe un afectuoso saludo.</p>
<p><strong>El ComitÃ© Organizador</strong></p>
</body></html>

I guess the issue is maybe related to reading the markdown document?

Answer 5 · 2021-12-08T14:58:59.000Z

Aha! Okay, that helps. The document which you linked to on GitHub doesn't appear to be there anymore, @dominicroye. Can you check the URL, please? This is what you included above: https://github.com/dominicroye/emayili_congress_aec, but now it gives a 404 error.

Answer 6 · 2021-12-08T14:59:44.000Z

Also, now that I understand the problem better this should be a quick fix!

Answer 7 · 2021-12-08T16:30:11.000Z

The markdown document is this one https://github.com/dominicroye/emayili_congress_aec/blob/main/msg.md ;-)

Thank you!!

Answer 8 · 2021-12-08T17:27:42.000Z

Hi @dominicroye, cool! Thank you. Okay, can you please install this version and test?

remotes::install_github("datawookie/emayili", ref = "spanish-locale")

If that sorts the problem then I'll merge into master.

Thanks, Andrew.

Answer 9 · 2021-12-08T19:35:38.000Z

Also I couldn't open that document. Perhaps it's in a private repository?

Answer 10 · 2021-12-09T10:35:36.000Z

Hi! Sorry, yes the repository was private. Now you can access the markdown document. Below you can see that it still doesn't encode correctly.

> envelope() %>% render("msg.md") %>% as.character() %>% cat()
Date:                        Thu, 09 Dec 2021 10:33:17 GMT
X-Mailer:                    {emayili}-0.7.0
MIME-Version:                1.0
Content-Type:                text/html;
                              charset=utf-8
Content-Disposition:         inline

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body>
<img src="https://aec2022.uvigo.es/wp-content/uploads/2021/03/logo-Congreso-AEC-v2022.png" width="150" style="float:block"><p>Estimada/o Heinz,</p>
<p>Desde el comitÃ© organizador del XII Congreso AEC queremos hacer un recordatorio del calendario del congreso para todos los que, a comienzos de 2020, nos hicisteis llegar un resÃºmen de vuestro trabajo previsto.</p>
<ul>
<li>
<p><em>Fecha lÃmite para el envÃo de resÃºmenes:</em><strong>21/01/2022</strong></p>
</li>
<li>
<p><em>ComunicaciÃ³n de aceptaciÃ³n de resÃºmenes:</em><strong>7/02/2022</strong></p>
</li>
<li>
<p><em>Fecha lÃmite de envÃo de trabajos:</em><strong>15/04/2022</strong></p>
</li>
<li>
<p><em>Fecha lÃmite para el envÃo de los trabajos corregidos:</em><strong>17/06/2022</strong></p>
</li>
</ul>
<p>Aunque presentÃ©is el mismo resÃºmen, recordad que tendrÃ©is que entrar de nuevo en la plataforma diseÃ±ada para las inscripciones y el envÃo de resÃºmenes y trabajos <a href="https://aec2022.uvigo.es/">https://aec2022.uvigo.es/</a>, y enviarlo antes del <em>21 de enero de 2022</em>.</p>
<p>Para consultar informaciÃ³n sobre el congreso podÃ©is entrar en nuestra pÃ¡gina web que se halla instalada en la web de la <a href="http://aeclim.org/actividades/congresos-aec/">AEC</a>, o tambiÃ©n podÃ©is escribirnos a la secretarÃa cientÃfica: <a href="mailto:congreso.12aec@usc.es">congreso.12aec@usc.es</a></p>
<p>Esperamos que, esta vez sÃ, podamos encontrarnos todos en Santiago en octubre del prÃ³ximo aÃ±o.</p>
<p>Muchas gracias por vuestra comprensiÃ³n.</p>
<p>Recibe un afectuoso saludo.</p>
<p><strong>El ComitÃ© Organizador</strong></p>
</body></html>

Answer 11 · 2021-12-09T14:47:34.000Z

Thanks, @dominicroye. Weird because it works 100% fine for me... but I'm on Linux.

I'm puzzled though. You mentioned earlier that this works fine for you when using readLines(). The revised implementation on the branch that I pointed you to is now using readLines() rather than readChar(). Can you please show me the precise code which you ran to successfully load this content without encoding issues?

Are you sure that you installed from the branch I indicated? It's the spanish-locale branch, not master.

remotes::install_github("datawookie/emayili", ref = "spanish-locale")

Thanks, Andrew.

Answer 12 · 2021-12-11T06:09:03.000Z

Hi @dominicroye, can you please get back to me about ☝️? I'd like to resolve this issue. I'm waiting to get this sorted before pushing a new version to CRAN. Can you either get back to me soon (by Monday) or I'll have to postpone this until a later release. Thanks, Andrew.

Answer 13 · 2021-12-11T14:35:34.000Z

Hi @datawookie! I just tried the changes without change. But when I read directly the "msg.md" file with readLines and the argument encoding = "UTF-8" it worked fine.

readLines("msg.md", encoding = "UTF-8")

Could be related to the wrong encoding?

Many thanks!

Answer 14 · 2021-12-11T14:47:31.000Z

Ah, I think that's a pretty important detail you failed to mention!

Answer 15 · 2021-12-11T15:15:46.000Z

Hi @dominicroye, made a quick tweak based on your information ☝️. Can you please install again from the same source and test? Feeling optimistic. 😄 Thanks, Andrew.

Answer 16 · 2021-12-11T16:24:24.000Z

Yes! It worked! Thank you very much. :-)