pandoc/lua-filters

`local fh = io.open(line)` Unable to read non-latin path and filenames

wenbopeng opened this issue · 11 comments

jgm commented

Please be more specific: give a short example that reproduces the problem with full instructions.

cpkio commented

I don't think this is totally true, @wenbopeng. The cyrillics path reads successfully with lua51 test.lua in Windows:

local fh = io.open('R:/Проверка/Тестовая директория/мой файл.txt')
print(
  fh:read()
)

It still may be true for chinese or any other complicated script glyphs.

Yes, my statement may be wrong, to be precise, it does not support Chinese file names and paths. utf-8 is not allowed, io.open only supports ANSI. A method that might work is UTF-8 -> UTF-16, UTF-16 -> ANSI

see: how can i use io.open to open a unicode path in lua - Stack Overflow, 2023-07-12 09:49

cpkio commented

@wenbopeng This is clearly OS-dependent stuff. I can successfully read a file with chinese glyphs in path:

local fh = io.open('R:/测试/测试.txt')
print(
  fh:read()
)

Copypasted glyphs, made a directory, created a file with same name & .txt… Lua5.1

I use include-files.lua, using the following code block in my markdown file

content....

```{.include}
D:/测试.md
```
content....

pandoc will report an error: Pandoc warnings:Cannot open file D:/测试.md | Skipping includes

However, if I use the following block of code

content....

```{.include}
D:/test.md
```
content....

The result is exactly right

Lua 5.4.4 Copyright (C) 1994-2022 Lua.org, PUC-Rio
Embedded in pandoc 3.1.4

@wenbopeng This is clearly OS-dependent stuff. I can successfully read a file with chinese glyphs in path:

local fh = io.open('R:/测试/测试.txt')
print(
  fh:read()
)

Copypasted glyphs, made a directory, created a file with same name & .txt… Lua5.1

image

cpkio commented

I dunno what to say. I use ConEmu on Winx64 with UTF8 enabled.

user@DESKTOP-ILKGP6O 16:39:01 R:\ $ pandoc lua     
Lua 5.4.4  Copyright (C) 1994-2022 Lua.org, PUC-Rio
Embedded in pandoc 3.1.2                           
> fh = io.open('R:/测试.txt')                        
> print(fh)                                        
file (00007ffded19fa90)                            
> print(fh:read())                                 
dfsdfsdf                                           
>