Add UTF-8 string manipulation functions to the lua stdlib
ronisbr opened this issue · 15 comments
Since we are using Lua 5.1, we do not have any library to manage UTF-8 string. This can be a problem when creating lua scripts that manipulates the buffer with UTF-8 strings.
we should expose select functions from mbyte.c . In particalar, i would be good for scripts to have direct access to neovim's specific idea of grapheme cluster (which is a bit limited but might get upgraded later).
Nice! In the mean time, we can always clone and use for example https://github.com/Stepets/utf8.lua
Btw, it will be very nice to have access to the functions in mbyte, especially those related to the conversion between a UTF-8 char and the display size of it.
On a related note (assuming this library includes offset calculations etc.), it would be useful to be able to directly compute those for files that are not loaded (without the current bufload step which may trigger side effects).
@bfredl What do you think remains here with vim.str_utf_{pos,start,end} now in core?
We have added functions for codepoints. for some usecases, functions for composing chars (i e vims idea of grapheme clusters, the smallest editable unit) might be needed.
Related to the original request (not so much the current plan): https://github.com/uga-rosa/utf8.nvim
Hi! Are there any plans to expose those functions anytime soon?
Are you volunteering to work on it?
Unfortunately, I cannot. I am just asking if there is any update since I do not follow Neovim development updates... Is there anything wrong about it?
The lack of UTF-8 support was the reason why this bug was opened: Vonr/align.nvim#15
That's why I decided to ping this issue.
For all my use cases, I bundled https://github.com/uga-rosa/utf8.nvim
On second thoughts, is there anything extremely wrong in bundling the code in https://github.com/uga-rosa/utf8.nvim inside Neovim? This can be used as the API is very similar to what we have in Lua 5.3. At a latter stage, we can replace those functions by low-level ones written in C.
If this is the case, I think I might help.
No, we will not simply bundle an external plugin. As bfredl explained, we need an API that exposes Neovim's own string handling capabilities. This is the only way of making sure that these functions play nicely with other Neovim APIs. And that requires thought and effort -- as much as other things that are higher priority currently.
Ok, so the answer to my question "Are there any plans to expose those functions anytime soon?" would be just "No, unfortunately, there are not".
I mean, this does not simply happen by itself, despite the suggestive impersonal phrasing. It will happen when someone who personally cares about this topic invests the time and effort to do it -- same as any open source development.
What is your point? I pretty much know how open-source development works. I did not demand anything from anyone. I just asked if anyone has any plans to close this issue. You really do not need to be harsh over one simple question.