chrisbra/unicode.vim

caching problems

teto opened this issue · 10 comments

teto commented

I am trying to package this plugin for nixos, a linux distribution with the peculiarity of enforcing read-only packages once installed, which triggers an error with loading this plugin
E482: Can't open file /nix/store/qgz55005madsjvhlcjsilc6195m55mjk-vimplugin-unicode-vim-2018-12-14/share/vim-plugins/unicode-vim/autoload/unicode/UnicodeData.vim for writing: read-only file system

One possibility I have is to generate the cache before the installation, apparently I would need to run vim and call UnicodeWriteCache but with the in front, I am not sure how/if I can call it n
vim -c "call UnicodeWriteCache('toto', 'tata')"

fu! <sid>UnicodeWriteCache(data, ind) "{{{2
    " Take unicode dictionary and write it in VimL form
    " so it will be faster to load
    let list = ['" internal cache file for unicode.vim plugin',
        \ '" this file can safely be removed, it will be recreated if needed',
        \ '',
        \ 'let unicode#unicode#data = {}']
    let format = "let unicode#unicode#data[0x%04X] = '%s'"
    let list += map(copy(a:ind), 'printf(format, v:val, a:data[v:val])')
    call writefile(list, s:directory. '/UnicodeData.vim')
    unlet! list
endfu

Does the cache make a big difference (on SSD for instance) ? I wonder if I could disable it instead. It looks pretty fast here.

chrisbra commented

Well the cache is there, so that Vim does not always have to parse the unicodedata file and can simply source the already once parsed file. Else using Unicode.vim will be considerably slower the first time you use some of its functionality.

At least this was back when I implemented it that way, it might not make such a big difference nowadays on a SSD drive.

So if you apply this patch:

diff --git a/autoload/unicode.vim b/autoload/unicode.vim
index 900b05c..2ab9fec 100644
--- a/autoload/unicode.vim
+++ b/autoload/unicode.vim
@@ -87,6 +87,23 @@ fu! unicode#Download(force) "{{{2
     endif
     return 1
 endfu
+fu! unicode#MkCache() "{{{2
+    " Create the cache for existing Unicode Data file
+    let cache_file = s:directory. '/UnicodeData.vim'
+    if !filereadable(cache_file)
+        call <sid>UnicodeDict()
+    else
+        call <sid>WarningMsg("Cache already exists")
+        return
+    endif
+    if !filereadable(cache_file) ||
+        \ getftime(cache_file) < getftime(s:UniFile) ||
+        \ getfsize(cache_file) < 100 " Unicode Cache Dict should be a lot larger
+        call <sid>WarningMsg("Something went wrong when trying to create the cache file")
+    else
+        call <sid>WarningMsg("Cache successfully created")
+    endif
+endfu
 " internal functions {{{1
 fu! unicode#CompleteUnicode() "{{{2
     " Completion function for Unicode characters
diff --git a/plugin/unicode.vim b/plugin/unicode.vim
index 1a163da..a8ce29e 100644
--- a/plugin/unicode.vim
+++ b/plugin/unicode.vim
@@ -40,6 +40,7 @@ com! -nargs=1       SearchUnicode   call unicode#PrintUnicode(<q-args>, '')
 com! -nargs=1 -bang UnicodeSearch   call unicode#PrintUnicode(<q-args>, <q-bang>=='!')
 com!               UnicodeTable    call unicode#PrintUnicodeTable()
 com! -nargs=1       DigraphNew     call unicode#MkDigraphNew(<f-args>)
+com!                UnicodeCache    call unicode#MkCache()
 " deprecated
 com!               DownloadUnicode call unicode#Download(1)
 com!               UnicodeDownload call unicode#Download(1)

Then you could run:

vim --clean --cmd ":set rtp^=$PWD" -c 'ru plugin/unicode.vim' -c 'UnicodeCache' -c ':q'

and generate the cache file. Would that work?

Note, this assumes that the UnicodeData.txt file is available.

Also note, I am not sure, if the unicode license allows to distribute the UnicodeData file separately from the Unicode website.

antonc42 commented

I'm no lawyer but is seems like it is allowed as long as you include a copy of the copyright notice. See https://www.unicode.org/license.html.

Will you be including this patch in a future release?

chrisbra commented
antonc42 commented

I was thinking it might be useful for automated deployment. I deploy my vim with ansible and with this I could generate the cache on deployment instead of on first use. Thanks for all the great work!

teto commented

thanks for the patch. There is some problem with the script to add new vim packages to nixos so I will try your patch a bit later.

teto commented

I updated the patch and the vim command kinda works (vim still output things and leaves the terminal with red text enabled) but when calling :UnicodeTable, it still attempts to write ?:

|| Erreur détectée en traitant function unicode#PrintUnicodeTable[2]..<SNR>188_UnicodeDict[28]..<SNR>188_UnicodeWriteCache :
|| ligne    9 :
|| E482: Can't open file /nix/store/az41aylmfdsh4nr0q82hc8jlv5zglilh-vimplugin-unicode-vim-2019-03-01/share/vim-plugins/unicode-vim/autoload/unicode/UnicodeData.vim for writing: read-only file system

even though cache is older than unicodeData.txt ?

ll /nix/store/az41aylmfdsh4nr0q82hc8jlv5zglilh-vimplugin-unicode-vim-2019-03-01/share/vim-plugins/unicode-vim/autoload/unicode/                
total 3888
-r--r--r-- 1 root root  135707 1970-01-01  html.vim
-r--r--r-- 1 root root 1769137 1970-01-01  UnicodeData.txt
-r--r--r-- 1 root root 2070250 1970-01-01  UnicodeData.vim
chrisbra commented

even though cache is older than unicodeData.txt

Is it really older? I believe than the plugin is trying to recreate the cache file in this case. search for getftime() and try to comment out this condition and see if this works

teto commented

both file have the same getftime of 1. (Nix sets every file to the same creation date EPOCH + 1 sec it seems). I guess a check could be changed from < to <= ?

teto commented

cool ! thanks

antonc42 commented

Yes, thanks!