koreader/koreader-base

mupdf tar fixes

q3cpma opened this issue · 14 comments

Frenzie commented

Reference to relevant issue: koreader/koreader#5624

Possibly held up by #762 but if you can do the same as #943 I don't think anybody would have any objections.

q3cpma commented
  • 👍1
NiLuJe commented

I'm building on Gentoo with no issues?

┌─(niluje@illyria:pts/10)─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────(~/MPLAYER/zsync2-koreader/build)─┐
└─(0.28:68%:17:40:%)── eix -I dev-lang/lua                                                                                                                                                                                                                                                                                                              ──(Sat, Feb 08)─┘
[I] dev-lang/lua
     Available versions:  
     (0)    5.1.5-r4 [M](~)5.1.5-r5
     (5.1)  [M](~)5.1.5-r100 [M](~)5.1.5-r101 [M](~)5.1.5-r102 [M](~)5.1.5-r103
     (5.2)  [M](~)5.2.3 [M](~)5.2.3-r1 [M](~)5.2.3-r2 [M](~)5.2.3-r3 [M](~)5.2.4^t [M](~)5.2.4-r1^t [M](~)5.2.4-r2^t
     (5.3)  [M](~)5.3.3 [M](~)5.3.3-r1 [M](~)5.3.3-r2 [M](~)5.3.5^t [M](~)5.3.5-r1^t [M](~)5.3.5-r2^t
       {+deprecated doc emacs readline static test test-complete ABI_MIPS="n32 n64 o32" ABI_RISCV="lp64 lp64d" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Installed versions:  5.1.5-r4(12:10:00 AM 01/28/2020)(deprecated readline -emacs -static ABI_MIPS="-n32 -n64 -o32" ABI_RISCV="-lp64 -lp64d" ABI_S390="-32 -64" ABI_X86="64 -32 -x32")
     Homepage:            http://www.lua.org/
     Description:         A powerful light-weight programming language designed for extending applications

[I] dev-lang/luajit
     Available versions:  (2) 2.0.5-r1 **2.1.0_beta3
       {lua52compat static-libs}
     Installed versions:  2.0.5-r1(2)(04:30:11 PM 07/10/2018)(-lua52compat -static-libs)
     Homepage:            http://luajit.org/
     Description:         Just-In-Time Compiler for the Lua programming language

Found 2 matches

(Well, barring the usual LuaRocks 3 weirdness, which I take care of like this).

Sidebar: I'm not sure anyone really actually wants to be using CBT instead of CBZ. By nature, tar is not seekable (the T stands for "tape", after all, not really a seekable medium ^^), so it's designed entirely around being streamed.

That point may be moot if mupdf doesn't actually seek in zips either, though ;D.

Frenzie commented

Clearly one person does. ;-)

In any case using CBZ with PNG or JPEG doesn't seem like it'd make much sense either. Those are already compressed in a very efficient manner; throwing ZIP on top just seems like a waste of cycles all around. And I recall that for some reason even uncompressed ZIP seemed to waste more cycles than expected.

Incidentally, what I noticed in my comics from Humble Bundle is that the PDF is preferable over CBZ as a rule of thumb. Fundamentally they're both largely just a collection of JPEGs, but PDF can include vectors, most often or at least most noticeably text.

NiLuJe commented

When I say CBZ, I usually mean with no compression (i.e., store, not deflate) ;).

There's a bit of an issue with PDFs for Comics in our case: the whole file is read and stored in memory on open. Which means a high-res GN will kill a 512MB device :/. (I can actually confirm that, we get oom-killed when trying to load the > 500MB PDF I had on hand ^^).

I haven't looked where the culprit is, but switching to an mmap would take care of that kind of issues.

Frenzie commented

Huh, I read Cells at Work on my H2O and that one was 457 MB. Not quite 512 but you'd think it'd be close enough for problems.

Most comics don't really work on my H2O because, well, they're not manga. :-P But the PDF vector advantage nevertheless persists.

NiLuJe commented

Might have been sneakier than that then, that particular file was very neatly laid out, which means there were fancy uncompressed (very) high-res TIFF images mixed w/ PDF vectors and stuff.

Since it crashed on open because of an oom-kill, I just assumed that it was trying to read the whole file in memory ;).

Frenzie commented

In case you grabbed the Humble Bundle in question, it's Cells at Work Volume 5.

I just double checked and memory use while loading that file is < 200 MB. It does, however, quickly balloon past 1 GB if you skip through pages, although I can't seem to get it past 1.3 GB. But that's on my desktop where I'm currently only using ~14 GB RAM (out of 32) so I don't know how that behavior maps to the H2O. If nothing else it'd be a lot harder to skip through that quickly.

poire-z commented

The memory usage you may see as you browse pages with PDF is probably just the visited pages' blitbuffers getting cached - that cache size is computed depending on the system available/free memory: https://github.com/koreader/koreader/blob/master/frontend/cache.lua#L15-L58

q3cpma commented
Frenzie commented

@poire-z Oh right, exactly. So on the H2O they'd just get unloaded a lot quicker, but there'd never be a problem except with really high DPI individual images or something.

q3cpma commented
q3cpma commented
q3cpma commented