gltfpack: Encode textures in its source format
Closed this issue · 4 comments
Motivation:
Currently the gltfpack tool can resize and encode textures, but only in ktx2 format, but this format could increase load time (decode time) or textures, and could suffer from compatiblity issues, especally for low-end devices.
Proposal:
Add support of resize and encode textures in its original format, currently jpeg, png, and webp.
Candidate encoders:
- jpeg: mozjpeg (lossy and lossless)
- png: pngquant (lossy), oxipng (lossless)
- webp: libwebp (lossy and lossless)
Non-goals:
Support this proposal in wasm build.
Risks:
- Introducing external libraries could introduce their possible vulnerabilities.
- Binary size could increase greatly (maybe 2x more).
- A lot of code and developing could be needed.
- Building and testing could be slower.
Alternatives:
Use 2 passes, first to output gltf, and process textures as image files using external tools, and a second pass to pack it to glb.
Both PNG options here are in Rust. MozJPEG wasn't updated in a year although that by itself is probably fine. Unsure if any of these provide resizing code by themselves (correct resizing requires careful filtering).
But more broadly speaking, I don't think it would make sense for gltfpack to gain more dependencies. KTX2 encoding is harder to do externally because of the necessary metadata and performance issues in existing CLI options (gltfpack does it optimally to the extent the underlying algorithms allow), wheres PNG/JPEG resize can be done externally as a pre-process or post-process, so the extra value here is minor convenience. On the flip side, integrating with a bunch of third party libraries is definitely not something I want to explore.
I've considered the use of stb_image+stb_image_write+stb_image_resize in gltfpack for this purpose before - they at least are straightforward to build - but I am worried that stb_image is not very secure wrt untrusted images. This also will not support WebP, and I assume that the resulting images will not be as small as they could be when a PNG/JPEG optimizer is used - I haven't tested the impact here though.
In general, I am currently treating non-KTX formats as "nice to have support for"; you kinda want super-compression for large assets or low-end devices, and PNG/JPEG/WebP result in a high memory consumption post-decode and slower rendering (even if the transmission size could be better). The reason why WebP support was added for gltfpack was precisely to allow the hybrid workflows (eg compress to webp before gltfpack) to avoid the need to integrate with any third party libraries for first-class support.
stb_image_write unfortunately is not really fit for the purpose; it routinely generates ~30% larger outputs in my testing. The documentation does list it as a limitation.
The PNG output is not optimal; it is 20-50% larger than the file
written by a decent optimizing implementation; though providing a custom
This can be partially mitigated by using sdefl.h (not used by gltfpack atm, not treated as production code) as a replacement; however that makes encoding very slow (~45 seconds to encode a 33 MB GLB file)
For PNG the additional issue is that stb_image_write does not support palettized images, which is the mode (for some types of content) that you would want to achieve maximum reduction in size.
Without the palette, the gains on real world data seem to be minimal and mostly due to undoing of redundant streaming splits in IDAT chunks. So overall this is all of questionable utility (modulo texture resizing) without a real PNG optimizer.
pngquant, in addition to being written in Rust, is GPL with a separate commercial license.
Overall I think this is all way too much work for me to want to maintain, let alone develop. For flows where PNG/JPEG/WebP is the expected storage format I think a separate step (that can run before or after gltfpack, as gltfpack will preserve PNG/JPEG/WebP images) would be appropriate.