Data import/export plugin improvements

[mosra]

Published from my internal TODO list in case someone would want to play with these.

Image and texture formats

• Sdl2ImageImporter using SDL2_image. It handles TGA, BMP, PNM, XPM, XCF, PCX, GIF, JPEG, TIFF, LBM and PNG in one package, so it might be a faster alternative to StbImageImporter, although it does have dependencies on libjpeg and such, so not that portable. Don't know about platform support.
• Jpeg2000Importer and Jpeg2000ImageConverter using JPEG 2000 -- compared to plain JPEG offers losless compression etc.
• OpenExrImporter and OpenExrImageConverter using the OpenEXR library for first-class HDR support. Currently the only supported input HDR format is (ahem) HDR (which is actually just RGB8 + a 8-bit exponent, so rather lossy) and there is MiniExrImageConverter, but without any compression. -- mosra/magnum-plugins@22d1ec9
• TinyExrImporter in case the above would be too painful to use on some platforms (https://github.com/syoyo/tinyexr)
• KtxImporter for importing files in the Khronos Texture Format (there's a draft version 2 now, too version 2 only, don't care about v1) -- mosra/magnum-plugins#103
• WebGLImageImporter that "just" uses the builtin WebGL/browser functionality to directly load JPEGs etc. into textures (on Emscripten that apparently needs to go through files)
  • could work for just URLs (OpenFile but not OpenData), also out-of-heap decoding
• DevIlImageImporter using DevIL, supports a lot of formats
• PNG/JPEG importer using WINAPI instead of libPNG and libJPEG
• https://github.com/randy408/libspng (faster than everything else, much nicer API)
  • Would be a good replacement for both libpng and stb_image once CgBI support lands (randy408/libspng#16) was closed as not planned
• Hand-written *.ico importer that delegates to PngImporter or BmpImporter for the actual data: https://en.wikipedia.org/wiki/ICO_(file_format)#Icon_resource_structure, storing different sizes as different mip levels (#369) -- mosra/magnum-plugins#79
• Importer using OIIO
• PSD layer import with https://github.com/MolecularMatters/psd_sdk (looks like "fun" to build, tho)
• PNG import using lodepng? It seems significantly slower than both stb_image and libpng and doesn't really offer anything that vanilla libPNG doesn't have, the only advertised advantage is "simple to build" (but come on, is libPNG hard to build to begin with?!) and the issue tracker looks nightmarish .. or maybe a "turbo version"?
• APNG animation import (there's https://aur.archlinux.org/packages/libpng-apng/)
• WebP import using https://github.com/webmproject/libwebp (it supports animations like GIF) -- mosra/magnum-plugins#121
• NumPy data import, useful for deciphering data dumps from Python
  • I should have a stashed code somewhere
  • inspiration: https://github.com/rogersce/cnpy, but without the ugly std::regex
  • or this? https://github.com/llohse/libnpy/blob/master/include/npy.hpp but without the ugly std::string (heh, times changed since the above line got written, the bar is now EVEN HIGHER)
• Google's WUFFS language ("operating system"??) has PNG, GIF, ... importers that claim to be the safest and fastest of all but the thing releases as a giant (1.5 MB) C file with everything inside, meaning it's rather annoying to integrate and no chance of this being packaged anywhere.
• import using https://github.com/libvips/libvips (and other operations, export, ...)
• OpenVDB for volume data
  • OIIO has import, but not export
  • https://github.com/AcademySoftwareFoundation/openvdb has import & export but not in a 3D-texture-ready form
• (Once image data passthrough is in), maybe integrate something like https://github.com/xiaozhuai/imageinfo for just querying image properties and passing the data through without having to rely on a ton of dependencies?

Image converters, compressors, editors

• DevIlImageConverter using DevIL
• Etc2ImageConverter using https://github.com/google/etc2comp
• zopflipng for compressing PNG images
• A plugin that integrates the pngcrush / optipng command-line utilities for compressing PNGs (or calls into Rust? https://github.com/shssoichiro/oxipng), finally something where the "file-only" converter APIs can get used
• BC1-5 compression
  • rgbcx from https://github.com/richgel999/bc7enc, BC7 is better handled by bc7e below (and what's the difference from https://github.com/richgel999/bc7enc16 ?)
  • https://github.com/nothings/stb/blob/master/stb_dxt.h -- mosra/magnum-plugins@370dc26 (missing BC4/5 right now)
  • https://github.com/castano/icbc, only BC1, but is faster than stb_dxt and has significantly better quality
  • https://github.com/darksylinc/betsy
• BC7 / ASTC compression, see this post for comparisons
  • https://github.com/GameTechDev/ISPCTextureCompressor
  • https://github.com/BinomialLLC/bc7e
  • https://github.com/Microsoft/DirectXTex
  • https://github.com/castano/nvidia-texture-tools
  • https://github.com/GPUOpen-Tools/Compressonator
  • https://github.com/darksylinc/betsy
  • https://github.com/ARM-software/astc-encoder, https://github.com/andrewwillmott/astc-encoder makes it work as a lib (sigh)
• BC6H (since we have Basis for LDR, we need something for HDR)
  • https://github.com/knarkowicz/GPURealTimeBC6H (we need an API to tell the imageconverter it needs a GPU context), wrapped in https://github.com/darksylinc/betsy as well
• ETC encoding
  • https://github.com/google/etc2comp
  • https://github.com/Ericsson/ETCPACK (slow?)
  • https://github.com/wolfpld/etcpak
• Rewrite jpegtran as a lossless JPEG image optimizer plugin (there's also https://github.com/cloudflare/jpegtran and https://github.com/tjko/jpegoptim (another frontend?))
• Integrate https://github.com/BinomialLLC/crunch (or the Unity fork of it) https://github.com/BinomialLLC/basis_universal for compressing GPU-compressed textures -- mosra/magnum-plugins#62
• DdsImageConverter dynamically using the above BC compressors
• xBRZ upscaler
• stb_image_resize, https://github.com/ramenhut/image-resampler, rescaling using FreeImage
  • ensure it does the right thing with sRGB formats (#125, http://www.ericbrasseur.org/gamma.html)
• GPU-compressed format decompression
  • https://github.com/hglm/detex, seems to be quite buggy tho
  • https://github.com/Themaister/Granite/tree/master/assets/shaders/decode for GPU compute ASTC/ETC/BC... decoding
  • https://github.com/google/astc-codec for software ASTC decoding
  • https://github.com/iOrange/bcdec small, has even tests 🎉, other candidates in https://github.com/aras-p/bcn_decoder_tester
• https://github.com/Triang3l/S3TConv for S3TC transcoding
• An IcoImageConverter that uses a PngImporter to pack data and then put them into an ico container (https://en.wikipedia.org/wiki/ICO_(file_format)#Icon_resource_structure), needs mip-level support (#369)
• Image converter using OIIO
• GIF animation export using https://github.com/notnullnotvoid/msf_gif (needs the batched multi-image APIs first)

Scene data

• Rewrite ObjImporter in a way that's actually performant (for example zero-copy like in the OpenDDL parser), material support (#205)
  • Triangulating quads
  • Non-trivial amount of time is also spent in merging data arrays (inspiration: https://github.com/syoyo/tinyobjloader/tree/develop/experimental)
  • Or delegate to e.g. https://github.com/thisistherk/fast_obj ?
  • Ensure all features from https://github.com/tinyobjloader/tinyobjloader#features are supported, benchmark on their 6M tri scene
• Avoid duplicate image data in OpenGexImporter -- currently each image file reference is taken as a new unique image
• StlImporter for the STL mesh format used by 3D printers, so we can have a viewer similar to what GitHub has -- mosra/magnum-plugins@da47aff
• AssimpImporter (not AssImporter) for importing various things using the Assimp library. Besides others a proper COLLADA support, FBX and Blender import.
• Pixar's Universal Scene Description import -- http://graphics.pixar.com/usd/ (Apple supports it in iOS 10)
• Alembic support -- http://www.alembic.io/ (efficient import of huge scene data, animation/simulation caches)
• Something that can eat the pbrt format used by the PBR book / pbrt -- especially the Moana Disney data set
• Pointcloud / photogrammetry import -- https://github.com/SBCV/Blender-Addon-Photogrammetry-Importer
• LEGO LDraw file importer: https://www.ldraw.org/article/218.html (could be used for stress-testing multi-draw / instanced scenarios, HAHAHAHAH)
  • https://github.com/pixeljetstream/ldrawloader
• MagicaVoxel .vox import: https://github.com/ephtracy/voxel-model
• IFC importer based on https://github.com/ifcquery/ifcplusplus
  • Assimp claims to do that too, but assimp/assimp#1889 and about a hundred other issues, files from https://technical.buildingsmart.org/standards/ifc/ifc-examples/ don't work at all
• FBX importer alternative to Assimp -- https://github.com/bqqbarbhg/ufbx, mosra/magnum-plugins#133
• Blender importer alternatives to Assimp
  • https://github.com/Flix01/fbtBlend-Header-Only (different file per version? huh)
• DWG files with https://github.com/codelibs/libdxfrw (windows-only?!), can also export?
  • Assimp supports DXF but not DWG

Scene / mesh converters

• Once Trade::AbstractSceneConverter is in (#371)
• Integrate https://github.com/zeux/meshoptimizer -- mosra/magnum-plugins@ae69193
  • alternative: https://github.com/microsoft/DirectXMesh (windows/msvc-only? sigh)
• http://www.pmp-library.org/ (it only reads/writes files, not in-memory data?)
• Integrate subdivision / simplification algos from Open3D?
• Integrate https://github.com/melax/sandbox/blob/e436aa9/bunnylod/progmesh.cpp (or is meshopt always superior?)
• https://github.com/kmammou/v-hacd for collision shape calculation
  • alternatively https://github.com/SarahWeiii/CoACD (SIGGRAPH 2022), doesn't work as a library yet
• https://github.com/qhull/qhull (library from 1995!) for convex hull calculation etc
• https://github.com/jpcy/xatlas for UV generation
• https://github.com/maggio-a/texture-defrag for repacking atlases in a better way
• glTF exporter -- mosra/magnum-plugins@3297069
• Export point meshes to .vox (https://github.com/ephtracy/voxel-model)
• https://github.com/tatsy/tinymesh , std::vector<Vec3> sigh, no support for arbitrary attributes
• https://github.com/mapbox/earcut.hpp or the non-single-header variant, takes a MeshPrimitive::Polygon input, produces Triangles, nothing else to be done there
  • Alternatively https://github.com/artem-ogre/CDT
• Corto as an alternative to Draco https://github.com/cnr-isti-vclab/corto
  • the benchmarks look promising
  • supports custom mesh attributes, which is really nice

Audio

• minimp3 dr_mp3 / PDMP3 loader (or just general MP3 support, we're no longer bound by patents) -- mosra/magnum-plugins#60
• minimp4 loader (does it add anything that's specially for audio?)
• Faad2Importer for AAC
• fdk-aac as an alternative for AAC? arguably better quality?
• integrate ffmpeg for a general audio format support -- mosra/magnum-plugins#55 (what about video?)
• https://github.com/adamstark/AudioFile

Video

Once appropriate IO APIs exist.

• ffmpeg
• https://github.com/axiomatic-systems/Bento4 for QuickTime, MPEG4, 3gp
• Something that uses the new Vulkan video APIs (design our API compatibly with those?)

Shader converters

• Naga -- http://kvark.github.io/naga/shader/2022/02/17/shader-translation-benchmark.html
  • various formats, much faster
  • needs to figure out how to call into Rust (or at least through an exe?)
• Tint for SPIRV -> WGSL -- https://dawn.googlesource.com/tint/
• https://github.com/intel/opencl-clang for OpenCL C -> SPIRV

Shader validators / profilers / assemblers

Mainly for being able to look at a shader assembly without having access to the HW itself.

• Radeon GPU analyzer https://github.com/GPUOpen-Tools/radeon_gpu_analyzer
  • only a binary, thus would need System::execute(), output redirection, temporary file creation etc
• ARM Mali Offline Compiler -- https://developer.arm.com/documentation/101863/7-5/Using-Mali-Offline-Compiler/Compiling-OpenGL-ES-shaders
  • only a binary, thus would need System::execute(), output redirection etc
  • under a login :(
  • they say shader binary compilation no longer available? https://developer.arm.com/documentation/101863/7-5/Platform-support/Binary-generation-support
• Something for NV?
• Something for Apple / Metal?
• https://github.com/microsoft/DirectXShaderCompiler
• https://github.com/madmann91/slang -- a potential candidate for a nice and well tested 3rd party obfuscator/minimizer/optimized

Text rendering

• https://github.com/tomolt/libschrift as an alternative to stb_truetype, might not be so underperforming in Debug?

[mosra]

Another thing to consider: Pixar's USD: http://graphics.pixar.com/usd/ it might be able to properly describe PBR and all other stuff that's required for (ahem) Pixar-quality scenes

[Deleted user]

@mosra How about DevILImageImporter, using DevIL?
Similar to SDL2ImageImporter, would provide optimized loading of various commonly used formats.

[mosra]

@alicemargatroid added that to the list, thanks!

[mosra]

Added https://github.com/google/etc2comp to the list.

[mosra]

DevIlImageImporter added via mosra/magnum-plugins#23.

[mosra]

Unique images in OpenGEX addressed quite some time ago in mosra/magnum-plugins@b677f90.

[mosra]

Added http://www.alembic.io/ to the list.

[mosra]

Added PNG/JPEG loading using WINAPI to the list.

[mosra]

Added pbrt format importer, image converters/compressors and audio importers to the list.

[williamjcm]

For the MP3 loader, I see that there's dr_mp3, which is based on minimp3.

Since I'll have a need for a MP3 loader plugin in the near future, I'll look into making a DrMp3AudioImporter plugin and opening a PR.

[mosra]

Nice. I assume the API could be very similar to other Dr* plugins, so doing that could be mostly a copypaste.