[Feature] "Embedded" lightmaps

[SirYodaJedi]

A feature in VHLT and forks, "embedded" lightmaps are lightmaps that are baked into the diffuse texture, for textures that cannot be lightmapped on certain engines.

These have a few use cases:

• Lightmapped water in engines that don't support it (Ex: Vanilla, QEX, GoldSrc)
• More realistic glass using additive rendermode in HLBSP. This is how it's most commonly used in GoldSrc mapping.
• More realistic Trans33 and Trans66 surfaces in engines that don't support lightmapping these (I think this is currently the case in Yamagi)
• Creating brush-style prop models with more accurate lighting (which is more consistent with brush lighting than something baked in Blender)

[SirYodaJedi]

This is an example of how an implementation might look, from a mapper's perspective:

KVs

Only _ewt_embedlightmap, _ewt_embedlightmap_resolution, and _ewt_embedlightmap_writeext are probably integral, but I figured I'd list some other KVs that have features which could be useful.

KV	Type	Explanation
_ewt_embedlightmap	bool	Enable "embedded" lightmaps (equivalent to zhlt_embedlightmap)
_ewt_embedlightmap_resolution	uint	Luxel density; defaults to same density as regular lightmaps (ex: 16:1 texels to luxels)
_ewt_embedlightmap_subdivsize	uint	Max texture resolution to use before subdividing. 256 should be a safe default (being the max res supported by GoldSrc's software renderer, as well as the Voodoo 1).
_ewt_embedlightmap_intensity	float	Scale the obviousness of the lightmap; default 1.0. Useful for creating stuff that will be turned into static prop models, which will be vertex-lit by the engine.
_ewt_embedlightmap_writeext	bool	Write external textures; always enabled for Q2BSP
_ewt_embedlightmap_extformats	csv	List of formats to write external textures as; default is TGA (formats: BMP, PCX, PNG, TGA, WAL).
_ewt_embedlightmap_compressext	bool	Enable (additional) compression for supported formats (formats: 24-bit PNG - 256-color paletted, 24-bit TGA - RLE compression, 32-but PNG - 255-color paletted with 1-bit alpha if extbinaryalpha is enabled).
_ewt_embedlightmap_extbinaryalpha	bool	Allow binary alpha for supported formats (just PNG atm, being the 255-color paletted with 1-bit alpha, if compressext if enabled), as not all engines support such

Output location

External textures are written to the following folders, depending upon output BSP format:
Q1BSP: <moddir>\textures\
Q2BSP: <moddir>\textures\<mapname>
HLBSP: <moddir>\materials\<mapname>

Naming

Textures are automatically named to avoid conflicts, and to avoid breaking functionality. A 10-character hash is generated for the name; if two generated textures are identical, then both faces will use the same texture. The first three characters of the original texture name are retained, which ensures that most special prefixes still work ({, !, #, *, +x, -x, ~, @, etc.), with the exception of HLBSP scrolling textures (as those wouldn't look nice with embedded lightmaps anyway). To keep the embedded lightmaps from intermingling with regular textures when using Q1BSP external textures in the mapper's filesystem, the string l_ is be placed between the three prefix characters and the hash.

So a texture name might look like this: +0~l_e2cfc89fae

Bloat

When embedded lightmaps are used with an animated, toggled, or randomly tiled texture set, baked versions of all possible textures are created. Naturally, that will exponentially increase file size, but such a use-case is unlikely to be common.

Additional notes

• BMP and PCX are always written as 8-bit.
• 32-bit TGAs are left uncompressed regardless of _ewt_embedlightmap_compressext, because the vanilla engines and many derivatives don't read RLE-compressed 32-bit TGAs correctly. 24-bit TGAs can be RLE-compressed with no issues.
• Bit depth notes:
• With the exception of WAL files and embedded Q1BSP miptex, generated 8-bit paletted textures use their own autogenerated palette.
• • When it comes to open-source palettization algorithms, Wadmaker's results in much more accurate results than GIMPs.
• • Alphatest textures probably should treat the alpha index as #ff00ff regardless of palette, in order to avoid issues with dithering.
• • Q1BSP miptex should assume a no-fulbright palette, but this might need to be a disableable function for Hexen 2 support.
• Other external formats could be added (ex: DDS, JPG, M8, M32); I only listed the ones that would be useful for Quake I+II and GoldSrc support. The KVs above were written with expandability in mind.
• • Example of DDS: _ewt_embedlightmap_extbinaryalpha would cause generated DDS textures with transparency to use DXT1 instead of DXT5 or BC7
• • Example of JPG: _ewt_embedlightmap_compressext would result in a compression level of 90 instead of 98, or maybe it would enable chroma subsampling (which I usually disable for JPEGs myself)
• I originally didn't list BMP, but then I realized that this feature could be useful for creating static props, and HLMDL uses BMP textures.
• GoldSrc doesn't support external textures (the path above is what Xash uses), but it'd still be useful to be able to write them externally for use-cases such as the aforementioned brush-styled MDLs.
• VHLT's zhlt_embedlightmapresolution apparently results in downscaling the texture (to save file size, allegedly). I don't think it makes sense to implement an equivalent; users can downscale the texture beforehand.