Bug with very large amount of textures, but only when in pk3

[mattreecebentley]

Very weird bug that only occurs on reloading saves, not starting a new game.
This is using the most recent build, under Windows x64.

When using a very large number of textures (1.6gb in jpeg format),
if the textures are in a pk3 upon reload of a save, some of the HD textures will be missing in-game,
and the particle effect textures don't work (show up as large squares as per the attached images).

If the pk3 is unzipped and deleted, these things do not occur.

Since this is occuring in the beta for something I am working on cannot supply a build demonstrating this publicly but am happy to supply one personally over email (matt@soulstudios.co.nz).

At any rate, looks like loading from the pk3 isn't working the same as loading from files/folders

s.

[mattreecebentley]

Discovered that the bug is not affected by file size, only the number of textures contained. The same ~1500 texture set, when compressed to DDS and then zipped, exhibited the same issue despite being only ~900MB.
The bug still exists with the most recent release.
In addition, many textures will not show in-game up when in a pk3 in this circumstance. But when upzipped show up fine.

[Shpoike]

QSS is meant to support zip64 (I ported the code from fte), so should be fine with zips bigger than 4gb etc (so long as the fseek's 'long offset' is actually 64bit).
The file table is malloced rather than imposing any file-count limits like QS(S)'s .pak loader still does.
QSS's gl_texmgr.c reallocs as needed in order to avoid imposing limits on textures (drivers may still have limits though, dds should ease that for the most part though).

Which reduces it to two possible limits...

1. QS requires FILE handles in far too many places. This may impose a limit due to handle usage, but they shouldn't be left open so shouldn't be an issue. Fixing this would require rewriting large chunks of the QS code, which would make merges harder.

2. at some threshhold, windows' zip util automatically switches to using the proprietary 'deflate64' algorithm for compression, which is incompatible with pretty much every quake engine out there - the only two algorithms supported (by Q3, DP, QSS, etc) are 'store'(ie: no compression) and 'deflate'(the regular non-64 version which is actually provided by zlib).
   A simple test would be to eg use fteqcc -x ../your.pk3 and see whether it managed to extract everything or not. I think 7zip and/or winrar UIs include extra info columns like the algorithm used to compress each file - if any say deflate64 then those file should be considered unreadable by quake engines.

Otherwise I've no idea.

[mattreecebentley]

I might post this on the QS engine page, but just for the record, I only ever use 7zip, so it's gotta be #1

[mattreecebentley]

Question - does quakespasm (not spiked) actually support .pk3's? My impression from reading the web was that it doesn't. If not, I can't report the bug there, as it doesn't occur when the pk3's are unzipped to the folder.

[Shpoike]

it doesn't.

that said, it does try to load the relrelease's .kpf file (also a renamed .zip file) anyway, to access a single text file. it just doesn't bother with .pk3 at all. yay hacks.
either way, any support for zips in QS is a completely different root from QSS (I kept QSS's zip code instead of duplicating it with QS's more limited stuff). It wouldn't help with the while FILE-ptr-used-everywhere problem anyway.

[mattreecebentley]

Would the file handle limit even be a factor, given that the textures work fine when unzipped? It's only when they're packaged together in a .pk3 that the issue occurs