Using PhysicsFS as FS interface
frabert opened this issue · 13 comments
I was wondering if it wouldn't be better to add the capability to load VDF files to PhysicsFS instead of using a homebrew solution like it is being done now in ZenLib. This would also make it easier for people to make mods and the like, since standard archive types could be used, together with real directories.
Let me now if this would make sense, so that I could start adding the support to PhysFS.
That looks like a really good library! Thanks for mentioning it!
I am currently in the making of the custom loader, will link the repo as soon as I can.
EDIT: Here it is
hehe, implementing directly into the library will make it a breeze to work with
I think that doing as little as possible in-house is a good thing, relying on the work of others is a great way of focusing effort on important rather than trivial things.
Anyway, it currently squashes all the tree structure. I am working on a way to avoid this.
Update: I've updated the archiver to maintain the correct tree structure, but the total path length should not exceed 256 characters. This should not be a problem, but I'm working on a completely different approach which will also enable to obtain other metadata such as the timestamp.
EDIT: Done. Needs testing though...
Mind joining us in https://gitter.im/REGoth/Lobby ? 😄
Thus has been done long ago. Working great!
The only problem with this implementation in PhysFS is that it ignores VDF folder structure. E.g. you can't mount only _work/Data/Textures it gathers all files and puts them under /. It completely ignores parent directories which I think should work otherwise. E.g. if you mount .zip in PhysFS with some folders inside it preserves the structure.
@bugproof This was done deliberately, to mimic the way the old REGoth filesystem abstraction worked
@frabert I see. I know that original vdfs and Gothic accepted full paths. When I tried that with PhysFS I was surprised.
I don't know if there any daedalus scripts in the game that use full paths, that would break the game I think.
All references to content from scripts and files are without paths. Also, we did not find any files with the same name in different directories, so we decided to ditch them
In fact, the original engine has to change directories depending on which kind of content is supposed to be loaded, too! This is where the infamous "Orcaxe-Bug" comes from:
Setting the current directory and loading a file are two separate functions in the original VDFS library. I believe there were race-conditions when the texture loader thread changed the working directory.
Not having to deal with useless paths makes everything a lot easier.
I compared performance vs original VDFS and PhysFS appeared like 40% slower. (I created proxy for vdfs32g and tested on original game) do you know what could cause it @frabert ? Implementation from https://github.com/GothicFixTeam/GothicFix was much faster than the original but it's windows-only. IIRC ReGoth was slow at loading vdfs too compared to how it looks originally
What was your benchmark like? What does "40% slower" mean? When opening a random file and reading it all at once? When reading a file in random locations?
The best bet in figuring out what's making it slower would be to profile it. As for why the original is faster, I have no idea since I have no idea what's inside it, either 😄
If you can profile the code and see where the hotspots are, maybe we could submit a patch to icculus and merge it into PhysFS. I can't imagine performance improvements being frowned upon!
Lastly, one consideration: the original VDFS library was probably tuned specifically for Windows, and only loaded VDF files. PhysFS has a whole another abstraction layer in it, since it can be used on mostly any modern (and not!) OS and can load a slew of archive formats. This likely means making compromises, and sacrificing a bit of performance for the sake of flexibility. Still, if your figure of 40% is accurate, that's quite a significant performance hit.