Undocumented File Format Structures (as of 0.99)
starknebula opened this issue · 25 comments
There exists undocumented data members inside the VOX file format. Proceeding the XYZI data, we can see tags labelled:
- nTRN
- nGRP
- nSHP
- LAYR
These appear to be non-standard when compared to the other Chunk types as they don't seem to be 4-byte aligned and are may be terminated with 0xFFFFFFFF.
Furthermore the old description for materials MATT appears to have been replaces by a new descriptor MATL which uses strings to identify material properties rather than pure binary.
If there's anything I can do to help let me know.
Please update the docs further, LAYR still isn't documented, also rOBJ seems to be missing.
bump. LAYR and rOBJ is still not documented. would be very appreciated.
Hello all,
like you, i tried to understand the new file format. i thinck im successfull.
the weird thing about this new format, is the format number (150) is the same in each voxel files i opened .... old files and new :)
i used hexamonkey (http://hexamonkey.com), who is a fabulous binary file analyser,
for tried to write the grammar fo the two vox file.
so i found that the byte offset you seen, is due to the STRING TYPE in each DICT.
and i think the LAYER Chunck is :
i seen that the unknown var is always -1, so i think its a reserved int on 4 bytes, like other reserved items.
so it display that :
here the grammar:
and here the result :
marvelous no ? :)
the grammar file is available on my repor : https://github.com/aiekick/MagicaVoxel_File_Formats
i will post on ,this repo the "voxFileWriter" im curently writing :)
Thank you! This is amazing!
i not seen the rOBj in my vox files. do you have sample vox file with this rOBJ ? for search the structure
i not seen the rOBj in my vox files. do you have sample vox file with this rOBJ ? for search the structure
This little 3x3x3 pyramide has the mentioned rOBJ-Tag. It got created with the latest version that supports multiple objects in one file, 0.99.1 under Windows. As it wasn't very complicated to create, it's licensed under WTFPL.
@DanielGilbert I created a pullrequest with the rOBJ chunk aiekick/MagicaVoxel_File_Writer#1
i have approved the request. :)
i added in my repo a basis vox writer with world mode support :) it can be improved i think but for the moment it take the job. my goal was just to export voxel block in new world mode from my Soft "SdfMesher" : https://twitter.com/hashtag/sdfmesher?src=hash
Hi all, found this issue because I'm working on a numpy-vox format lib and I also noticed stuff in the saved vox files not in the file format description.
Just adding a +1 really. @ephtracy, any chance of documenting the current file format doc please?
Or in the very least, updating the model version written to files (apparently it's still at 150), so we can at least know if we're trying to open a file with a model format we don't yet know how to parse.
the reason why the version is still 150 is to make the file readable in older versions (the old version only accepts files with version 150).
the rendering setting are not open yet because they are changing frequently. But you can still read it since it is just in the DICT format. I will update the format of the Layer chunks though.
I will update the format of the Layer chunks though.
@ephtracy Could you please provide an update on .vox format documentation?
+1 ... would rather like to know what rOBJ does
(I'm guessing it's lighting, background, skybox and camera information, and I really would like to use that so I can import scenes directly into my game engine)
What is IMAP about? A chunk I've found in the PrinceOfPersia scene: https://www.patreon.com/posts/prince-of-persia-26600425
About the IMAP and NOTE chunk:
IMAP:
Represents the palette "Index" Map
size = 256
256 x int(1 byte)
The palette of a VOX file contains 256 colors, all identified by their index and containing a specific color (r,g,b,a).
The RGBA chunk contains the color information of the palette but not the exact indexes (let's assume for the moment that the order of RGBA is random).
To overcome this problem and find the exact palette index of each color, the IMAP chunk stores the RGBA/palette index mapping.
- Example:
RGBA =[{ r:148, g: 63, b:125, a:255 }, { r:32, g: 54, b:53, a:255 }, { r:0, g: 25, b:1, a:255 }, ...]
IMAP =[3, 5, 41, 63, 84, 201, 17, ...]
In this voxel file, the color{ r:0, g: 25, b:1, a:255 }(index2in RGBA) 's true palette index is41.
By the way ... voxel color = RGBA[voxel.index - 1]
NOTE:
Contains all Color type Names
- int32 : number of names (would be 32)
- for each name:
- STRING
@hugodecasta Thanks for the hint regarding IMAP, saved my day!
I had a model where in MV I would see index 1 as a certain color, but while importing and reading the file, it insisted the index is 242. The voxel had that index before, but I would later change it in MV, so the index (and RGBA order) must have stayed the same. Very confusing.
@hugodecasta Thanks for the hint regarding IMAP, saved my day!
I had a model where in MV I would see index 1 as a certain color, but while importing and reading the file, it insisted the index is 242. The voxel had that index before, but I would later change it in MV, so the index (and RGBA order) must have stayed the same. Very confusing.
Yes, the RGBA data saving order is still a mystery to me too (maybe it has to do with the inner working of MV: moving stuff around for other purposes. Who knows).
@hugodecasta Yes. Also: When IMAP chunk is not present (needs to be checked when importing), the index of the RGBA chunk and voxel color index match, like one would expect.
@hugodecasta Yes. Also: When IMAP chunk is not present (needs to be checked when importing), the index of the RGBA chunk and voxel color index match, like one would expect.
Until an official behavior documentation hard to find out why such different behaviors
About the IMAP and NOTE chunk:
Thanks so much for this description, it seems to be the only one online that describes what this chunk actually does (and importantly, in which direction it "maps").
However, in files produced by MV that contains such a chunk, which I always ignored in the past, and produces correct colors. Looking at the chunk, it contains a fairly "random mapping", i.e. there's not a lot of correlation between the array index and the value in the array. Trying to apply it according to your description results in wrong colors.
It seems to me the voxel indices in a XYZI chunk correspond directly to the entries in an RGBA chunk (+1) regardless of the presence of a IMAP chunk, and IMAP needs to always be ignored?? Is this remapping for UI purposes only?
Anyway, for posterity, here's what I did to create a remapped palette:
for (int i = 0; i < 255; i++) {
remapped_palette[imap[i]] = palette[i + 1];
}
If you're wondering why the indices are off, my palette array stores the transparent color at index 0, so all of the RGBA entries are shifted up by 1, to make them correspond directly to voxel indices. The 0th entry in imap has a value of 1 if its a 1:1 mapping.
It seems to me the voxel indices in a
XYZIchunk correspond directly to the entries in anRGBAchunk (+1) regardless of the presence of aIMAPchunk, andIMAPneeds to always be ignored?? Is this remapping for UI purposes only?
It's also used for reading palette labels.