Issues when rendering some chunks sections
Paulomart opened this issue · 5 comments
Hey!
Thanks for the awesome work on maintaining this iconic project! Im currently trying to move my rendering pipeline to use this version of mcmap.
In that process I discovered a very strange problem: Some chunk sections seem 'scrambled up'. All blocks inside are positioned quite random.
See for example see this render (1.16.4 world):
Note: The images in this issue have been scaled by GitHub, open the image in a new tab to see the full details.
This happens for all sorts of version and server software combinations. But I should note here, that the world savefile that was used here was written by a mineflayer bot. I'm also not sure if its an issue with their implementation of the anvil storage format or with mcmap itself.
I tried to write down how I approached this and where I think might be the cause of the issue.
My first impression was, that something wrote the chunks wrong to disk. So I started up a Minecraft client and loaded the world.
As you can see the red marked section was loaded correctly. This was the same chunk section that rendered wrong before. Its the scrambled up section on the right. Same savefile. Same chunks.
Now I was sure that the chunks wore at least in some kind of readable format on disk. I saved and quitted Minecraft and rendered the world again.
Before | After load & save |
---|---|
As you can see Minecraft somehow fixed these chunks and as result mcmap was able to render them like it should. You can still see some chunks that are 'scrambled up', these chunks were just not loaded by the client. Note: The Minecraft client generated some new terrain that I cropped away.
The Minecraft client must have altered the chunk section in some form. I opened both worlds using NBT Studio and compared the chunk section in question.
It seems pretty clear what happened here: The chunk section must have been stored in some inefficiency way, the Minecraft client compacted it from 342 down to 256 longs.
I looked at the post116
function. Maybe this function could not deal with this strange case?
mcmap/src/chunk_format_versions/section_format.cpp
Lines 6 to 30 in e4440a7
I added a debug statement to analyse the different blockstates lengths and index_lengths. This were the counts and the lengths that occurred while rendering the world.
1876 [Deep Debug] blockStates length: 256; index_length: 4;
35 [Deep Debug] blockStates length: 342; index_length: 4;
508 [Deep Debug] blockStates length: 342; index_length: 5;
When ignoring all chunk sections that have blockStates->size() == 342
you can see that more then the scrambled sections get removed. When also filtering on the index_length == 4
, we get only the scrambled sections removed.
blockStates->size() == 342 |
&& index_length == 4 |
(inverted) |
---|---|---|
I did not test this behaviour with other blockStates
lengths. From my testing with very different worlds and versions, some sort of error happens in all versions from 1.13 to 1.16.5.
Expand for a test matrix with different versions
Note: This also contains worlds (1.8 - 1.12) that were rendered using a legacy version of mcmap. These dont show this error.
Reminder: These worlds were generated using the specified server software, then 'downloaded' by mineflayer and then rendered.
I tried 'fixing' the chunk reading in mcmap myself, even reimplemented the function heavily inspired from nms code:
for (uint16_t index = 0; index < 4096; index++) {
const uint16_t j = index * index_length;
const uint16_t k = j >> 6;
const uint16_t l = (index + 1) * index_length - 1 >> 6;
const uint16_t i1 = j ^ k << 6;
uint16_t blockIndex = 0;
if (k == l) {
blockIndex = (int) ((unsigned((*blockStates)[k])) >> i1 & mask);
} else {
int j1 = 64 - i1;
blockIndex = (int) (((unsigned((*blockStates)[k])) >> i1 | (*blockStates)[l] << j1) & mask);
}
buffer[index] = blockIndex;
}
Sadly this didnt bring any success and I'm not really sure how to go on from here: It seems that there is an issue how mineflayer writes the chunks to disk. But at the same time Minecraft somehow is able to handle this. I would assume that it would be best to fix this in mineflayer, but I'm not really sure I can pinpoint the bug over there.
Maybe you could give some hints or help getting this resolved? Any help is really appreciated. Thanks for reading this really long issue.
Wow man thanks for such a detailed issue ! You narrowed down perfectly what I suspect is happening.
I have seen this before. Minecraft changed the way the chunks are saved in 1.16, to forego space for efficiency, and those scrambled chunks used to appear everywhere at first: they are the interpretation of data written in one format with the expectation of the other. This format is linked to the DataVersion
tag that appears in your screenshots.
What I suspect is happening here is that there is a mismatch between the version and the format used by the mod, and this is the result. What can also happen is that I mislabeled the Dataversion
number that introduced this format, and mcmap is blundering hard.
I can look into it, do you have a region file where this is happening ? I can also get into more details about those format changes to help if this is not on mcmap. Probably will be though.
Edit: It would seem the format is wrong on the index size.
On your screenshots, the palette has 15 elements -> the index size should be 4, as you can fit up to 16 on 4bits.
This is the binary value of the first long in the output of the mod:
0000000010000100001000010000100001000010000100000000000000000000
And this is the output of the first long of the minecraft output:
0001000100010001000100010001000100010001000100010000000000000000
You can clearly see repeating patterns (similar adjacent blocks), but of different sizes ! This clearly confuses mcmap. So, two possibilities:
- The chunks are considered as corrupted, and Minecraft re-generates them on open & save;
- I am missing something and Minecraft is able to understand this and correct it.
Thanks for the reply.
I have seen this before. Minecraft changed the way the chunks are saved in 1.16, to forego space for efficiency, and those scrambled chunks used to appear everywhere at first: they are the interpretation of data written in one format with the expectation of the other. This format is linked to the DataVersion tag that appears in your screenshots.
I assume you mean the zero padding for the long arrays: I have tested this with different DataVersions
: The problem also appears when mcmap is using the pre116
method to load chunk sections. So this happens on chunk sections that are zero padded and non zero padded.
I can look into it, do you have a region file where this is happening ? I can also get into more details about those format changes to help if this is not on mcmap. Probably will be though.
Sure, I just forgot to attach it. Here it is: world-1.16.4-bukkit.zip
On your screenshots, the palette has 15 elements -> the index size should be 4, as you can fit up to 16 on 4bits.
Yes. That confused me as well. I tried to adjust the decoding function to use the other length, so it would respect that.
You can also see that Mineflayer writes 4 bits of zeros and then continues to use 5 bits per block. I dont see how any code would be able to guess the 4 bits at the beginning.
The chunks are considered as corrupted, and Minecraft re-generates them on open & save;
Im really really sure this is not the case: As you can see in the rendered images after load and save, there is a world generation border with the surrounding chunks. If Minecraft was not able to read some chunk section, these borders would also happen inside the area of question.
Another point on this: The level.dat
file, which stores the seed, was just copied over from another world, so the seeds could never match. There is no per chunk seed, as far as I know, right?
Minecraft is able to understand this and correct it.
I think this is whats happening. But I cannot understand why. I will look at Mineflayers implementation later today and check again if there is some funky business going on. Maybe its possible to just fix it on their side, as this seems the only case it issues arises.
Thanks again for looking into this :)
Hey is this still an issue ? Has it been fixed on the Mineflayer side ? I looked at the data again today and it does seem like a format error of the world and not mcmap. For reference, one of the erroneous chunks uses pre-1.16 format and states a DataVersion of 2567
, when the 1.16 format has been used from DataVersions of 2534
onwards. I may be able to create a patch to check format errors, but I certainly do not want to make mcmap more complex than it already is ...
Hey, sorry for leaving this untouched so long. It really seems that there is an issue in Mineflayer. Sadly I was not able to nail it down further then described here, but I tested that Mineflayer is able to load and write chunks in this corrupt format (Even asserting the blocks one by one worked out).
I certainly do not want to make mcmap more complex than it already is
I agree 100% with you. But maybe Im or you are able to submit an fix based on such patch? My expertise in the world region format is pretty limited and so far I havent been able to figure it out...
Can you verify the mineflayer branch fixes the issue ? It works for the world you provided.
Now we need to discuss how this should be bundled. First of all, what code does bukkit use to generate regions ? Is it using minecraft binaries or all homebrew code ? Because if this is an issue in Minecraft itself, adding this to the project would not be an issue.