Region fixer crashing when attempting to fix certain worlds
Opened this issue · 39 comments
Describe the bug
On my multiverse server I have quite a number of maps (a little over 40 atm) of varying ages and levels of minecraft fuckery/corruption affecting them- that I'm currently trying to deal with. While some of the maps were fixed successfully, others would cause Region fixer to throw out an "Ops! Something went really wrong and regionfixer crashed." error. Bug reports below.
Full copied text from the MS-DOS view
All worlds run the same command and arguments, only the path being different:
python regionfixer.py --fc --fm --fw -p 24 "E:\Minecraft\Minecraft Bukkit Server\worlds\creative"
DO note that i run this from a batch file to help with the iteration for each world, but i doubt it should cause any issues.
Also, since it seems that the Multiverse way of structuring dimensions in their own wold_[dimension_name] structure is not recognized by Region fixer- eg: "Overworld" worlds path can be left as worlds\creative", and Region fixer also detects the level.dat (and all the other) files present there correctly, Nether/End worlds require full path to the dimensions, Ex: worlds\creative_nether\DIM-1" and worlds\creative_the_end\DIM1 respectively to be read at all, and cannot make use of level.dat or other data present in the world folder.
Expected behavior
While i understand it can't literally work miracles with all my wonky worlds, i'm expecting it to at least not completely crash when reading some of them? Again, logs attached below.
Screenshots
n/a
Files that would help solving the issue
if requested i can provide some of the worlds causing those issues
Desktop (please complete the following information):
- OS: Win10 2004
- Python version: 3.7.3
- Region Fixer Version 0.3.3
Additional context
Bug report 1 (and the most common):
<class 'TypeError'>
Traceback (most recent call last):
File "E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 374, in fix_problematic_chunks
chunk = region_file.get_chunk(*local_coords)
File "E:\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 599, in get_chunk
return self.get_nbt(x, z)
File "E:\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 574, in get_nbt
data = self.get_blockdata(x, z) # This may raise a RegionFileFormatError.
File "E:\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 566, in get_blockdata
raise ChunkDataError(err)
nbt.region.ChunkDataError: Error -3 while decompressing data: incorrect data check
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "regionfixer.py", line 588, in <module>
value = main()
File "regionfixer.py", line 548, in main
fix_bad_chunks(args, w)
File "regionfixer.py", line 71, in fix_bad_chunks
counter = scanned_obj.fix_problematic_chunks(problem)
File "E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 1398, in fix_problematic_chunks
counter += regionset.fix_problematic_chunks(status)
File "E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 936, in fix_problematic_chunks
counter += self._set[r].fix_problematic_chunks(status)
File "E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 391, in fix_problematic_chunks
out += dc.decompress(i)
TypeError: a bytes-like object is required, not 'int'
Bug report 2 (just one instance):
**********
*** Exception while scanning:
*** 0f31e1bc-88f0-3d0a-95c5-47cd965cb541-6507148097825114692.dat
**********
*** Printing the child's traceback:
*** Exception:<class 'EOFError'>Compressed file ended before the end-of-stream marker was reached
**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\scan.py, line 746, in scan_data
*** _ = nbt.NBTFile(filename=s.path)**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 628, in __init__
*** self.parse_file()**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 655, in parse_file
*** self._parse_buffer(self.file)**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 493, in _parse_buffer
*** tag._parse_buffer(buffer)**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 486, in _parse_buffer
*** name = TAG_String(buffer=buffer).value**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 345, in __init__
*** self._parse_buffer(buffer)**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 350, in _parse_buffer
*** read = buffer.read(length.value)**********
*** File C:\Users\Vlad\AppData\Local\Programs\Python\Python37-32\lib\gzip.py, line 276, in read
*** return self._buffer.read(size)**********
*** File C:\Users\Vlad\AppData\Local\Programs\Python\Python37-32\lib\_compression.py, line 68, in readinto
*** data = self.read(len(byte_view))**********
*** File C:\Users\Vlad\AppData\Local\Programs\Python\Python37-32\lib\gzip.py, line 482, in read
*** raise EOFError("Compressed file ended before the "
**********
Same for me
And for me! Any fix so far?
This project is very much dead, so don't expect any fixes anytime soon. Also given 1.8 is soon approaching which will probably completely change how worlds are saved yet again, this tool will most likely die with it.
This project is very much dead, so don't expect any fixes anytime soon. Also given 1.8 is soon approaching which will probably completely change how worlds are saved yet again, this tool will most likely die with it.
Yes I know, now this project is dead, rip
It's dead for now. I don't have the free time to update this and the NBT library and there are not many people cooperating so, yeah, it's dead for now. I kind of feel bad saying this but right now it feels like a chore (it was fun before).
This project is very much dead, so don't expect any fixes anytime soon. Also given 1.8 is soon approaching which will probably completely change how worlds are saved yet again, this tool will most likely die with it.
I didn't know that 1.8 was changing the save format. Do you have any information/source about this?
I said "probably"; but it does seem reasonable to expect that, considering world saves now have to deal with a completely new generation algorithm (and probably new parameters to set up for it), increased height limit and introduction of the 3d biomes to achieve these new terrain features.
We already had changes in the game like that without changes in the world format. For example: generation algorithm changed a few times, the height limit changed at least once withouth repercuting.
A really hope they don't change the world format.
Although the project is dead for now, your tool really helped me and fixed my issue by deleting, not fixing the chunk. Thanks for your great work, @Fenixin ❤️ !
Although the project is dead for now, your tool really helped me and fixed my issue by deleting, not fixing the chunk. Thanks for your great work, @Fenixin ❤️ !
Yeah, maybe ut should be called Chunk-remover, because I think that's the most used feature :P
Thaks a lot for the kind comment, it means a lot!
@VL4DST3R , ok, so this may be too late but, could you provide onen of the worlds that crash regionfixer? At least, could you provide a region file for type of crash?
You can use PM or post here a link.
@VL4DST3R , ok, so this may be too late but, could you provide onen of the worlds that crash regionfixer? At least, could you provide a region file for type of crash?
You can use PM or post here a link.
Sure thing, I'm sure i have a couple with both flavors of errors. I'll post a Dropbox link tomorrow when i finish work since I'll need to batch re-run regionfixer on all the maps we have to find them again.
Using the latest build, (no changes in errors btw) here are two levels that have the first issue: overworld and a nether dimension to differentiate a bit, and finally with the latter, which turns out i have only one map affected by it. Do tell me when you get around to downloading them so i can delete them from db.
Thanks! I've got the files. You can delete them.
Great, im curious if you manage to find what the issue is!
Thanks! I've got the files. You can delete them.
Any updates on this?
Not really, sorry. I'm in the end of term marking examns like crazy. I hope soon.
I ran into the same fault (exact same error) a few days ago but I found a solution for my specific case.
To upgrade from spigot 1.17.1 to 1.18 the spigot team recommended to --forceUpgrade first on 1.17.1 and then again on 1.18
Problem was my 3277-Spigot-3892929-2071671 "failed to read some chunks" leading to Error upgrading world.
Running regionfixer.py after that failed --forceUpgrade lead to the crash documented by OP.
Solution was to update to 3284-Spigot-3892929-0ebef35. Upgrading the map worked flawlessly that way (with regionfixer.py cleaning up some minor stuff before the upgrade).
I have pushed a change that adds EOFError to the list of exceptions. Those files that crashed regionfixer should be now correctly flagged as broken data files.
I think this error could be handled better but for now this is enough.
I'll leave this as open for some time, I want to test a few things.
Thanks for the worlds and the reports!
Hello Alejandro, first thanks a lot for all your efforts with this great tool! :-)
Not sure if you're still interested in new reports for this issue, but for me the tool still crashes using latest master GIT branch version and this region file while trying to fix the 4 broken chunks:
r.-1.-1.zip
Nothing critical in this region, so I can just go ahead and delete them, still wanted to let you know this problem seems to still persist.
Thanks again
Andreas
Hello @ab-tools!
Thanks very much for your report! I really appreciate it. I have downloaded the file and you can remove it if you want.
I'll look into it when I get the time (who knows when...).
Thanks again!
Thanks for your reply, Alejandro, you're welcome and no hurry from my side!
Just pushed a new branch with a possible fix for this issue. Could anybody test it?
This is the branch: https://github.com/Fenixin/Minecraft-Region-Fixer/tree/bytes-fix
Just pushed a new branch with a possible fix for this issue. Could anybody test it?
This is the branch: https://github.com/Fenixin/Minecraft-Region-Fixer/tree/bytes-fix
On it, testing now. BTW tested the one you pushed a few hours ago in the hopes it would handle the corruption better, same issue on that one, but i guess you knew that already.
EDIT: nevermind, it seems its the same with the one i tested, which means that no, the issues i'm experiencing are still here, here is some of the output log (same error shows up for all affected worlds)
Unreadable player files:
No problems found.
Unreadable data files:
No problems found.
Chunk problems:
---------------------------------------------
| Problem | Corrupted Wrong located Total |
---------------------------------------------
| Count | 16 107 21942 |
---------------------------------------------
Region problems:
No problems found.
######### Repairing chunks with status: Corrupted ##########
Repairing chunks in regionset "Level/Region files for Overworld":
The chunk 25,31 in region file region\r.0.-2.mca couldn't be fixed.
The chunk 27,31 in region file region\r.0.-2.mca couldn't be fixed.
Ops! Something went really wrong and regionfixer crashed.
Bug report:
<class 'TypeError'>
Traceback (most recent call last):
File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 375, in fix_problematic_chunks
chunk = region_file.get_chunk(*local_coords)
File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 622, in get_chunk
return self.get_nbt(x, z)
File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 597, in get_nbt
data = self.get_blockdata(x, z) # This may raise a RegionFileFormatError.
File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 589, in get_blockdata
raise ChunkDataError(err)
nbt.region.ChunkDataError: Error -3 while decompressing data: invalid distance too far back
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "regionfixer.py", line 574, in <module>
value = main()
File "regionfixer.py", line 534, in main
fix_bad_chunks(args, w)
File "regionfixer.py", line 71, in fix_bad_chunks
counter = scanned_obj.fix_problematic_chunks(problem)
File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 1453, in fix_problematic_chunks
counter += regionset.fix_problematic_chunks(status)
File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 971, in fix_problematic_chunks
counter += self._set[r].fix_problematic_chunks(status)
File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 392, in fix_problematic_chunks
out += dc.decompress(i)
TypeError: a bytes-like object is required, not 'int'
I sent you 2 worlds a while back, and those two are still some of those causing errors on the latest build. Have you by chance tried running the fixer on those maps yourself?
If you need i can provide them again.
I sent you 2 worlds a while back, and those two are still some of those causing errors on the latest build. Have you by chance tried running the fixer on those maps yourself?
If you need i can provide them again.
Those are the world I'm testing this with, and they work. The error you posted tells me that you are not in the branch bytes-fix, I really think you downloaded/didn't check out the correct branch. Could you please check that?
Are you sure-sure the build there is the one thats updated? I double-checked and it acts the same?
Ooops... sorry about that, give me a moment...
LMFAO i was about to go crazy!
I'm sorry about that! It should be there now.
Well well well, new stuff!
Unreadable player files:
No problems found.
Unreadable data files:
No problems found.
Chunk problems:
---------------------------------------------
| Problem | Corrupted Wrong located Total |
---------------------------------------------
| Count | 16 107 21942 |
---------------------------------------------
Region problems:
No problems found.
######### Repairing chunks with status: Corrupted ##########
Repairing chunks in regionset "Level/Region files for Overworld":
The chunk 25,31 in region file region\r.0.-2.mca couldn't be fixed.
The chunk 27,31 in region file region\r.0.-2.mca couldn't be fixed.
The chunk 28,3 in region file region\r.0.-2.mca couldn't be fixed.
The chunk 30,0 in region file region\r.0.-2.mca couldn't be fixed.
The chunk 31,11 in region file region\r.0.1.mca couldn't be fixed.
The chunk 5,12 in region file region\r.1.-1.mca couldn't be fixed.
The chunk 6,6 in region file region\r.1.-1.mca couldn't be fixed.
The chunk 6,9 in region file region\r.1.-1.mca couldn't be fixed.
The chunk 17,17 in region file region\r.1.-1.mca couldn't be fixed.
The chunk 23,23 in region file region\r.1.-1.mca couldn't be fixed.
The chunk 26,25 in region file region\r.1.-1.mca couldn't be fixed.
The chunk 3,22 in region file region\r.1.-2.mca couldn't be fixed.
Repaired 0 chunks in this regionset.
Repairing chunks in regionset "POIs files for Overworld":
Repaired 0 chunks in this regionset.
Repairing chunks in regionset "Entities files for Overworld":
Repaired 0 chunks in this regionset.
Repaired 0 chunks with status: Corrupted
No chunks to fix with status: Missing Entities tag
####### Repairing chunks with status: Wrong located ########
Repairing chunks in regionset "Level/Region files for Overworld":
Repaired 107 chunks in this regionset.
Repairing chunks in regionset "POIs files for Overworld":
Repaired 0 chunks in this regionset.
Repairing chunks in regionset "Entities files for Overworld":
Repaired 0 chunks in this regionset.
Repaired 107 chunks with status: Wrong located
I let it run on all my maps, so it will be a hot minute until its done with all of them but so far so good!
Those mentions about chunks that cannot be fixed, what does that mean? Or what should i do?
I let it run on all my maps, so it will be a hot minute until its done with all of them but so far so good!
Those mentions about chunks that cannot be fixed, what does that mean? Or what should i do?
A chunk is a compressed file, if something is changed it's usually lost. Although very infrequent, sometimes you can decompress the file byte by byte and when you reach error you have something useful. But this is very infrequent.
So, if those chunks cannot be fixed they should be replaced or removed.
Also, I found a bug, regionfixer says that the chunk is fixed, but nothing happens, will look into it (hopefully) in the next days. Please, take this version of regionfixer as beta, be careful and bakcup all your worlds!
And thanks a lot for testing!
Also, I found a bug, regionfixer says that the chunk is fixed, but nothing happens, will look into it (hopefully) in the next days. Please, take this version of regionfixer as beta, be careful and bakcup all your worlds!
💀
I hope i didn't shoot myself in the foot from all this excitement. You.. sure it doesn't do anything? I do have backups but id rather not wipe everything and restore all worlds.
And thanks a lot for testing!
Any time you need! This is very important to me as well, if it wasn't obious. Do tell me when you get around to making fixes stick.
💀
I hope i didn't shoot myself in the foot from all this excitement. You.. sure it doesn't do anything? I do have backups but id rather not wipe everything and restore all worlds.
I hope I didn't shoot you in the foot! AFAIK Regionfixer is writing the new fixed chunks in the region file but if you rescan the world the chunk appears as broken again... which is strange to say the least. Right know I don't trust fixing chunks at all, but the worst will happen to your worls is that nothing will be fixed. I really need to investigate this further.
Any time you need! This is very important to me as well, if it wasn't obious. Do tell me when you get around to making fixes stick.
I'll do!
hey @Fenixin long time no see XD Hope this finds you well, I just came checking to see if this project had any updates/will ever be updated or if you dropped it entirely? Fingers crossed for your return!
Hello!
@VL4DST3R, Thanks for asking. My life is chaos right now. I'd love to have the time for this project. That said I hope to do some stuff this summer, not earlier than july.
See you then!
Rooting for you to get it all sorted out!
Any updates on this issue? I recently needed to fix some corrupted worlds and there's nothing out there as far as i know that could (potentially) help besides this, but it still has the same issue as per my original issue above. Please don't abandon this :<