Relocations are broken
Closed this issue · 56 comments
It appears that relocations are currently being applied incorrectly. For example, in Dark Cloud, in EscapeDungeonMode there should be a jump to SearchItemIndexNo like this:
001f3f58 50 f6 06 0c jal CDngStatusData::SearchItemIndexNo undefined SearchItemIndexNo(int param_1)
However, instead, the jump is to an invalid address like this:
001f3f58 a0 ec 0d 0c jal SUB_0037b280
If you look in Ghidra' relocation table view, you can see that this is due to a relocation at this address (which is incorrect):
001f3f58 0x4 0x2c1d 50 f6 06 0c SearchItemIndexNo__14CDngStatusDataFi
Both IDA and ps2dis do not alter the initial value and produce correct results. This can be seen in a number of other places too, but I picked one example for convenience.
@Adubbz I notice the return type of SearchItemIndexNo is still set to DefaultDataType (undefined)
Does this still occur if you change the return type of that function to the correct return type? I ask because I have noticed some syncing issues with ghidra regarding the DefaultDataType. They are mainly with thiscall but I'm wondering if this is due to the same thing.
I went into ps2dis with a file I had laying around and went to the offset of the jal in question and changed the instruction to match what is listed above. It produced jal 0x0037b280. May you take a screenshot of what ps2dis produces with Dark Cloud along with a screenshot of what ghidra is producing?
Also an unrelated side note, I sent you an invite to a private repo the other day and then deleted it. Not sure if you noticed. I wanted to contact you about it and what it was but couldn't find any contact information.
By default, Ghidra’s output is a jump to an invalid address as follows:
001f3f58 a0 ec 0d 0c jal SUB_0037b280
But after manually patching it myself to match IDA/ps2dis then it becomes:
001f3f58 50 f6 06 0c jal CDngStatusData::SearchItemIndexNo undefined SearchItemIndexNo(int param_1)
Which is the correct output. It’s not an issue of return types, it’s that the jump location is completely wrong because it got changed by a relocation when it shouldn’t have. You’ll note that the bytes Ghidra has, a0 ec 0d 0c, are completely different from IDA/ps2dis which give 50 f6 06 0c because of the relocation.
Sorry if that wasn’t clear from my original post, by the sounds of it I think you might’ve read the correct/incorrect output as being the other way round. I can still provide screenshots when I’m back at my PC if needed, but hopefully that makes things clearer.
Just for clarification, is there supposed to be a relocation at this address? I don't think that relocations are currently supported. I think irx/erx/erl use a non standard relocation format but I have to confirm that.
At that specific address I don't believe so (At least, not based on radare or IDA), but one is applied anyway. A non-standard relocation format does sound like a plausible explanation for the behaviour
I'll see about overriding the mips elf relocation to see if it helps, but I'd need to be able to have an example of the issue. If you can replicate it with something in the open source ps2sdk and send me the binary it would be helpful.
In the meantime you can just right a 4 line python script to fix it. I'm on my phone so this may not come out right.
if __name__ == "__main__":
table = currentProgram.getRelocationTable()
for relocation in table.getRelocations():
table.remove(relocation)
If you have the relocation window open you will need to close and reopen it to see the changes.
Actually when importing the binary, go to options and uncheck 'Perform symbol relocations' it should fix this issue for now.
Yeah that does do the trick. Not sure if it's necessarily correct to complete ignore whatever relocations are there, but I'm not familiar enough with the PS2 to know either way.
Yeah that does do the trick. Not sure if it's necessarily correct to complete ignore whatever relocations are there, but I'm not familiar enough with the PS2 to know either way.
Of course it is not the best idea to completely ignore all relocations, it is just a temporary solution.
Are you able to reproduce this with code from the ps2dev ps2sdk? It would be helpful if you can supply a binary with this issue which you have the rights to distribute.
@Adubbz I've finally managed to reproduce this. I'm moving it to the top of my non school related to do list.
Awesome, thank you!
@Adubbz may I have the results of running readelf --sections on the binary? I have a huntch that it may have been built using Metrowerks and it appears that they may have done some "non-standard" things. I actually stripped out the relocations and the example of the issue I had still ran just fine. I manually checked the relocation from what pcsx2 does here and got the same results that are applied by ghidra.
I actually haven't seen a valid relocation in a ps2 binary yet other than in ps2dev/ps2sdk erl binaries. ERX/IRX modules work differently but I'm working on a loader for those. I think it may be safe to just ignore them but I need more information.
Here's the output from readelf --sections:
readelf --sections SCUS_971.11
There are 16 section headers, starting at offset 0x2b120c:
Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .shstrtab STRTAB 00000000 1a2480 000081 01 0 0 1
[ 2] .strtab STRTAB 00000000 1a2510 02b0c1 01 0 0 1
[ 3] .symtab SYMTAB 00000000 1cd5e0 031610 10 2 9083 1
[ 4] main PROGBITS 00100000 000100 1a2380 01 WAX 0 0 128
[ 5] .relmain REL 00000000 2057e0 07d350 08 3 4 0
[ 6] title PROGBITS 01dabd00 1a2480 000000 01 WAX 0 0 128
[ 7] .reltitle REL 00000000 282b30 01d330 08 3 6 0
[ 8] dun PROGBITS 01dabd00 1a2480 000000 01 WAX 0 0 128
[ 9] .reldun REL 00000000 29fe60 011368 08 3 8 0
[10] heap PROGBITS 01f06b00 1a2480 000000 01 WA 0 0 16
[11] .mwcats LOUSER+0x4a2a82 00000000 1febf0 006468 01 4 0 4
[12] .mwcats LOUSER+0x4a2a82 00000000 205060 000588 01 6 0 4
[13] .mwcats LOUSER+0x4a2a82 00000000 2055f0 0001f0 01 8 0 4
[14] .comment PROGBITS 00000000 2b11c8 00002b 01 0 0 1
[15] .reginfo MIPS_REGINFO 00000000 2b11f4 000018 01 0 0 4
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
p (processor specific)
Ok so I've done some investigating and there definitely is some relocation going on. This may take awhile to fix though. I'm implementing a loader to handle IRX module imports and such and am also sticking in a way to fix up MW symbols when applicable so they can be demangled by the GNU demangler demangler to handle both gnu and mw.
@sp193 is there anything inherently different about the way the ps2 handles relocations from a standard MIPS processor? I've taken a look at how they are handled by pcsx2 and opl's udnl and I get the same results from ghidra's relocation implementation.
If you are talking about relocations, surely it is related to ELF instead? No, the EE and IOP cores are mostly standard MIPS (EE is largely a MIPS III, while the IOP is a MIPS I). We are using GCC, was only modified to support the EE-specific features (2nd pipeline, MMI and custom FPU) and the IOP IRX (which is largely an ELF, but with module information and a fixed layout).
If you are talking about relocations, surely it is related to ELF instead? No, the EE and IOP cores are mostly standard MIPS (EE is largely a MIPS III, while the IOP is a MIPS I). We are using GCC, was only modified to support the EE-specific features (2nd pipeline, MMI and custom FPU) and the IOP IRX (which is largely an ELF, but with module information and a fixed layout).
I see. The relocations appear in elf files which I'm assuming are compiled by the metrowerks ps2 compiler as there are .mwcats sections present. If it wasn't for the '.rel' prefix in the section names I would have just considered the some kind of compiler specific information which aren't actually relocations. Unfortunately everything points to them really being relocations, its just that the data doesn't add up.
Thank you.
I have not found any authoritative statement that says that the ELF header's e_type field shall determine whether relocations must be applied, but would it be a solution to check whether the ELF e_type is ET_REL (and the Sony custom types for IRX and ERX) before applying relocations? The static ELFs we that have usually have around, have ET_EXEC instead of ET_REL.
.a and .erl files have ET_REL.
.irx have 0xFF80
.erx have 0xFF91
I have not found any authoritative statement that says that the ELF header's e_type field shall determine whether relocations must be applied, but would it be a solution to check whether the ELF e_type is ET_REL (and the Sony custom types for IRX and ERX) before applying relocations? The static ELFs we that have usually have around, have ET_EXEC instead of ET_REL.
.a and .erl files have ET_REL.
.irx have 0xFF80
.erx have 0xFF91
Yes. I pulled the custom relocations from the ps2dev binutils patch. I'm waiting on the fix for NationalSecurityAgency/ghidra#1260 to make it into a public release of ghidra before I make any code public but I have a ton of stuff laying around.
I also got the DVP overlay sections and stuff setup properly. The only thing I can't seem to get correct are relocations to non-dvp overlay sections since the overlay data usually doesn't exist.
I also have iop IMG and rom img loading based off ROMIMG , .mdebug parsing, stabs parsing (WIP), gcc v2 and mw demangling, custom loader to handle the different sections and external references to irx libraries, irx loading and database based off irxtool.
The only loading related thing I haven't been able to do is erx loading simply because I don't have any.
I have a lot of cleaning up to do before I make anything public though. I have a poor habit of writing chicken scratch while I'm figuring things out.
I still need to figure out how to properly convert the binutils relocation macro into proper code:
HOWTO (R_MIPS_DVP_11_S4, // type //
4, // rightshift //
2, // size (0 = byte, 1 = short, 2 = long) //
11, // bitsize //
FALSE, // pc_relative //
0, // bitpos //
complain_overflow_signed, // complain_on_overflow //
bfd_elf_generic_reloc, // special_function //
"R_MIPS_DVP_11_S4", // name //
FALSE, // partial_inplace //
0x03ff, // src_mask //
0x03ff, // dst_mask //
FALSE); // pcrel_offset //
I look forward to this fix.
I look forward to this fix.
The fixes won't come until the release of ghidra 9.2. There was an issue with extension points that has since been resolved. I also need to do some major cleanup and reorganization but hopefully it shouldn't be to long.
Thank you, I was having this issue I believe as well. I was getting some reference that was not in memory, and IDA was showing the references correctly :/
I don't have a license for IDA, but I'm itching to start work on a PS2 game. I'm wondering if maybe a build could be created for a dev build of Ghidra 9.2? Even if that requires compiling ghidra on our own.
I don't have a license for IDA, but I'm itching to start work on a PS2 game. I'm wondering if maybe a build could be created for a dev build of Ghidra 9.2? Even if that requires compiling ghidra on our own.
I'm not sure if @beardypig wants to do that. I've gone ahead and done so on my fork though.
https://github.com/astrelsky/ghidra-emotionengine/releases/tag/1.4.1.1
I was thinking about how best to do it, you'd need to keep it up to date with the commits on the ghidra repo...
I was thinking about how best to do it, you'd need to keep it up to date with the commits on the ghidra repo...
Yes. Although the java code that is present is unlikely to require frequent updates to stay up to date with the ghidra repo I would actually advise against it.
OK. We can merge your PR and manually add a 9.2 DEV build to the release :)
I've been trying to get Eclipse to work after I reformatted my PC, but it's not playing nice :(
Unfortunately, using this new build on Ghidra 9.2 DEV, it didn't seem to fix the issue, relocations are still broken.
Unfortunately, using this new build on Ghidra 9.2 DEV, it didn't seem to fix the issue, relocations are still broken.
Oh. My apologies I misunderstood. I literally just rebuilt the 4b90a8c for 9.2. The build I have with fixed relocations needs to be reworked due to recent changes in Ghidra's demangler. I'll see about tidying it up and putting something out there as soon as I can. I cannot guarantee its stability though.
If you're just looking for a relocation fix then you can just apply #16 (comment) as that is what is going on under the hood.
I see. That's fine, I'll look forward to this release, as right now the games affected by this suffer tremendously. On a side note, is there planned support for the .mdebug section? A game I'm interested in has nearly 10mb of data in that section.
…
On Tue, Apr 14, 2020, 5:31 PM Andrew Strelsky @.***> wrote: Unfortunately, using this new build on Ghidra 9.2 DEV, it didn't seem to fix the issue, relocations are still broken. Oh. My apologies I misunderstood. I literally just rebuilt the 4b90a8c <4b90a8c> for 9.2. The build I have with fixed relocations needs to be reworked due to recent changes in Ghidra's demangler. I'll see about tidying it up and putting something out there as soon as I can. I cannot guarantee its stability though. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSCIYMUSNAQCV6FAFJUNV3RMT56HANCNFSM4H6VPVHA .
I do have that mostly complete. I just haven't figured out the line number encoding yet. I've extracted and parsed everything else from the .mdebug though, filenames, functions, stabs etc.
It's been a long time and I've been itching to reverse a certain binary. If someone can fix the plugin to make it load properly with Ghidra (Relocations should work, debug symbols including function argument type + names, struct / classes, etc), then I would be very thankful.
Thank you!
It's been a long time and I've been itching to reverse a certain binary. If someone can fix the plugin to make it load properly with Ghidra (Relocations should work, debug symbols including function argument type + names, struct / classes, etc), then I would be very thankful.
Thank you!
Please don't upload files you don't own the rights to.
I do have some of it working, it just needs to be cleaned up. I will make what I have public once ghidra 9.2 is released. It is supposed to be released soon but I do not know exactly when.
Okay. Apologies for uploading the binary.
In terms of 9.2, if you could make a separate branch, that would be good enough. I'm willing to build Ghidra 9.2 if it would let me start reversing this binary. As far as I'm aware there's no indication for when 9.2 will officially be released, and it's been 8 months since the last release. I hate to be pushy like this, but I'm getting a little frustrated with how long it's taken to release.
Okay. Apologies for uploading the binary.
In terms of 9.2, if you could make a separate branch, that would be good enough. I'm willing to build Ghidra 9.2 if it would let me start reversing this binary. As far as I'm aware there's no indication for when 9.2 will officially be released, and it's been 8 months since the last release. I hate to be pushy like this, but I'm getting a little frustrated with how long it's taken to release.
You may give this a shot. I won't put it up in a branch until I've removed the unused code and figure out how I want to handle the irx modules.
Thank you @astrelsky <3
Unfortunately @astrelsky, the plugin throws an exception during analysis.
Unfortunately @astrelsky, the plugin throws an exception during analysis.
Oh. Sorry about that. It's part of the reason I haven't released it yet. If you build ghidra with pr #2214 it will go away
I've gotten much further this time. However, it still errors.
Also, I have some more things to mention.
#1) Do you know what data is contained in the .mwcats section? I can't seem to find any information on it, and you mention it above.
#2) It seems not all of the debug information is properly loaded. All of the structs are created, but they are completely empty, even though full struct layouts are included in the debug information. Function argument names are also not applied, and neither are local variables inside of functions either. (Information is included for all of that.)
The reason for the crash is because I didn't include the data files for the irx analyzer since it wasn't right. Just disable it for now I'm going to redo it with information from https://psi-rockin.github.io/ps2tek/
As for the debug information I'm assuming you mean the mdebug. Unfortunately the documentation for it is super vague and I couldn't figure out how to get more out of it.
If you have any information on the mcats sections that would be great because I could never find any.
Also if you need to load an overlay you can. If you go to file-> add to program and select an overlay file it should load.
No, I don't mean mdebug. I mean .debug, which I believe is DWARFv1. (.mdebug was for a totally different game which I'll be looking at later.) I'm suspecting mwcats might be additional debugging information specific to Metrowerks. I'll see if I can find anything about it.
No, I don't mean mdebug. I mean
.debug, which I believe is DWARFv1. (.mdebug was for a totally different game which I'll be looking at later.) I'm suspecting mwcats might be additional debugging information specific to Metrowerks. I'll see if I can find anything about it.
I would say that is a ghidra issue then. I think ghidra only officially supports dwarf2 although it does load higher versions.
If .debug really is dwarf1 ill definitely look into it. It shouldn't be too bad to hack something together since ghidra already has dwarf support.
Darn. Hrmmm, ok.
I'm going to look to putting this up into my fork and cleaning it up so it is workable. It's past time.
Sounds good to me.
Hopping off for the night, but I did manage to make a discovery. It appears they have some relation to other debug info. Ie: A single entry lines up with debug info entries. See below
EDIT: I should note that these sections are from a .o file, not directly from an ELF.
@astrelsky I hope everything is going well. I have an issue to report though.
It seems like many pointers are not interpretted correctly.
I compiled this code:
When disassembling in Ghidra, I get this:
I notice that 0x221700 is exactly double the real pointer address. Seems like an issue with the plugin to me.
@astrelsky I hope everything is going well. I have an issue to report though.
It seems like many pointers are not interpretted correctly.I compiled this code:
When disassembling in Ghidra, I get this:
I notice that 0x221700 is exactly double the real pointer address. Seems like an issue with the plugin to me.
If you are using my dev branch you may open an issue in my fork. I suspect that the program used in the screenshot isn't copyrighted. If so may you attach it please?
Will do.
I am unable to create issues on your repository. The tab just doesn't exist, presumably because it is a fork. I have uploaded the elf here.
cw_dbg_opt0.zip
I am unable to create issues on your repository. The tab just doesn't exist, presumably because it is a fork. I have uploaded the elf here.
cw_dbg_opt0.zip
Didn't realize they weren't enabled. I've enabled them.
Now that Ghidra 9.2 (and 9.2.1) have released, is this possible to have fixed in the main version of the plugin?
Now that Ghidra 9.2 (and 9.2.1) have released, is this possible to have fixed in the main version of the plugin?
Yes. Sorry I've been procrastinating. Will see to putting in the relocation fix tomorrow.
Well, @beardypig seems to be inactive anyways.
My 9.2 fix PR (#36) is still unmerged after 6 weeks of no activity.
Well, @beardypig seems to be inactive anyways.
My 9.2 fix PR (#36) is still unmerged after 6 weeks of no activity.
I'll see if I can get a hold of him. If not I will just merge it as I know it is correct.