security: Out-of-bound r/w lead to arbitrary code execution

[HexRabbit]

Background

By design, the Gameboy has only 16 bits of addressing space, which means that no matter how large the memory is, the CPU can only access a maximum range of 0x0000 - 0xFFFF (64KB). Unfortunately, this addressing space is not only allocated to ROM and RAM but also to peripheral devices (such as audio and video output) that are mapped through MMIO, which also occupies part of the space.

This means there's less resources can be placed on ROM or loaded into RAM, so the banking mechanism is designed to solve this problem.

The concept of bank switch is to map an unaddressed memory location (e.g. 0x18000 - 0x19FFF) to a memory space (e.g. 0xA000 - 0xBFFF) where the bank can be switched freely, and then by controlling the current bank number through a register to switch between banks.

The location of the RAM / ROM bank can be found in the specification:

• ROM: 0x4000 - 0x7FFF
• RAM: 0xA000 - 0xBFFF

There are many types of Gameboy cartridges (e.g. MBC2, MBC3 ... etc.) and they offer different sized banks. For example, if you write X to any address from 0x0000 to 0x3FFF of an MBC2 cartridge, the ROM bank number will be switched to X. (reference)

This part is implemented in gb_memory_write() to support different cartridge types.

Vulnerabilities

The bug exist in two place in the jitboy RAM bank implementation:

• MBC3 handler: An off-by-one flaw, since MBC3 supports only 4 RAM banks (0 ~ 3), the comparison should be value < 4, without the equal sign.
• MBC5 handler: MBC5 supports up to 16 RAM banks, however there's no any check against the value variable.

If we trigger a write to ROM on some specific memory address range (e.g. ld (0x4000), A), the instruction will be translated by DynASM (at here, here) into gb_memory_write() and further be compiled into the jitted function, finally got executed during emulation.

Since mem->ram_banks is initialized with a fix-sized heap buffer (MAX_RAM_BANKS * 0x2000 == 4 * 0x2000), only 4 RAM banks are reserved for the emulator, so either MBC3 or MBC5 handler could cause a OOB (out-of-bound) access on heap, and by this primitive we can leak addresses and write to any function handler allocated on heap to control the execution flow.

Exploitation

After the vulnerability is found, I manually fuzzed the heap to find crashes, luckily I found a easy one to exploit.

It crashed on an instruction call qword ptr [rax+0x1d0], and since we can control the memory on both rax+0x1d0 and rdi point to, this is an easy win for system("/bin/sh").

https://github.com/HexRabbit/CTF-writeup/blob/master/2022/EOFCTF-qual/pwnboy/exp/exploit.c

My exploit starts with a bank switch to place out-of-bound heap memory into external memory (0xA000 - 0xBFFF), then leak the libc addresses and search the heap for a specific function table used by SDL2 library, which will be called during the screen rendering process.

After gathering addresses, next step is to calculate system() function address by some easy math, however it looks like the compiler GBDK only support up to 16 bit arithmetics, therefore I need to implement big number (64bit) addition/subtraction myself lol.

Finally, modify the function pointer to system() and place the argument "/bin/sh" for it, trigger the bank switch to write back the modified heap memory to pop a calculator!