Heimdall can flash files bigger than partitions

Question

Heimdall can flash files bigger than partitions

Opened this issue 3 years ago · 16 comments

Apparently I can flash a file that is bigger than the RECOVERY partition:

Here I've a recovery.img that is a bit over 8MiB (8388608 bytes)

$ du --bytes recovery.img
9691136 recovery.img`

And here's print-pit on Galaxy SIII (GT-I9300 variant) I managed to flash this image:

$ heimdall print-pit
Heimdall v1.4.2

Copyright (c) 2010-2017 Benjamin Dobell, Glass Echidna
http://www.glassechidna.com.au/

This software is provided free of charge. Copying and redistribution is
encouraged.

If you appreciate this software and you would like to support future
development please consider donating:
http://www.glassechidna.com.au/donate/

Initialising connection...
Detecting device...
Claiming interface...
Setting up interface...

Initialising protocol...
Protocol initialisation successful.

Beginning session...

Some devices may take up to 2 minutes to respond.
Please be patient!

Session begun.

Downloading device's PIT file...
PIT file download successful.

Entry Count: 16
Unknown 1: 1598902083
Unknown 2: 844251476
Unknown 3: 30797
Unknown 4: 0
Unknown 5: 0
Unknown 6: 0
Unknown 7: 0
Unknown 8: 0

--- Entry #0 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 80
Attributes: 2 (STL Read-Only)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 0
Partition Block Count: 1734
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: BOOTLOADER
Flash Filename: sboot.bin
FOTA Filename:

--- Entry #1 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 81
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 1734
Partition Block Count: 312
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: TZSW
Flash Filename: tz.img
FOTA Filename:

--- Entry #2 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 70
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 34
Partition Block Count: 16
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PIT
Flash Filename: mx.pit
FOTA Filename:

--- Entry #3 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 71
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 50
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: MD5HDR
Flash Filename: md5.img
FOTA Filename:

--- Entry #4 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 1
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 8192
Partition Block Count: 8192
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: BOTA0
Flash Filename: -
FOTA Filename:

--- Entry #5 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 2
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 16384
Partition Block Count: 8192
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: BOTA1
Flash Filename: -
FOTA Filename:

--- Entry #6 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 3
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 24576
Partition Block Count: 40960
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: EFS
Flash Filename: efs.img
FOTA Filename:

--- Entry #7 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 4
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 65536
Partition Block Count: 16384
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PARAM
Flash Filename: param.bin
FOTA Filename:

--- Entry #8 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 5
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 81920
Partition Block Count: 16384
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: BOOT
Flash Filename: boot.img
FOTA Filename:

--- Entry #9 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 6
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 98304
Partition Block Count: 16384
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: RECOVERY
Flash Filename: recovery.img
FOTA Filename:

--- Entry #10 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 7
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 114688
Partition Block Count: 65536
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: RADIO
Flash Filename: modem.bin
FOTA Filename:

--- Entry #11 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 8
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 180224
Partition Block Count: 2097152
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: CACHE
Flash Filename: cache.img
FOTA Filename:

--- Entry #12 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 9
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 2277376
Partition Block Count: 3145728
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: SYSTEM
Flash Filename: system.img
FOTA Filename:

--- Entry #13 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 10
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 5423104
Partition Block Count: 1146880
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: HIDDEN
Flash Filename: hidden.img
FOTA Filename:

--- Entry #14 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 11
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 6569984
Partition Block Count: 16384
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: OTA
Flash Filename: -
FOTA Filename:

--- Entry #15 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 12
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 6586368
Partition Block Count: 0
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: USERDATA
Flash Filename: userdata.img
FOTA Filename: remained

Ending session...
Rebooting device...
Releasing device interface...

Here the recovery is 8MiB as there are 16384 blocks and one block is 512 bytes.

But I can still manage to flash this recovery file:

$ heimdall flash --RECOVERY recovery.img
Heimdall v1.4.2

Copyright (c) 2010-2017 Benjamin Dobell, Glass Echidna
http://www.glassechidna.com.au/

This software is provided free of charge. Copying and redistribution is
encouraged.

If you appreciate this software and you would like to support future
development please consider donating:
http://www.glassechidna.com.au/donate/

Initialising connection...
Detecting device...
Claiming interface...
Setting up interface...

Initialising protocol...
Protocol initialisation successful.

Beginning session...

Some devices may take up to 2 minutes to respond.
Please be patient!

Session begun.

Downloading device's PIT file...
PIT file download successful.

Uploading RECOVERY
100%
RECOVERY upload successful

Ending session...
Rebooting device...
Releasing device interface...

I didn't check yet if it wrote part of the image in the RADIO partition or not.

Note that I've complete backups of the important partitions, so here if it erased the RADIO partition I can easily restore the backup.

I don't know yet how the Thor protocol implemented by heimdall works so I don't know how easy it is to get the partition size in advance, but if that's possible it might be a good idea to add a check and refuse to flash a file if it's too big.

If it's not possible to auto-detect that we could probably still implement something like that by finding ways to warn the users about that behavior and somehow and making the flashing process a two step process where the users would first download the PIT first and then feed the pit to heimdall in order to enable heimdall to check the partition sizes.

However the --pit option in heimdall flash doesn't seem to be doing that already and it's probably meant for repartitioning only:

$ heimdall download-pit --output gt-i9300.pit
Heimdall v1.4.2

Copyright (c) 2010-2017 Benjamin Dobell, Glass Echidna
http://www.glassechidna.com.au/

This software is provided free of charge. Copying and redistribution is
encouraged.

If you appreciate this software and you would like to support future
development please consider donating:
http://www.glassechidna.com.au/donate/

Initialising connection...
Detecting device...
Claiming interface...
Setting up interface...

Initialising protocol...
Protocol initialisation successful.

Beginning session...

Some devices may take up to 2 minutes to respond.
Please be patient!

Session begun.

Downloading device's PIT file...
PIT file download successful.

Ending session...
Rebooting device...
Releasing device interface...

$ heimdall flash --RECOVERY recovery.img --pit gt-i9300.pit
Heimdall v1.4.2

Copyright (c) 2010-2017 Benjamin Dobell, Glass Echidna
http://www.glassechidna.com.au/

This software is provided free of charge. Copying and redistribution is
encouraged.

If you appreciate this software and you would like to support future
development please consider donating:
http://www.glassechidna.com.au/donate/

Initialising connection...
Detecting device...
Claiming interface...
Setting up interface...

Initialising protocol...
Protocol initialisation successful.

Beginning session...

Some devices may take up to 2 minutes to respond.
Please be patient!

Session begun.

Downloading device's PIT file...
PIT file download successful.

Uploading RECOVERY
100%
RECOVERY upload successful

Ending session...
Rebooting device...
Releasing device interface...
So we may (or may not) need to add a new --with-pit or similar argument in --help along with warnings depending on what's already there in heimdall.

Denis.

Answer 1 · 2021-09-22T09:37:41.000Z

I didn't check yet if it wrote part of the image in the RADIO partition or not.

While I have not checked this either, it used to be in postmarketOS that flashing the boot image (with main OS kernel) to i9300 would result in the recovery (on a different partition) becoming unbootable. This was fixed by making sure the boot image was smaller than the boot partition.

Considering this, I think it's likely that Heimdall just continues writing onto the next partition, but I haven't formally verified this.

Answer 2 · 2021-09-28T12:30:31.000Z

On devices with newer bootloaders an error (without any information about what the issue is) is returned instead. On at least galaxy tab s from 2013/2014 this happens.

I agree that an error from heimdall would be better in any case though, should be possible to implement from the information in the PIT

Answer 3 · 2021-11-30T13:32:35.000Z

It can be implemented very easily.
Just multiply block size by block count, and checks if it's bigger than expected.

P.S. No, it should be impossible as userdata.img.ext4 has 0 blocks, for example, and Block Size may be an offset.
P.S. If it's an offset instead of block size, we can order them by ascending and by using that get the partition sizes.

Answer 4 · 2021-12-10T08:56:54.000Z

Here's a WIP: 12d80ef196de (from this branch: https://git.sr.ht/~grimler/Heimdall/log/size_check)

There are two issues that I have not figured out yet:

The size of a block differs between devices, older have 512 and newer 4096. I hardcoded 512 for now, but we need to get this from the pit, or device, somehow instead.
I tried flashing stock android to three devices, and it failed on one due to the BOOTLOADER (sboot) partition being too small for the file. Maybe this means that the BOOTLODER partition works a bit differently, and that we should not check the size in this case. Could also mean that the device should be repartitioned, maybe odin would even do that automatically for me if I flash stock android with it

Answer 5 · 2021-12-10T10:12:48.000Z

I have to say I am not quite sure about this, but can you check block size from the image file instead of the phone?

$ fdisk -lu boot.img
Disk boot.img: 96 MiB, 100663296 bytes, 196608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

For partitioning I think there is heimdall flash --pit

Answer 6 · 2021-12-10T14:48:40.000Z

@TimoSairiala fdisk, and file, and other tools seem to report/use the same block size no matter which image I look at unfortunately

Looking closer at the pit I think the information is there in the header though, in a part that is not parsed at all right now. For i9300 the PIT starts with:

00000000: 7698 3412 1000 0000 434f 4d5f 5441 5232  v.4.....COM_TAR2
00000010: 4d78 0000 0000 0000 0000 0000 0000 0000  Mx..............
00000020: 0200 0000 5000 0000 0200 0000 0100 0000  ....P...........
00000030: 0000 0000 c606 0000 0000 0000 0000 0000  ................

and my guess is that 0200 at 00000020 is the block size (in multipliers of 256). For herolte we instead have something like:

00000000: 7698 3412 1800 0000 434f 4d5f 5441 5232  v.4.....COM_TAR2
00000010: 4c53 4937 3432 3000 0400 0000 0000 0000  LSI7420.........
00000020: 0800 0000 5000 0000 0200 0000 0100 0000  ....P...........
00000030: 0000 0000 0004 0000 0100 0000 0000 0000  ................

0800 instead of 0200, i.e. 512*4=4096. Will have to check with a couple of more devices to verify if this hypothesis seem to hold.

@TheAirBlow you seem to have documented the pit format for https://samsung-loki.github.io/samsung-docs/docs/PIT/, do you have any guess what the other numbers in second half of row 2, and on row 3 and 4 might be?

Answer 7 · 2021-12-10T15:22:45.000Z

@TimoSairiala fdisk, and file, and other tools seem to report/use the same block size no matter which image I look at unfortunately

Looking closer at the pit I think the information is there in the header though, in a part that is not parsed at all right now. For i9300 the PIT starts with:
00000000: 7698 3412 1000 0000 434f 4d5f 5441 5232  v.4.....COM_TAR2
00000010: 4d78 0000 0000 0000 0000 0000 0000 0000  Mx..............
00000020: 0200 0000 5000 0000 0200 0000 0100 0000  ....P...........
00000030: 0000 0000 c606 0000 0000 0000 0000 0000  ................
and my guess is that 0200 at 00000020 is the block size (in multipliers of 256). For herolte we instead have something like:
00000000: 7698 3412 1800 0000 434f 4d5f 5441 5232  v.4.....COM_TAR2
00000010: 4c53 4937 3432 3000 0400 0000 0000 0000  LSI7420.........
00000020: 0800 0000 5000 0000 0200 0000 0100 0000  ....P...........
00000030: 0000 0000 0004 0000 0100 0000 0000 0000  ................
0800 instead of 0200, i.e. 512*4=4096. Will have to check with a couple of more devices to verify if this hypothesis seem to hold.

@TheAirBlow you seem to have documented the pit format for https://samsung-loki.github.io/samsung-docs/docs/PIT/, do you have any guess what the other numbers in second half of row 2, and on row 3 and 4 might be?

Hello, I am more focused to get Hreidmar GUI working. I'll look at the PIT reader in the aboot.mbn itself when I'll be able to.

P.S. I'll download a few random firmwares for some devices, and check their PITs, and by using math check if the total byte size of every partitions's size will be near the device's eMMC size.

Answer 8 · 2021-12-10T16:12:59.000Z

I confused the offsets, the first pitentry starts at the end of row two, so 0800 0000 / 0200 0000 is not part of the header.

I think the field might still say something about the blocksize though, for i9300 we have 0200 0000 and

--- Entry #0 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 80
Attributes: 2 (STL Read-Only)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 0
Partition Block Count: 1734
Partition Name: BOOTLOADER
Flash Filename: sboot.bin

while for herolte with 0800 0000 we have

--- Entry #0 ---
Binary Type: 0 (AP)
Device Type: 8 (Unknown)
Identifier: 80
Attributes: 2 (STL Read-Only)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 0
Partition Block Count: 1024
File Offset: 1
File Size: 0
Partition Name: BOOTLOADER
Flash Filename: sboot.bin

Device Type: 8 (Unknown) could be mmc, but with blocksize 4096

Answer 9 · 2021-12-16T16:08:16.000Z

I've tagged a release in the repo I am maintaining which contains a commit to compare the file and partition sizes before flashing.

The check is (for now) only based on the actual file size. For sparse images (i.e. {system,cache,userdata,hidden}.img and maybe others) the file size doesn't equal the expanded size, so we still risk flashing something that is too large to these partitions. A better check would instead get the image size from the header of the sparse image.

As discussed above, I've also changed how the PIT header is printed. We seem to have a string "COM_TAR2", followed by a string related to the CPU or bootloader, and then a number that might be a version of some sort (0000, 0003, 0004 and 0005 have been observed, if you have a device with another version please let me know).

Some of the fixes from the various PRs here on github and gitlab are also merged, in particular:

The libusb reset patch needed on ubuntu #478
Support for flashing images larger than 3.5 GB #459
Avoid libusb_set_interface_alt_setting when it anyways doesn't do anything #497, fixed in 2ecc98020c
Some compilation issues on freeBSD, Android and macOS have been fixed.

Pre-compiled binaries can be downloaded for ubuntu 20.04, archlinux and alpine linux. The source can be found and browsed at https://git.sr.ht/~grimler/Heimdall.

Answer 10 · 2021-12-16T16:48:06.000Z

I've tagged a release in the repo I am maintaining which contains a commit to compare the file and partition sizes before flashing.

The check is (for now) only based on the actual file size. For sparse images (i.e. {system,cache,userdata,hidden}.img and maybe others) the file size doesn't equal the expanded size, so we still risk flashing something that is too large to these partitions. A better check would instead get the image size from the header of the sparse image.

As discussed above, I've also changed how the PIT header is printed. We seem to have a string "COM_TAR2", followed by a string related to the CPU or bootloader, and then a number that might be a version of some sort (0000, 0003, 0004 and 0005 have been observed, if you have a device with another version please let me know).

Some of the fixes from the various PRs here on github and gitlab are also merged, in particular:

The libusb reset patch needed on ubuntu #478

Support for flashing images larger than 3.5 GB #459

Avoid libusb_set_interface_alt_setting when it anyways doesn't do anything #497, fixed in 2ecc98020c

Some compilation issues on freeBSD, Android and macOS have been fixed.

Pre-compiled binaries can be downloaded for ubuntu 20.04, archlinux and alpine linux. The source can be found and browsed at https://git.sr.ht/~grimler/Heimdall.

Thanks for researching the PIT innerworkings! Gotta check it with some random devices firmware's PIT to confirm this. Looking for adding this into Hreidmar, and displaying PIT in GUI manner.
I will look more into the header. Never looked into the PIT much, more focused on other things.

Answer 11 · 2021-12-16T18:18:00.000Z

@TheAirBlow Thanks, please let me know if you come in contact with any devices where the last bytes of the header is something other than 0000, 0003, 0004 or 0005 (seems weird that 0001 and 0002 are skipped, if it is a version number), and I'll see if I can write something up for your wiki pages!

Answer 12 · 2021-12-17T04:32:31.000Z

@TheAirBlow Thanks, please let me know if you come in contact with any devices where the last bytes of the header is something other than 0000, 0003, 0004 or 0005 (seems weird that 0001 and 0002 are skipped, if it is a version number), and I'll see if I can write something up for your wiki pages!

You can download random firmwares by using this, unzip, and untar the CSC. Here is the PIT that you need. Use Download&Decrypt mode for this.

Also, Samsung Galaxy A20s (SER) seems to have different numbers in the header.

Answer 13 · 2021-12-17T06:25:36.000Z

Also, I think we need to look into the Odin mode PIT flashing (located in aboot.mbn) and check how it read the PIT

P.S. Let me reboot into Windows real quick and check it out

Answer 14 · 2022-05-25T13:18:01.000Z

"Let me reboot into Windows real quick." - that looks like "famous last words" :'(

Answer 15 · 2022-05-29T19:15:25.000Z

About the PIT format, some devices don't have COM_TAR2.

For instance here's a GT-I9100 PIT: https://git.replicant.us/replicant/vendor_replicant-data/tree/devices/PIT/GT-I9100G/stock/16G.pit

Denis.

Answer 16 · 2022-05-30T13:38:08.000Z

About the PIT format, some devices don't have COM_TAR2.

For instance here's a GT-I9100 PIT: https://git.replicant.us/replicant/vendor_replicant-data/tree/devices/PIT/GT-I9100G/stock/16G.pit

Denis.

@GNUtoo, it is a very old PIT from the times when most of the PIT header was reserved for later use.