Unpacking misses files

Question

Unpacking misses files

glebb opened this issue 4 years ago · 15 comments

Unpacking some specific cdlc's result to missing files or otherwise corrupt output.
Example file:
CF: https://ignition4.customsforge.com/cdlc/58380
Direct download: https://drive.google.com/file/d/1tj4b4mF-1_s6iYkvGlrJuyhFSxmy-Tun/view

Unpacking done as: pyrocksmith --no-crypto --unpack Alice-Cooper_House-of-Fire_v1_0_p.psarc

In case of this specific file, xml files for arrangements under songs/arr folder seem to be missing (except for vocals). This is the only problem I discovered while looking into this, but there might be other problems as well.

As a result, converting the file to mac also doesn't work (target cdlc won't play). If conversion is done using toolkit, the arrangement files are there and cdlc works on mac as well.

Also: --convert doesn't respect --no-crypto, but fixing that doesn't solve the underlying problem.

Answer 1 · 2021-04-27T12:33:04.000Z

Hello,

Thanks for reporting this. Could you share as well the unpack version from RSToolkit ?

--convert cannot follow --no-crypto since we have to redo the encryption to convert

Answer 2 · 2021-04-27T12:49:35.000Z

Here's the unpack produced by RSToolkit: https://drive.google.com/file/d/1KOWk4iglgaXfiquVWzhVpUoRv6Tqk-Uu/view

Answer 3 · 2021-04-27T13:05:15.000Z

I'm completely in the dark what it comes to processing binary files and what the whole encryption in this context even means, but what I meant with the --no-crypto comment is, that like in this case when the source file is not encrypted (why it's not, I have no idea... It's the first time I've encountered such file), doing pyrocksmith --convert will fail as it's unable to read the psarc and --no-crypto has no effect (zlib.error: Error -5 while decompressing data: incomplete or truncated stream). But if the conversion function's PSARC read method is changed (in this case manually) to not use encryption, the conversion succeeds (though the file still doesn't work, as explained in the original description).

Answer 4 · 2021-04-27T15:20:18.000Z

And this file works in game on PC and Mac when converted with RSToolkit ?
I get exactly the same file before encryption and it just looks as if the sng files are corrupted

Answer 5 · 2021-04-27T15:48:38.000Z

I don't have windows running rocksmith at the moment but I guess it works. I know the RSToolkit converted version runs on mac, here's the file I converted myself from the same source: https://drive.google.com/file/d/1Nk5DD7BLJSgS46uqwflsr2yE49JMcDjL/view?usp=sharing

BTW. This converted version unpacks just fine using pyrocksmith. Maybe toolkit is somehow able to fix the sng, if it indeed is corrupted.

Answer 6 · 2021-04-27T15:59:06.000Z

when I compare both the _m and _p versions it looks like the _p version is truncated and missing some parts. My guess is that this particular file does not work on PC

Answer 7 · 2021-04-27T16:25:54.000Z

I had my friend test it and it seems like the original file plays fine on PC.

Answer 8 · 2021-04-27T17:31:26.000Z

After some investigation, it looks to me that the pc sng files are missing the dsa signature at the end: eg i was expecting 56 null bytes at the end of each sng file. Strangely, the mac version you sent me is packing a lot of zeros - which is odd too.
At least I understand why it might work in that case.

In crypto.py you can change

iv, data = data[8:24], data[24:-56]

to

iv, data = data[8:24], data[24:]

That seems to work... but probably need more investigations

Answer 9 · 2021-06-06T20:24:32.000Z

This issue has become frequent during the past month. Thanks for the quick fix, that worked for all my occurences (more than 10).
I'm parsing my entire library again with the fix to see if there's any case where the fix doesn't work, i'll come back to share the results.

Answer 10 · 2021-06-07T10:04:49.000Z

Hi @Warths, thanks for reporting this. Do you have more examples of these failing files ? Where do they come from ? How were they packed ?

You might want to check if that's not a regression. Could you try with this commit cfcc5f7 ?

Answer 11 · 2021-06-07T10:57:39.000Z

Those are failing too :
https://ignition4.customsforge.com/cdlc/59581
https://ignition4.customsforge.com/cdlc/58644
https://ignition4.customsforge.com/cdlc/57352
https://ignition4.customsforge.com/cdlc/26047

There is more for certain, but i didn't record this specific failure before i noticed the issue :/

All of those got updated or created in the past 3 month. No clue what could have changed, i don't recall EoF or RSTK being updated during this time frame.

I'll give feedback about the regression test if i have the time to do it. i'm still parsing my library with the latest commit atm, and it will take a few more days

Answer 12 · 2021-06-07T12:23:08.000Z

Push the fix into master. I still don't understand why these files seem to be missing the sng signatures...
It does not look like a regression from the switch to cryptography

Answer 13 · 2021-06-07T23:24:24.000Z

I've run into a different unpack error. It's not related to the previous one. Didn't know if I should open a new issue
This file : https://drive.google.com/file/d/0B4ciUNd6Y-gLcDhOc3psaGh0ZnM/view?resourcekey=0-FlKkMpe4w0-S749ifFOjng
Result in an AssertionError
The file, however, is healthy as far as i know, and perfectly playable in Rocksmith.

Answer 14 · 2021-06-08T11:01:03.000Z

The xblock file looks corrupted. You can remove the assert

Answer 15 · 2021-11-21T15:39:07.000Z

This works now with master.