F-Droid can't build - not RB

Question

F-Droid can't build - not RB

licaon-kter opened this issue 2 years ago · 67 comments

ref: https://monitor.f-droid.org/builds/log/com.starry.myne/9#site-footer

/LE: fyi https://gitlab.com/fdroid/fdroiddata/-/commit/de76cb1a0c45616ec0afcea093ced38f5f1a9c4d

Answer 1 · 2023-03-23T10:52:05.000Z

Also 11 https://gitlab.com/fdroid/fdroiddata/-/commit/ef553af3b3a7c7102f9ee63cdc444dab93d5df7b

verification of APK with copied signature failed
Comparing reference APK to APK with copied signature...
Unexpected diff output:
Binary files /tmp/tmp9ao0125l/unsigned_binaries_com.starry.myne_11.binary/content/assets/dexopt/baseline.prof and /tmp/tmp9ao0125l/_tmp_tmp9ao0125l_sigcp_com.starry.myne_11/content/assets/dexopt/baseline.prof differ
Binary files /tmp/tmp9ao0125l/unsigned_binaries_com.starry.myne_11.binary/content/classes.dex and /tmp/tmp9ao0125l/_tmp_tmp9ao0125l_sigcp_com.starry.myne_11/content/classes.dex differ
diff -r /tmp/tmp9ao0125l/unsigned_binaries_com.starry.myne_11.binary/content/res/GM.json /tmp/tmp9ao0125l/_tmp_tmp9ao0125l_sigcp_com.starry.myne_11/content/res/GM.json
1,3362c1,3362
< {
<   "v": "5.7.8",
<   "fr": 24,
<   "ip": 0,
<   "op": 151,
<   "w": 1080,
<   "h": 1080,
<   "nm": "EMOJI_CORACAO",
...

Answer 2 · 2023-03-23T11:10:53.000Z

Thanks for notifying,

Also 11 https://gitlab.com/fdroid/fdroiddata/-/commit/ef553af3b3a7c7102f9ee63cdc444dab93d5df7b

verification of APK with copied signature failed
Comparing reference APK to APK with copied signature...
Unexpected diff output:
Binary files /tmp/tmp9ao0125l/unsigned_binaries_com.starry.myne_11.binary/content/assets/dexopt/baseline.prof and /tmp/tmp9ao0125l/_tmp_tmp9ao0125l_sigcp_com.starry.myne_11/content/assets/dexopt/baseline.prof differ
Binary files /tmp/tmp9ao0125l/unsigned_binaries_com.starry.myne_11.binary/content/classes.dex and /tmp/tmp9ao0125l/_tmp_tmp9ao0125l_sigcp_com.starry.myne_11/content/classes.dex differ
diff -r /tmp/tmp9ao0125l/unsigned_binaries_com.starry.myne_11.binary/content/res/GM.json /tmp/tmp9ao0125l/_tmp_tmp9ao0125l_sigcp_com.starry.myne_11/content/res/GM.json
1,3362c1,3362
< {
<   "v": "5.7.8",
<   "fr": 24,
<   "ip": 0,
<   "op": 151,
<   "w": 1080,
<   "h": 1080,
<   "nm": "EMOJI_CORACAO",
...

This looks like json file for some Lottie animation, I'll have to check mappings.txt file of the last build to confirm though, but I'm not sure why this would cause issues 🤔

Answer 3 · 2023-03-23T11:22:17.000Z

.json aside for 11 classes.dex and baseline differ, will need to diff them locally next

ref: https://f-droid.org/docs/Reproducible_Builds/#potential-sources-of-unreproducible-builds

Answer 4 · 2023-03-23T16:49:37.000Z

.json aside for 11 classes.dex and baseline differ, will need to diff them locally next

Thanks, please update me if you find something which i could change to fix it, as i don't really havs much knowledge/experience of reproducible builds 😅

Answer 5 · 2023-03-24T17:21:12.000Z

Here's the diff log:
my.zip

Answer 6 · 2023-03-26T02:27:00.000Z

Here's the diff log: my.zip

Thanks, but unfortunately I've no idea how to read/understand what went wrong using these logs, as i don't really know anything about reproducible builds. Can you link me some guide which i can read to check reproducibility of builds myself using whatever tool used by f-droid to ensure reproducibility.

Answer 7 · 2023-03-27T06:46:48.000Z

As linked the base prof thing can be seen here: https://f-droid.org/docs/Reproducible_Builds/#potential-sources-of-unreproducible-builds

@linsui is the rest of the diff log a know type already?

Answer 8 · 2023-03-27T06:56:57.000Z

Looks like only the java part is different.

Answer 9 · 2023-03-27T06:58:28.000Z

@licaon-kter I've read your linked article but wasn't able to figure out what exactly was causing it, I'm using gradle 7.4.x so gradle issue as linked in that article shouldn't be a culprit, same goes for coreLibraryDesugaring as app doesn't uses core library desugaring, about windows newlines, we've talked about and fixed it in app's yml file in f-droid data repository when i opened a RFP issue for the listing and timestamp related issue was also resolved around same time in the merge request. And other reasons about NDK and build paths etc aren't related here as far as i can understand.

Answer 10 · 2023-03-27T08:14:32.000Z

fyi https://gitlab.com/fdroid/fdroiddata/-/commit/3d4883f3635f24dffd49e6e2e48df91634548e24

Answer 11 · 2023-03-27T08:17:31.000Z

@starry-shivam can you build an APK from commit: 6f9af6d with these two as false? https://github.com/Pool-Of-Tears/Myne/blob/main/app/build.gradle#L37-L38

Answer 12 · 2023-03-27T08:19:15.000Z

@starry-shivam can you build an APK from commit: 6f9af6d with these two as false? https://github.com/Pool-Of-Tears/Myne/blob/main/app/build.gradle#L37-L38

Sure, will do in the evening.

Answer 13 · 2023-03-27T13:14:42.000Z

@licaon-kter here you go, I've uploaded it on mediafire as i couldn't find any ways to upload it here directly.

Answer 14 · 2023-04-04T14:58:05.000Z

Hello, any updates on this?

Answer 15 · 2023-04-05T10:06:36.000Z

Better ;)

The baseline issue might be fixed as explained here: https://f-droid.org/docs/Reproducible_Builds/#potential-sources-of-unreproducible-builds

The rest is still in lottie.json: myne1.log

Answer 16 · 2023-04-05T17:17:26.000Z

The baseline issue might be fixed

Cool!

The rest is still in lottie.json: myne1.log

Any idea what can i do to solve this? I'm not sure why it'll cause the issue..

Answer 17 · 2023-04-05T17:19:09.000Z

Maybe you can sort it before it's packaged?

Answer 18 · 2023-04-05T17:21:34.000Z

Maybe you can sort it before it's packaged?

Sort it like? It's basically a json file stored in the raw resource directory.

Answer 19 · 2023-04-05T17:25:09.000Z

Oh, sorry, it's not a order problem. Maybe it's about the line end. Then we can fix it on our side.

Answer 20 · 2023-04-06T01:02:06.000Z

Oh, sorry, it's not a order problem. Maybe it's about the line end. Then we can fix it on our side.

Nice! Btw since we're here, can anyone check why the screenshots for both of my apps this and GreenStash are kinda mixed up with old screenshots (which I've already deleted from fastlane directory long ago) Idk why it happens but it looks pretty weird..

Answer 21 · 2023-04-06T02:47:46.000Z

I thought it's a known issue. But I don't know how to fix it.

Answer 22 · 2023-04-06T03:52:22.000Z

We do know but not yet fixed :)

Answer 23 · 2023-04-06T04:44:34.000Z

alright, not a big deal.

Answer 24 · 2023-08-10T10:54:41.000Z

Trying 2.3.0

diff log: myne230.log

local APK (Java17): com.starry.myne_17.apk.zip (remove .ZIP extenstion)

upstream APK: https://github.com/Pool-Of-Tears/Myne/releases/tag/v2.3.0

@linsui @obfusk

Answer 25 · 2023-08-10T16:36:45.000Z

Looks like broken resources: https://issuetracker.google.com/issues/287967713
Also one annotation in classes.dex, and newlines in META-INF/services/.

@licaon-kter the broken resources are on your end this time, and so are the windows newlines in META-INF/services/; that's really weird. Anything change in your setup?

Answer 26 · 2023-08-16T10:57:55.000Z

@obfusk

Anything change in your setup?

Not that I know, and doesn't affect any other apps either. 🤷

Answer 27 · 2023-08-16T11:32:01.000Z

I've rebuilt in another instance and my builds are repro between them

VM images rebuilt April 20 and June 30, fyi

Answer 28 · 2023-08-16T14:40:25.000Z

Ive rebuilt in another instance and my builds are repro between them

That's good to know. I see Google increased the priority and severity of the bug. Hopefully they'll fix it or at least give us some idea of what's going on. I don't really know how to debug this.

Answer 29 · 2023-10-18T07:36:38.000Z

Any updates on this?

Answer 30 · 2023-10-18T11:42:03.000Z

2.6.0 is fine https://gitlab.com/fdroid/fdroiddata/-/commit/8dbf581aae28376a5ab70e7ad5ad51cb5877c739 :)

Answer 31 · 2024-02-20T07:48:27.000Z

and again https://gitlab.com/fdroid/fdroiddata/-/jobs/6201720684#L4340

will add a diff asap

Answer 32 · 2024-02-20T08:06:11.000Z

I think it was probably because of different API key in the github builds see #114 i did that because apparently some people were using the publicly available API key to hit rate limits resulting in metadata not being loaded from time to time.

Answer 33 · 2024-02-20T08:16:39.000Z

indeed, the only difference is the API key, in clear text 😄 🤷

Answer 34 · 2024-02-20T08:24:59.000Z

Yes, but at least it's not publicly available on GitHub. If someone wanted it, they would need to go through the hoops of decompiling and finding it out. I'm perfectly fine with F-Droid using it though if it's somehow possible to pass it via F-Droid's build process, like I'm doing in my GitHub actions..

Answer 35 · 2024-02-20T08:40:24.000Z

No, everything that is used to build the APK needs to be in "the open", no secrets allowed

Yes, we can add it to the recipe: https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/com.starry.myne.yml#L143

but then "If someone wanted it, they would need to go through the hoops of opening the recipe on Gitlab and finding it out."

Answer 36 · 2024-02-20T08:56:49.000Z

Still better than being publicly available here on GitHub for anyone take and abuse it 🤷 can you give me some email address I'll email it to you, or if you have telegram please msg me at https://t.me/starryboi

Answer 37 · 2024-02-20T09:00:01.000Z

At best we can add a base64 so that it won't be scraped easily.

Answer 38 · 2024-02-20T09:02:17.000Z

We can see the key by comparing the apks, right? 🤣 There are libs that decrypting the key at runtime so it would be harder to extract it.

Answer 39 · 2024-02-20T09:19:27.000Z

At best we can add a base64 so that it won't be scraped easily.

It'll be nice, thanks

Answer 40 · 2024-02-20T09:19:47.000Z

yeah, I've already seen it 😄 ends in ...LgY yes?

What I don't know @starry-shivam where, how to inject it during build

Answer 41 · 2024-02-20T09:29:35.000Z

yeah, I've already seen it 😄 ends in ...LgY yes?

Correct

What I don't know @starry-shivam where, how to inject it during build

You just need to push it inside local.properties, like here.

Answer 42 · 2024-02-20T10:01:46.000Z

testing locally

Answer 43 · 2024-02-20T10:15:06.000Z

🎉 https://gitlab.com/fdroid/fdroiddata/-/commit/bcc9b00d0eea654f1caf200fd0d491b36e7a34d0

Answer 44 · 2024-07-01T03:12:06.000Z

https://monitor.f-droid.org/builds/log/com.starry.myne/391#site-footer

Only the zip metadata is different.

Answer 45 · 2024-07-01T03:25:20.000Z

https://monitor.f-droid.org/builds/log/com.starry.myne/391#site-footer

Only the zip metadata is different.

Weird, I didn't even make many changes in the latest release. You can check for changes here: v3.9.0...v3.9.1

Answer 46 · 2024-07-01T03:52:42.000Z

Please test if you can still reproduce 3.9.0 apk.

Answer 47 · 2024-07-01T03:56:24.000Z

Please test if you can still reproduce 3.9.0 apk.

How can i test it? Should i build an apk from v3.9.0 tag and send it here?

Answer 48 · 2024-07-01T04:05:07.000Z

You can build and sign it and compare it with the apk in release with diff.

Answer 49 · 2024-07-01T04:24:25.000Z

You can build and sign it and compare it with the apk in release with diff.

I don't know which tool you use to compare apks. Should I just try to compare the SHA-256 hashes of both files or something like that?

Answer 50 · 2024-07-01T04:26:42.000Z

Yes, the hash should be the same. If the hash is different, please also check the content of the apk with diff -r. You can use diffoscope to inspect it.

Answer 51 · 2024-07-02T17:07:15.000Z

@linsui The hashes are different, but I think it is most likely because the GitHub build is compiled in a Linux environment with gradle directly, while the one I built is compiled with Android Studio on Windows 11. There must be different line endings (CRLF?), and Android Studio itself might do some extra stuff when archiving the APK file, though I'm not sure about that. However, diff -r didn't show anything useful, like what exactly are the differences, other than just indicating they are different.

Answer 52 · 2024-07-02T17:10:22.000Z

Please unzip them then run diff -r. :) If you want to check the apk directly you can use diffoscope but I thought it's not available on Windows.

Answer 53 · 2024-07-02T17:29:30.000Z

I checked the diff with Meld and it shows these three files which are present in the GitHub build of v3.9.0 but missing in the build I compiled just now with Android Studio. They seem to be some signing-related keys, though I’ve signed this build with the same signature that I use for the GitHub builds.

Answer 54 · 2024-07-02T18:06:13.000Z

They are signature files. Hmmm, that shouldn't affect our reproducible build. Can you upload the unsigned apk here?

Answer 55 · 2024-07-03T05:24:19.000Z

They are signature files. Hmmm, that shouldn't affect our reproducible build. Can you upload the unsigned apk here?

Sure, but it seems GitHub doesn't support uploading APK files in comments. I'll upload it somewhere else and share the link in a moment.

Answer 56 · 2024-07-03T05:35:19.000Z

You can zip it.

Answer 57 · 2024-07-03T06:42:26.000Z

You can zip it.

Oh, good idea. Here you go
Myne-v3.9.0.zip

Answer 58 · 2024-07-03T07:51:03.000Z

Your build has other difference. I thought they are mainly caused by CRLF.

Answer 59 · 2024-07-03T14:15:30.000Z

Your build has other difference. I thought they are mainly caused by CRLF.

Hmm, maybe. Still, I'm not really sure what exactly caused v3.9.1 to become unreproducible, especially considering I didn't even make any major changes between v3.9.0 and v3.9.1 like I usually do with other releases.

Answer 60 · 2024-07-03T14:38:57.000Z

Yep, we also see this problem on other apps. It seems a toolchain is changed but we don't know what it is yet.

Answer 61 · 2024-07-05T15:50:06.000Z

@starry-shivam ref: https://gitlab.com/fdroid/fdroiddata/-/issues/3299#note_1985568674

so around here, I guess: https://github.com/Pool-Of-Tears/Myne/blob/v3.9.1/.github/workflows/release.yml#L55 make it do apksigner sign --alignment-preserved true ... (you are using some other external actions so not sure how to inject that in https://github.com/r0adkll/sign-android-release/blob/f30bdd30588842ac76044ecdbd4b6d0e3e813478/lib/signing.js#L66 )

eg. others have another setup like: https://github.com/ImranR98/Obtainium/blob/v1.1.13/.github/workflows/release.yml#L75

Answer 62 · 2024-07-08T04:16:02.000Z

you are using some other external actions so not sure how to inject that in https://github.com/r0adkll/sign-android-release/blob/f30bdd30588842ac76044ecdbd4b6d0e3e813478/lib/signing.js#L66 )

Looking at the code, I don't think the GitHub Actions plugin allows users to pass additional arguments. I think I'll have to run APK signer manually. I'll try and see how it goes.

Answer 63 · 2024-07-08T14:02:38.000Z

I've been trying to manually sign (#192), but I keep getting a "command not found" error. The error message isn't helpful because it doesn't specify which command has failed or the correct line number in the workflow file. I suspect it doesn't find the apksigner command, but the ANDROID_HOME environment variable seems to point to the correct path: /usr/local/lib/android/sdk. It would be great if someone could look at this and see if I'm doing something wrong. I'm not particularly good with GitHub Actions, which is why I was using a third-party plugin to get the job done.

Answer 64 · 2024-07-08T14:33:50.000Z

ls: cannot access 'app/build/outputs/apk/release/app-release-unsigned.apk': No such file or directory

Answer 65 · 2024-07-09T13:08:24.000Z

Based on the docs here: https://github.com/obfusk/apksigcopier#what-about-signatures-made-by-apksigner-from-build-tools--3500-rc1

I've updated the recipe instead: https://gitlab.com/fdroid/fdroiddata/-/commit/be7626377555aa92f37e9310a3e7b36bc04df195

feel free to star the Google issue https://issuetracker.google.com/issues/351408623

Answer 66 · 2024-07-09T13:08:38.000Z

so #192 can be closed

Answer 67 · 2024-07-09T17:47:54.000Z

Based on the docs here: https://github.com/obfusk/apksigcopier#what-about-signatures-made-by-apksigner-from-build-tools--3500-rc1

I've updated the recipe instead: https://gitlab.com/fdroid/fdroiddata/-/commit/be7626377555aa92f37e9310a3e7b36bc04df195

feel free to star the Google issue https://issuetracker.google.com/issues/351408623

Wow thanks! I'll close that PR now.