[firebase-perf] Fails the build when using dependencies in Multi-release JAR format

Question

[firebase-perf] Fails the build when using dependencies in Multi-release JAR format

mateuszkwiecinski opened this issue 3 years ago · 31 comments

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs in the code in this repository.
If you have a general question, need help debugging, or fall into some
other category use one of these other channels:

For general technical questions, post a question on StackOverflow
with the firebase tag.
For general Firebase discussion, use the firebase-talk
google group.
For help troubleshooting your application that does not fall under one
of the above categories, reach out to the personalized
Firebase support channel.

[REQUIRED] Step 2: Describe your environment

Android Studio version: 🤷‍♀️
Firebase Component: firebase-perf Gradle plugin
Component version: platform('com.google.firebase:firebase-bom:29.0.0')

[REQUIRED] Step 3: Describe the problem

When any of project dependencies use mr-jar format firebase-perf plugin tries to process classes which shouldn't be visible/accesible. In my case it produces exceptions like Record requires ASM8

Execution failed for task ':app:mergeExtDexDebug'.
> Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
   > Failed to transform converter-moshi-2.9.0.jar (com.squareup.retrofit2:converter-moshi:2.9.0) to match attributes {artifactType=android-dex, asm-transformed-variant=debug, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=23, org.gradle.category=library, org.gradle.libraryelements=jar, org.gradle.status=release, org.gradle.usage=java-runtime}.
      > Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
         > Failed to transform moshi-1.13.0-SNAPSHOT.jar (com.squareup.moshi:moshi:1.13.0-SNAPSHOT:20211115.162901-24) to match attributes {artifactType=android-asm-instrumented-jars, asm-transformed-variant=debug, org.gradle.category=library, org.gradle.dependency.bundling=external, org.gradle.jvm.version=8, org.gradle.libraryelements=jar, org.gradle.status=integration, org.gradle.usage=java-runtime, org.jetbrains.kotlin.localToProject=public, org.jetbrains.kotlin.platform.type=jvm}.
            > Execution failed for AsmClassesTransform: ~/.gradle/caches/modules-2/files-2.1/com.squareup.moshi/moshi/1.13.0-SNAPSHOT/6d591b37313d7f58c2cc7776bb2c346ec7b712a8/moshi-1.13.0-SNAPSHOT.jar.
               > Record requires ASM8
   > Failed to transform moshi-1.13.0-SNAPSHOT.jar (com.squareup.moshi:moshi:1.13.0-SNAPSHOT:20211115.162901-24) to match attributes {artifactType=android-dex, asm-transformed-variant=debug, dexing-enable-desugaring=true, dexing-is-debuggable=true, dexing-min-sdk=23, org.gradle.category=library, org.gradle.dependency.bundling=external, org.gradle.jvm.version=8, org.gradle.libraryelements=jar, org.gradle.status=integration, org.gradle.usage=java-runtime, org.jetbrains.kotlin.localToProject=public, org.jetbrains.kotlin.platform.type=jvm}.
      > Execution failed for AsmClassesTransform: ~/.gradle/caches/modules-2/files-2.1/com.squareup.moshi/moshi/1.13.0-SNAPSHOT/6d591b37313d7f58c2cc7776bb2c346ec7b712a8/moshi-1.13.0-SNAPSHOT.jar.
         > Record requires ASM8

I originally reported the issue to Moshi: square/moshi#1424 which then was reported to AGP: https://issuetracker.google.com/issues/206655905 and my motivation here is to ask if there is something you can do to support mr-jars in the two year period before AGP 7.2.0 becomes stable? (plus, at the moment of posting, there is no way to use firebase-perf plugin at all, because the preview AGP version with the fix hasn't been published yet)

Steps to reproduce:

Have an android project with mr-jar dependency
Make sure it compiles
Apply firebase-perf plugin
The build start to fail

Relevant Code:

Repro project (last commit enables firebase-perf plugin)
mateuszkwiecinski/github_browser#151

ZacSweers commented 3 years ago

Thanks!

Answer 1 · 2021-11-20T08:48:15.000Z

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

Answer 2 · 2021-11-22T16:07:08.000Z

Hi @mateuszkwiecinski, thanks for reporting. Unfortunately I cannot reproduce the issue on my end. I tried cloning the repro provided on the link, however it doesn't seem to have any code pertaining to Firebase Performance. So I tried creating a new project with adding multirelease jar dependency (moshi) and Firebase Performance, but I was still unable to repro the issue. Could you could provide us a minimal reproducible example with the issue (MCVE), so we can have a deeper investigation? Thanks!

Relevant code in build.gradle:

implementation("com.squareup.moshi:moshi:1.12.0")
kapt("com.squareup.moshi:moshi-kotlin-codegen:1.12.0")
implementation platform('com.google.firebase:firebase-bom:29.0.0')
implementation 'com.google.firebase:firebase-perf-ktx'

Answer 3 · 2021-11-22T19:24:17.000Z

Hey @argzdev!
You can find minimal repro here mateuszkwiecinski/github_browser#151 (on this specific branch). You can find 2 relevant commits there. First adds all dependencies and you can see the project still compiles on Github Actions, the second one applies firebase-perf plugin which causes the build to fail: commit + GA run

Relevant code in build.gradle:

I think the core difference is you used moshi 1.12.0, and they switched to mr-jar format in 1.13.0 which is not released yet thus only -SNAPSHOT versions are available: https://oss.sonatype.org/content/repositories/snapshots/com/squareup/moshi/moshi-adapters/1.13.0-SNAPSHOT/

Answer 4 · 2021-11-23T14:30:15.000Z

Hi @mateuszkwiecinski, thanks for guiding me through. I was able to clone the project on the specific branch, however I still don't encounter the issue when I sync gradle, build, or even run the app. I also tried changing the version on my sample project but also the same result.

Relevant code - build.gradle:

implementation("com.squareup.moshi:moshi:1.13.0-SNAPSHOT")
kapt("com.squareup.moshi:moshi-kotlin-codegen:1.12.0")
implementation platform('com.google.firebase:firebase-bom:29.0.0')
implementation 'com.google.firebase:firebase-perf-ktx'

settings.gradle

repositories {
    google()
    mavenCentral()
    jcenter() // Warning: this repository is going to shut down soon
    maven {
        url "https://oss.sonatype.org/content/repositories/snapshots/"
        mavenContent { snapshotsOnly() }
    }
}

Am I missing anything?

Answer 5 · 2021-11-23T15:41:21.000Z

I can reproduce the issue with help of @mateuszkwiecinski project by following the steps of the failing Github Action:

git clone git@github.com:mateuszkwiecinski/github_browser.git
cd github_browser
git checkout 006baf9215eb508190447db50039518da3c50543
./gradlew :app:minifyMockedReleaseWithR8

I get the error:

* What went wrong:
Execution failed for task ':app:minifyMockedReleaseWithR8'.
> Could not resolve all files for configuration ':app:mockedReleaseRuntimeClasspath'.
   > Failed to transform moshi-1.13.0-SNAPSHOT.jar (com.squareup.moshi:moshi:1.13.0-SNAPSHOT:20211123.150350-27) to match attributes {artifactType=android-asm-instrumented-jars, asm-transformed-variant=mockedRelease, org.gradle.category=library, org.gradle.dependency.bundling=external, org.gradle.jvm.environment=standard-jvm, org.gradle.jvm.version=8, org.gradle.libraryelements=jar, org.gradle.status=integration, org.gradle.usage=java-runtime, org.jetbrains.kotlin.platform.type=jvm}.
      > Execution failed for AsmClassesTransform: /Users/wasyl/.gradle/caches/modules-2/files-2.1/com.squareup.moshi/moshi/1.13.0-SNAPSHOT/59c588d010076e7f764c96eebfc6de82bcacb68/moshi-1.13.0-SNAPSHOT.jar.
         > Record requires ASM8

Answer 6 · 2021-11-23T16:23:58.000Z

Hi @lwasyl @mateuszkwiecinski, thanks for the detailed steps, I was able to repro the issue. However, I'm not sure if mr-jars are supposed to be supported in the SDK, so I'll mark this as a feature request for now and inform an engineer to see what can be done here.

Answer 7 · 2021-11-23T16:57:42.000Z

Here's minial sample repository https://github.com/mateuszkwiecinski/firebase-perf-doesnt-work in case it's still needd.

git clone git@github.com:mateuszkwiecinski/firebase-perf-doesnt-work.git
cd firebase-perf-doesnt-work
git checkout f76929e361a257e9c28d8b1fde0bc43010b3d48a
./gradlew :app:assembleDebug

@argzdev I'm not sure I understood what does "supported" mean. According to https://issuetracker.google.com/issues/206655905 you do "support" them, in a way fireabse-perf plugins shouldn't have access to unrelated bytecode.
Are you 100% sure firebase-perf plugin should touch parts of the jar that don't meant to be used? that doesn't sound right 🤔

I personally don't like calling this a "feature request" because it's not like some data present in multi release jar file isn't parsed by firebase-perf but it rather completely breaks the project. It is a blocker, without a workaround other than removing firebase-perf depenedncy at all, it makes the plugin unusable, but idk 🤷

Answer 8 · 2021-11-23T17:02:48.000Z

You do raise a good point, @mateuszkwiecinski. I think in this case, it's more valid to tag this as a bug. Thank you both for your findings and explanations, I'm sure our engineers can investigate this further with the given information.

Answer 9 · 2021-11-23T17:23:34.000Z

Hi there folks, I'm raising this with our engineering team so that we can take a look. Thanks for all the info so far! We'll let you know if we have additional questions.

Answer 10 · 2021-11-24T17:55:58.000Z

Hi @mateuszkwiecinski! This issue will be fixed by AGP 7.2.0-beta01, and it is planned for release in three weeks. In addition, it won't take 2 years for it to be stable, 7.2.0 stable is planned to release in mid February of next year. As such, we will treat this as a feature request, since it seems to require a lot of investigation to change the Fireperf plugin implementation to be compatible with the current AGP versions, and it's unlikely that we can release a workaround on the Fireperf plugin side before AGP 7.2.0-beta01 is released.

Let me know if you have more questions!

Answer 11 · 2021-11-24T18:10:20.000Z

All clear, thanks!

Answer 12 · 2021-12-16T07:44:09.000Z

@jfcong - we get a lot of questions from the community that feel hamstrung by firebase's unwillingness to address this bug but also cannot upgrade to a pre-release version of AGP. Would there be a middle ground here to resolve this by either not reading classes in resources/META-INF (they're not going to be run at runtime anyway), gracefully skipping over classes in a newer Java class version that firebase cannot parse, or offering an opt-out mechanism to skip certain classes (jetifier's plugin does this)?

I think there are options available that would both benefit users today while also help avoid future issues like this with the firebase plugin.

Answer 13 · 2021-12-21T23:31:57.000Z

Hi @ZacSweers, thanks for all the suggestions! We are currently low on staffing due to the holiday season, but we will revisit this after the holidays.

Answer 14 · 2022-01-11T20:39:32.000Z

Hi @ZacSweers, we have included a fix in the next Fireperf gradle plugin release where we will filter out classes in META-INF/ directory, so look out for that in a couple of weeks. Thanks!

Answer 15 · 2022-01-21T05:27:41.000Z

looks like this one has been fixed with https://firebase.google.com/support/release-notes/android#performance_gradle_plugin_v1-4-1 ?

Answer 16 · 2022-01-21T10:42:53.000Z

looks like this one has been fixed with https://firebase.google.com/support/release-notes/android#performance_gradle_plugin_v1-4-1 ?

Still getting the same issue with perf-plugin:1.4.1.

Answer 17 · 2022-01-25T21:52:06.000Z

Hi everyone, we've discovered that there is another processing step done by AGP that tries to visit the class to collect information about it whether or not firebase registers interest in visiting the class, so there is no fix available for this issue on Firebase side even when we filtered out classes in the META-INF/ directory from instrumentation. We will update the release note to reflect this.

This should still be fixed with AGP 7.2.0-beta01, but one other workaround to try is to downgrade to a version of Fireperf plugin before we migrated to use the new Transform API. The latest version before the migration was 1.3.5.

Answer 18 · 2022-01-26T05:29:44.000Z

This should still be fixed with AGP 7.2.0-beta01, but one other workaround to try is to downgrade to a version of Fireperf plugin before we migrated to use the new Transform API. The latest version before the migration was 1.3.5.

Downgrading firebase-perf to 1.3.5 with AGP 7.1.0 faced another issue.

> Task :app:transformClassesWithFirebasePerformancePluginForRelease FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:transformClassesWithFirebasePerformancePluginForRelease'.
> 'java.lang.String com.android.utils.FileUtils.relativePossiblyNonExistingPath(java.io.File, java.io.File)'

Downgrading firebase-perf to 1.3.2 and the build passed.

Answer 19 · 2022-01-27T09:54:14.000Z

This situation completely sucks. You can't requires people to use canary release that will not be stable before months just because you are unable to backport a fix to stable release (by the way, this bug was reported 2 months before AGP 7.1 was stable).

Version 1.3.2 breaks gradle configuration cache and is not an option.
Version 1.3.5 is broken when using AGP 7.1.
Version 1.4 don't work with anything but AGP 7.2 which does not exist yet.

I guess removing firebase perfs from my project is today the best options…

Answer 20 · 2022-01-27T10:02:35.000Z

@Jean-Daniel We reached the same conclusion. It's not worth it to keep performance plugin around if it does not work with anything stable. It's truly ridiculous

Answer 21 · 2022-01-27T10:59:22.000Z

You can't requires

You're blaiming wrong people. There root cause lies in Android Gradle Plugin https://issuetracker.google.com/issues/206655905, not in firebase-perf 🤷

Answer 22 · 2022-01-27T11:07:43.000Z

@mateuszkwiecinski Well it's still the same company, so if anybody has an ability to make things faster is another google team. So I guess firebase people should care enough to try and force Android Gradle Plugin team to backport fix for currently stable release. It's in their interest I guess? And no worries, we are complaining there too.

Answer 23 · 2022-01-28T12:06:34.000Z

You're blaiming wrong people. There root cause lies in Android Gradle Plugin

When people write software that crashes because of an Android Bug, they don't redirect the users to Google, they write a workaround and update their software.

You have many options:

Do nothing and tell user to stop using the plugin.
When transforming an unsupported file, emit a warning and skipping it instead of crashing the compilation.
Release a 1.3.6 version that do not use removed API (com.android.utils.FileUtils.relativePossiblyNonExistingPath)

Answer 24 · 2022-01-31T23:06:48.000Z

Hi all, we understand the frustration with the current situation, and we want to provide some updates/workarounds for this issue:

Here are some updates:

AGP v7.2.0-beta01 was made available last week and will work with the latest perf plugin v1.4.1.
AGP 7.2 RC1 is scheduled to release on 1st of March and 7.2 final is scheduled on the 9th of March.

In the mean time, here are some work arounds worth trying out:

Use perf plugin 1.3.5 with AGP v6.8.3. Perf plugin v1.3.5 and below is not compatible with AGP v7.0.0+; in order for the perf plugin to be compatible with AGP v7.0.0+, we have to use the new transform API, which introduces the build failure where even if the perf plugin filters out classes in the META-INF/ directory from instrumentation, AGP still tries to visit the classes to collect information about them. However, AGP is working on a more permanent fix for this where we will have an option to filter out failed classes completely.
Wait until March 9 to integrate the latest perf plugin version and AGP.
Try a beta version of AGP with the latest perf plugin version and move to the production version after March 9th.

We will continue to post updates on this issue as soon as we have them, but in the mean time, please try out the workarounds mentioned above.

Answer 25 · 2022-02-03T13:01:05.000Z

AGP 7.1 release notes mention the following:

Firebase Performance Monitoring compatibility
AGP 7.1 is incompatible with the Firebase Performance Monitoring Gradle plugin version 1.4.0 and lower. The AGP Upgrade Assistant will not automatically update the plugin to version 1.4.1, so if you are using firebase-perf and wish to upgrade AGP to 7.1, you need to do this particular upgrade manually.

But, as per your previous comment (and my personal experience), it is not true that perf plugin 1.4.1 is compatible with AGP 7.1. Am I missing something?

Answer 26 · 2022-02-03T19:05:27.000Z

I think it depends what libraries you include in your product. I you have one with multi JDK support (like Moshi which is commonly use), you can't use 1.4.1.

Answer 27 · 2022-02-04T17:53:21.000Z

Hi @davidmigloz, the release note is referring to general compatibility between the perf plugin and AGP. There's a binary build incompatibility due to https://issuetracker.google.com/issues/210484302 (perf 1.4.0 use variant APIs that were removed in 7.1) which has been fixed in perf plugin 1.4.1. So perf 1.4.1 is compatible with AGP 7.1 in general, unless there are incompatible libraries used, like the one in this github issue.

Answer 28 · 2022-02-05T11:02:08.000Z

Understood, thank you very much for the clarification.

Answer 29 · 2022-03-07T09:37:54.000Z

I think AGP 7.1.2 should resolve this issue, as it includes the following fix - https://issuetracker.google.com/issues/216160972

Answer 30 · 2022-03-07T17:42:52.000Z

As @romtsn said, this issue has a fix cherry-picked into AGP 7.1.2 - closing this issue now. Thanks everyone for your patience and input for resolving this issue!