e-mission/e-mission-docs

⬆️ Track upgrade to cordova android@9.0.0 and ios@6.0.1

shankari opened this issue · 44 comments

shankari commented

This bug is a follow on to #519, tracking the upgrade to the even more recent versions of cordova.

shankari commented

There are a few plugins that don't install correctly:

  • phonegap-plugin-push because of dependency cordova-support-google-services@1.3.1
  • cordova-plugin-x-socialsharing; because of dependency es6-promise-plugin@4.2.2
  • cordova-plugin-local-notification because of dependency cordova-plugin-badge@0.8.8
  • all of our plugins, because of invalid format
shankari commented

Let's first fix our plugins, so that we know what the correct format should look like

shankari commented

There is also a workaround:

We can first install the "broken" plugin using npm directly, which works:

$ npm install cordova-plugin-badge@0.8.8
npm WARN edu.berkeley.eecs.emission@2.8.1 No repository field.
npm WARN edu.berkeley.eecs.emission@2.8.1 No license field.

+ cordova-plugin-badge@0.8.8
added 1 package from 1 contributor, removed 1 package and audited 1706 packages in 11.249s

47 packages are looking for funding
  run `npm fund` for details

found 173 vulnerabilities (45 low, 75 moderate, 53 high)
  run `npm audit fix` to fix them, or `npm audit` for details

And then if we try to install the dependent plugin, it finds the "fetched" plugin and installs it.

$ npx cordova plugin add cordova-plugin-local-notification
Installing "cordova-plugin-local-notification" for android
android-sdk version check failed (/Users/kshankar/e-mission/upgrade_platform/platforms/android/cordova/android_sdk_version), continuing anyways.
Plugin dependency "cordova-plugin-device@2.0.3" already fetched, using that version.
Dependent plugin "cordova-plugin-device" already installed on android.
Installing "cordova-plugin-badge" for android
Subproject Path: CordovaLib
Subproject Path: app
Subproject Path: CordovaLib
Subproject Path: app
Installing "cordova-plugin-local-notification" for ios
Plugin dependency "cordova-plugin-device@2.0.3" already fetched, using that version.
Dependent plugin "cordova-plugin-device" already installed on ios.
Plugin dependency "cordova-plugin-badge@0.8.8" already fetched, using that version.
Installing "cordova-plugin-badge" for ios
Adding cordova-plugin-local-notification to package.json

$ npx cordova plugin list | grep notification
cordova-plugin-local-notification 0.9.0-beta.2 "LocalNotification"
shankari commented

For the non-emission plugins, they don't list their dependencies in package.json. This means that when we use npm install, their dependencies are not fetched, and they can't be installed. However, the dependencies are listed in their plugin.xml, so when we run npx cordova prepare, they are retrieved. A second npx cordova prepare successfully installs them.

$ npm install

> fsevents@1.2.13 install /Users/kshankar/e-mission/upgrade_platform/node_modules/fsevents
> node install.js

added 1707 packages from 1157 contributors and audited 1707 packages in 25.55s

47 packages are looking for funding
  run `npm fund` for details

found 173 vulnerabilities (45 low, 75 moderate, 53 high)
  run `npm audit fix` to fix them, or `npm audit` for details

$ ls -al node_modules/cordova-plugin-badge
ls: node_modules/cordova-plugin-badge: No such file or directory
$ ls -al node_modules/cordova-support-google-services
ls: node_modules/cordova-support-google-services: No such file or directory

$ npx cordova prepare
Failed to install 'phonegap-plugin-push': CordovaError: Failed to fetch plugin cordova-support-google-services@~1.3.1 via registry.
Probably this is either a connection problem, or plugin spec is incorrect.
Check your connection and plugin name/version/URL.
Could not determine package name from output:
cordova-support-google-services@1.3.2 node_modules/cordova-support-google-services
    at /Users/kshankar/e-mission/upgrade_platform/node_modules/cordova-lib/src/plugman/fetch.js:146:43
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
Failed to restore plugin "phonegap-plugin-push" from config.xml. You might need to try adding it again. Error: Failed to fetch plugin cordova-support-google-services@~1.3.1 via registry.
Probably this is either a connection problem, or plugin spec is incorrect.
Check your connection and plugin name/version/URL.
Could not determine package name from output:
cordova-support-google-services@1.3.2 node_modules/cordova-support-google-services
...
Failed to install 'cordova-plugin-x-socialsharing': CordovaError: Failed to fetch plugin es6-promise-plugin@^4.1.0 via registry.
Probably this is either a connection problem, or plugin spec is incorrect.
Check your connection and plugin name/version/URL.
Could not determine package name from output:
es6-promise-plugin@4.2.2 node_modules/es6-promise-plugin
    at /Users/kshankar/e-mission/upgrade_platform/node_modules/cordova-lib/src/plugman/fetch.js:146:43
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
Failed to restore plugin "cordova-plugin-x-socialsharing" from config.xml. You might need to try adding it again. Error: Failed to fetch plugin es6-promise-plugin@^4.1.0 via registry.
Probably this is either a connection problem, or plugin spec is incorrect.
Check your connection and plugin name/version/URL.
Could not determine package name from output:
es6-promise-plugin@4.2.2 node_modules/es6-promise-plugin
....
Failed to install 'cordova-plugin-local-notification': CordovaError: Failed to fetch plugin cordova-plugin-badge@>=0.8.5 via registry.
Probably this is either a connection problem, or plugin spec is incorrect.
Check your connection and plugin name/version/URL.
Could not determine package name from output:
cordova-plugin-badge@0.8.8 node_modules/cordova-plugin-badge
    at /Users/kshankar/e-mission/upgrade_platform/node_modules/cordova-lib/src/plugman/fetch.js:146:43
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
Failed to restore plugin "cordova-plugin-local-notification" from config.xml. You might need to try adding it again. Error: Failed to fetch plugin cordova-plugin-badge@>=0.8.5 via registry.
Probably this is either a connection problem, or plugin spec is incorrect.
Check your connection and plugin name/version/URL.
Could not determine package name from output:
cordova-plugin-badge@0.8.8 node_modules/cordova-plugin-badge
...
<emission plugin failures>

$ ls -al node_modules/cordova-plugin-badge
total 80
...
$ ls -al node_modules/cordova-support-google-services
total 40

$ npx cordova prepare
<emission plugin failures only>

$ npx cordova plugin list
...
cordova-plugin-local-notification 0.9.0-beta.2 "LocalNotification"
cordova-plugin-x-socialsharing 5.6.8 "SocialSharing"
phonegap-plugin-push 2.3.0 "PushPlugin"
...
shankari commented

For the e-mission plugins, there are two issues:

shankari commented

Next steps:

Depending on the responses to the cordova-lib bug, decide whether to upgrade to cordova@9.0.0 or not

  • if the bug is not resolved, this will involve publishing our plugins
  • but how will we deal with the private versions of the other plugins?
shankari commented

Ended up resolving https://github.com/apache/cordova-lib/issues/845 instead because I got sucked into it. The issue was with an old version of phonegap in the dependencies. Upgrading to a newer version of phonegap fixes everything in #554 (comment)

Our plugins still fail because of

Failed to install 'edu.berkeley.eecs.emission.cordova.usercache': CordovaError: Using "requireCordovaModule" to load non-cordova module "fs" is not supported. Instead, add this module to your dependencies and use regular "require" to load it.

So modified next steps:

  • Switch to the most recent version of all plugins
  • Fix our plugins
    • Edit the data collection plugin to use a wrapper class for the activities (e-mission/e-mission-data-collection#80)
    • Change the hook for all of them as per the instructions above
    • Fix all the e-mission plugins to have the correct name (should do it anyway)
    • Switch to the most recent version of google-play-services (we are using 11, current is 17)
  • Fix local notification (potentially upgrade to the android10 changes)
  • (optional) create a private version of phonegap-plugin-push that doesn't depend on cordova-plugin-google-services
shankari commented

Switch to the most recent version of all plugins

Installed and ran cordova-plugin-update

It detected and applied the following updates

cordova-plugin-customurlscheme (4.3.0 => 5.0.1)
cordova-plugin-email-composer (0.8.15 => 0.9.2)
cordova-plugin-x-socialsharing (5.6.5 => 6.0.0)
cordova-plugin-inappbrowser (3.2.0 => 4.0.0)
cordova-plugin-local-notification (0.9.0-beta.2 => 0.9.0-beta.3)
cordova-plugin-advanced-http (2.5.1 => 3.0.0)
cordova-plugin-ionic-webview (4.2.1 => 5.0.0)
shankari commented

While comparing the changes, the newly added plugin had removed "OKHTTP_VERSION": "3.10.0" from cordova-plugin-advanced-http. If we start getting any weird crashes, might need to add that back.

shankari commented

After these upgrades, iOS compiles. Making the hook changes first so that we can get android to compile as well and start checking in partial updates.

shankari commented

Made hook changes and fixed all names. Was able to add all plugins.
Current status:

  • iOS still compiles
  • android fails with the error
> This project uses AndroidX dependencies, but the 'android.useAndroidX' property is not enabled. Set this property to true in the gradle.properties file and retry.

The only plugin that appears to currently use AndroidX is

$ grep -r androidx plugins/
plugins//cordova-plugin-x-socialsharing/plugin.xml:    <framework src="androidx.legacy:legacy-support-v4:1.0.0" />
plugins//cordova-plugin-x-socialsharing/src/android/nl/xservices/plugins/FileProvider.java:public class FileProvider extends androidx.core.content.FileProvider {

Downgrading it to 5.6.8, upgrading gradle version to 6.5 and retrying.

shankari commented

both android and iOS now compile successfully.

Current status:

  • Fix local notification (potentially upgrade to the android10 changes)
  • Switch to the most recent version of all plugins
  • Fix our plugins
    • Edit the data collection plugin to use a wrapper class for the activities (e-mission/e-mission-data-collection#80)
    • Change the hook for all of them as per the instructions above
    • Fix all the e-mission plugins to have the correct name (should do it anyway)
    • Switch to the most recent version of google-play-services (we are using 11, current is 17)

Next step: Edit the data collection plugin

shankari commented

Actually, next step: trying to run the app 😄

app runs fine in android, but basically hangs in iOS. This is really weird because it gets to

[Log] Finished changing state from {"url":"/splash","templateUrl":"templates/splash/splash.html","controller":"SplashCtrl","name":"splash"} to {"url":"/intro","templateUrl":"templates/intro/intro.html","controller":"IntroCtrl","name":"root.intro"} (cordova.js, line 1413)

But the intro template is not displayed. I should probably revert to the older version of the webview, ensure that it works and then roll forward.

shankari commented

ios combinations are below:

cordova-plugin-ionic-webview cordova-ios works?
4.2.1 5.1.1 ✔️
5.0.0 5.1.1 ✔️
4.2.1 6.1.0 ❌
5.0.0 6.1.0 ❌

So basically, the upgrade to cordova-ios@6.1.0 breaks ionic webview.

shankari commented

Created MCVE
https://github.com/shankari/ionic-webview-mcve
and filed related issue ionic-team/cordova-plugin-ionic-webview#594

Let's stay on cordova-ios@5.1.1 until this is resolved.

Now, next step is to edit the data collection plugin

shankari commented

while editing the data collection plugin, one thought is whether we should send the incoming data directly as an int or as text. Right now, we get it as text and convert to int. But the default parameter on android is int, so we could just sent the int directly.

Decision is to send the int directly.

Pros:

As long as the ints are stable (and they have been for many years now), this should be stable.

shankari commented

done editing the data collection plugin, but can't yet confirm that it works because I can't install the apk on the phone. Let's upgrade to the most recent version of the google play services library and test everything together.

shankari commented

Switching to the most recent version of the google play services library (17.0.0) introduces androidx.
We can switch to androidx, but we can't have any code that still depends on the android support libraries.

Searching for android.support in the plugins, we get a fairly long list of dependencies, which effectively come down to:

  • our plugins

    • cordova-plugin-em-datacollection
    • cordova-plugin-em-unifiedlogger
    • cordova-plugin-em-jwt-auth
    • cordova-plugin-em-transition-notify
    • cordova-plugin-em-settings
    • cordova-plugin-em-serversync

  • other plugins

    • cordova-plugin-local-notification
    • phonegap-plugin-push
    • cordova-plugin-x-socialsharing
    • cordova-plugin-email-composer

We can fix all of our plugins. For the other plugins:

If it is hard to fix the first two, we can just use https://github.com/dpa99c/cordova-plugin-androidx-adapter which uses a cordova hook to fix the source code dynamically.

shankari commented

Removed all our android.support dependencies EXCEPT in jwt-auth. There is a recent checkin to the AppAuth library to switch to androidx but there has not been a release since 2018. I tried to use jitpack to install the master branch, and got a compile error with importing an auth0 library. I can't find that code in the repo https://github.com/openid/AppAuth-Android

Going back to the working version of the AppAuth library since I am not sure who (if anybody) is using AppAuth. Might remove OpenID support if nobody steps up to test it.

Also, the new cordova-local-notification plugin does have 2 vestigial references to android.support

Fortunately, it looks like setting useAndroidX to true also sets enableJetifier to true.

[Gradle Properties] Detected Gradle property "android.useAndroidX" with the value of "true", Cordova's recommended value is "false"
[Gradle Properties] Detected Gradle property "android.enableJetifier" with the value of "true", Cordova's recommended value is "false"

According to the docs:

android.enableJetifier: When this flag is set to true, the Android plugin automatically migrates existing third-party libraries to use AndroidX dependencies by rewriting their binaries. The flag is false by default if it is not specified.

So maybe this will automagically work for these few holdouts.

shankari commented

Remaining instances of android.support

platforms/android/app/src//main/java/de/appplant/cordova/plugin/notification/Builder.java
platforms/android/app/src//main/java/de/appplant/cordova/plugin/notification/Options.java

platforms/android/app/src//main/java/de/appplant/cordova/emailcomposer/Provider.java

platforms/android/app/src//main/java/com/adobe/phonegap/push/PushPlugin.java
platforms/android/app/src//main/java/com/adobe/phonegap/push/FCMService.java
platforms/android/app/src//main/java/com/adobe/phonegap/push/BackgroundActionButtonHandler.java
platforms/android/app/src//main/java/com/adobe/phonegap/push/PushHandlerActivity.java
shankari commented

So maybe this will automagically work for these few holdouts.

jetifier did not work

/Users/kshankar/e-mission/upgrade_platform/platforms/android/app/src/main/java/com/adobe/phonegap/push/FCMService.java:28: error: cannot find symbol
import android.support.v4.app.RemoteInput;
                             ^

...

Let's try the workaround plugin next https://github.com/dpa99c/cordova-plugin-androidx-adapter

shankari commented

that did work. Woo!

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.5/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 7s
40 actionable tasks: 16 executed, 24 up-to-date
Built the following apk(s):
	/Users/kshankar/e-mission/upgrade_platform/platforms/android/app/build/outputs/apk/debug/app-debug.apk

We can still migrate plugins or submit PRs, but this seems like it is close to being done.

shankari commented

Current status:

  • Switch to the most recent version of all plugins
  • Fix our plugins
    • Edit the data collection plugin to use a wrapper class for the activities (e-mission/e-mission-data-collection#80)
    • Change the hook for all of them as per the instructions above
    • Fix all the e-mission plugins to have the correct name (should do it anyway)
    • Switch to the most recent version of google-play-services (we are using 11, current is 17) (testing pending)
  • Fix local notification (potentially upgrade to the android10 changes)

Next steps: Fixing local notification + testing

shankari commented

Switching to the most recent version of google-play-services also allows us to use the potentially lighter weight method to interact with the Google Play API
https://android-developers.googleblog.com/2017/11/moving-past-googleapiclient_21.html

This simplifies the code for the services significantly and can probably simplify it even further if we had the daring to refactor further. Performed first level of simplification: switching from PendingResult to Task and handling all the resulting code refactoring. Will remove all references to GoogleApiClient and the connect/disconnect code in the next step. Any further rework will wait until we have finished testing this.

shankari commented

While testing this in the emulator, we get ~ 7-8 location intents and then we stop getting them. Need to test on real phones before proceeding with the refactoring.

shankari commented

Trying to test on an older emulator. Alas, it generates a WSOD. Upgrading the emulator's chrome to the most recent version to retry...
e.g. appium/appium#7179

Note that there are multiple apk sites and we may need to use multiple of them to find an apk that works with our desired API level.
e.g. the apkmirror highlighted in the answer above only had apks with minSDK=AndroidO so they wouldn't install.
while https://apkpure.com/android-system-webview/com.google.android.webview had an apk that did install even on Marshmallow.

shankari commented

I think that this is a problem with the emulator since it happens on pre-O as well. Will test on real phones later. Let's test some corner cases wrt enabling and disabling location. Tried turning location permissions on/off and location type to device-only. We don't get the promised Geofence exit with error if the phone is in "waiting_for_trip_start". But on initialize, we get the proper prompts to turn everything on.

Moving on to second round of refactoring and then trying to fix trip end notifications.

shankari commented

Second round of refactoring also done. Moving on to trip end notification fixing (at least on android).

shankari commented

After some minor fixes, trip end notifications on both android and iOS work. iOS doesn't seem to have actions, though. Not sure how much time I have to fix that though.

shankari commented

Fixed iOS actions - this was basically javascript fixes for the new spec format. Now, time to configure for a real server and for push notifications, and then publish an alpha version for people to start testing at least the tracking.

shankari commented

On android, we get an error saying "String resource ID #0x0" from the push plugin. The push plugin generates this error from

            Log.v(LOG_TAG, "execute: jo=" + jo.toString());
            senderID = getStringResourceByName(GCM_DEFAULT_SENDER_ID);

where

public static final String GCM_DEFAULT_SENDER_ID = "gcm_defaultSenderId";

From the android documentation, there needs to be a resource file with that key, but we don't have it.
https://developers.google.com/android/guides/google-services-plugin

Do we need to create a fork that includes phonegap/phonegap-plugin-push#2873? Why doesn't the existing cordova-plugin-google-services work?

shankari commented

ok, so the google-services.json file is in the correct location platforms/android//app/google-services.json. The format appears to be correct

"project_info": {},
  "client": [],
  "configuration_version": "1"
}

So we don't seem to have this xml file in the prior upgrade as well, but I know that the push plugin worked there. Let's try to re-run it and confirm.

shankari commented

Hm, so I got the same error in the previous build too. But I know that it worked when I tested it on my older laptop. Going back to check that...

shankari commented

ok, so I checked the repo in my old (UC Berkeley laptop) and there is a google-services/debug/values/values.xml. So why don't we have it on this laptop? Is it just the newer version of android studio?

shankari commented

Even on the old laptop, if I check out a fresh repo, the file doesn't show up. That is good in a way, because it indicates that the build process is indeed reproducible. But it is bad because I don't understand why it doesn't work now when it did earlier.

shankari commented

I went back to commit e-mission/e-mission-phone@59c9ae4 before the script fixes, and have the SAME outcome. According to https://developers.google.com/android/guides/google-services-plugin, we can recreate the XML files with the correct values and everything Should Just Work. Using that workaround to publish this initial version using the files from the old laptop.

shankari commented

ionic-team/cordova-plugin-ionic-webview#594 resolved, moved back up to ios@6.1.0

shankari commented

SSL issues again while connecting on iOS, resolved on connecting without VPN
iOS now basically works

shankari commented

android still has issues with the google play services. The google play services xml file is not recognized by the build process, is not merged, and is not included in R.java.

Trying to figure out where to put the xml file so that it will be recognized. Last step!

shankari commented

cp ../config_files/values.xml platforms/android/app/src/main/res/values/google-services.xml seems to work. Will have to debug this later.

shankari commented

Uploaded beta versions to the Play store and to TestFlight. Current status:

  • Switch to the most recent version of all plugins
  • Fix our plugins
    • Edit the data collection plugin to use a wrapper class for the activities (e-mission/e-mission-data-collection#80)
    • Change the hook for all of them as per the instructions above
    • Fix all the e-mission plugins to have the correct name (should do it anyway)
    • Switch to the most recent version of google-play-services (we are using 11, current is 17) (testing pending)
  • Fix local notification (potentially upgrade to the android10 changes)
  • Emulator testing done (except for push notifications, which don't really work in the emulator)
shankari commented

Only real bug reported from beta testing is #555.
The Samsung user ran into issues with background operation.
One iOS user indicated that their tracking worked even though they force killed the app and did not restart when prompted.
Need to see whether this is actually true and whether we can remove the prompt

shankari commented

#555 is caused by an incompatible change in the emailComposer plugin.
Trying to change to the new version exposed the fact that ngCordova is now deprecated.
Removed all instances of ngCordova and switched to Plain Old Cordova Calls.
Retested survey push and email. Everything works. Going to create the new release now.

shankari commented

One sec, before creating the new release, we need to commit and release all the plugins and fix the URLs.