openid/OpenYOLO-Android

Document interactions with Android O's Autofill

dxslly opened this issue · 15 comments

dxslly commented

Android O introduced the Autofill framework. With it forms may be automatically filled in via an inline UX as seen in modern web browsers and an save dialog may be shown. This can lead to a poor experience in apps that use OpenYOLO if unplanned for. The two interactions that stand out to me are:

  1. Using both a Hint flow and inline Autofill UX. (e.g. after using the OpenYOLO hint flow, a user is brought to an account creation screen to provide additional information, tapping on a form bring up the inline Autofill UX).
  2. Seeing two save dialogs. One from the OpenYOLO provider and an implicit save dialog from the Autofill framework.

We should decide on a standard recommendation for both clients and providers, document these recommendations and implement them if possible.

Some of my thoughts on this are:

  1. Clients using the hint flow should disable inline Autofill via setImportantForAutofill(IMPORTANT_FOR_AUTOFILL_NO). This still allows power users to override via a long press.
  2. The OpenYOLO save flow should be preferred over the Autofill save dialog. There are two solutions to handling this: a) the client can attempt to not trigger the implicit Autofill save dialog or b) the Autofill provider can choose to not want to save any data. For a) if the user never triggers the inline Autofill UX via 1) the save dialog should not be shown. For b) the Autofill provider's logic would be something like "if app intends to use the OpenYOLO save flow, do not show the Autofill save dialog". At the moment there is no way to detect if this, but we could consider adding a proxy for this such as "I am using OpenYOLO" (e.g. via some metadata in the manifest). An alternative solution is to conditionally prefer the Autofill save dialog. The logic to not show the OpenYOLO save dialog could be "if on O and above device, saving password based credentials and the OpenYOLO provider is the active Autofill provider".

Overall I think telling clients "Mark fields as not important for Autofill" on their account creation/sign in is an easy message that should cover both issues. We could even add a method to the client that would do this for clients (e.g. CredentialClient.disableAutofill(Activity activity). In addition we could also add some helpers to enable OpenYOLO provider that are also Autofill providers handle 2.b.

What do you guys think?

tikurahul commented

This works for now. But, i think the longer term solution is to have an implementation of an https://developer.android.com/reference/android/service/autofill/AutofillService.html (which uses the OpenYOLO client API to fill the view). I think that makes things consistent from the perspective of a user. Especially important in the context of filling things like addresses and credit card info (ancillary account info).

StanKocken commented

I'm not sure telling the client to put IMPORTANT_FOR_AUTOFILL_NO is the right direction.
It seems hacky and if a provider is compatible with Autofill but not with OpenYOLO, it will be a disadvantage for them. The long press is pretty hidden, not sure a lot of user know about it.

Having a way for any provider if the application is compatible with OpenYOLO seems a good way: then he can avoid doing the Autofill or accept the saving. A meta data within the manifest sounds a good option for me. This can be automatically added by our API (like a Permission or Activity) so it's transparent for the developer. Including the OpenYOLO version number can help for future compatibility as well.

    <meta-data
        android:name="net.openid.openyolo-api"
        android:value="0.2.1" />
nerdyverde commented

We want to make this as easy as possible for both client apps and users. Encouraging developers to use IMPORTANT_FOR_AUTOFILL_NO would seem to run counter to this. One of two undesirable scenarios plays out:

  1. The developer applies IMPORTANT_FOR_AUTOFILL_NO via the layout file and the user is expected to long press in the case of Autofill providers that don't support OpenYOLO. As Stan mentioned, this puts those Autofill providers at a disadvantage and requires esoteric knowledge on the part of the user.
  2. The developer applies IMPORTANT_FOR_AUTOFILL_NO programmatically and sets the value according to whether or not there is an OpenYOLO provider available. This seems better than scenario 1 to me, but still requires more work on the part of the developer.

With that in mind, I also prefer the approach of including meta-data in the manifest. We can provide a convenience method that allows providers to consume the meta-data and adjust their behaviour accordingly. Providers could then use the presence of this meta-data to short-circuit their handling of onFillRequest() in their Autofill implementation (you'll have to pardon the unimaginative method name):

public void onFillRequest(FillRequest fillRequest, CancellationSignal cancellationSignal, FillCallback fillCallback) {
    String packageName = assistStructure.getActivityComponent().getPackageName();

    if (isOpenYoloSupported(packageName)) {
        fillCallback.onSuccess(null);
    } else {
        // proceed with normal Autofill processing
    }
}

This would eliminate issues with both filling and saving, since onSaveRequest() won't fire in the absence of SaveInfo returned with the FillResponse for onFillRequest(). And most importantly in my mind, it puts the onus of correctly handling OpenYOLO vs Autofill squarely in the hands of providers that implement both.

iainmcgin commented

I think this is a frustratingly complex issue. In the ideal case, OpenYOLO and form-fill would never conflict because OpenYOLO retrieve and hint calls would happen _before_any actual form would be displayed, so there's no conflict on the input side of the flow. However, this typically requires deeper reorganization of an app's authentication steps, which most developers are unlikely to do.

On the save side of the flow, double-save may even be desirable - consider the case where an OpenYOLO hint is used to generate a credential, but the user also enters an address. We would either want OpenYOLO to save the credential and form-fill to save the address, or form-fill to save everything. In the simpler case where the only thing to be saved is the credential itself, we would want OpenYOLO exclusively to do this, for guaranteed precision rather than relying on heuristics.

A variety of context-dependent solutions may be required:

  1. Provide variants of hint() and retrieve() which automatically disable form-fill for the current activity, using David's suggested method. This does seem tricky, particularly if layouts and fragments are late-inflated.

  2. Provide a method which disables OpenYOLO save() for the current activity. This will likely need some reorganization of our API.

  3. Provide SPI utilities to assist in detecting and tracking the known state of the client, to disable OpenYOLO or auto-fill from their end.

@tikurahul I don't think implementing an auto-fill provider backed by OpenYOLO is feasible - I don't think the mechanics of auto-fill really allow for OpenYOLO to be transparently nested in this way. Some generic implementation of the auto-fill heuristics would also need to be provided by a support app or the platform as well.

sganov commented

Both blending approaches have caveats:

  • If the app optimizes for YOLO by hiding its UI from autofill this would lead to suboptimal UX if there is no YOLO provider installed as it requires a long press. Even if the app does this dynamically based on the presence of YOLO providers there is not guarantee they would have data but the fill service may resulting in the user not getting a suggestion until force filling which is not very discoverable.

  • If the service optimizes for YOLO by checking an app manifest entry which is also problematic as there may be no Open YOLO providers on the device and the user would get no suggestions since the autofill service suppressed itself while it may have data. Even if the service delegates only when there are YOLO providers they may not have data which the service has missing an opportunity to help the user.

One can hide the UI from autofill, query YOLO and if no results, force request autofill. This solves the problem with getting the data if any of the two systems has it but the saving may need two dialogs as the service may want to save the address while delegate the credentials to YOLO.

I think the fundamental problem is that there are two independent credential store frameworks working in parallel with their separate APIs, UI, settings, etc. This requires either the app developer or the autofill provider developer to add special handling to make them blend in and not collide - this is hard and has edge cases. Further, the UI may be made similar but would never converge (including settings). The user would be getting different UX based on whether the app uses YOLO or autofill.

An approach to make Open YOLO and autofilll play nicely together could be adding to AutofillService extendable APIs for querying for credentials on top of which Open YOLO could be implemented. These APIs could be generic enough to allow evolution of the Open YOLO protocol or rolling a different protocol altogether.

This would solve the colliding UI problem, unify settings, allow platform level optimizations for the Open YOLO flow, have a single APIs for developers to work against. This makes the assumption that autofill providers and YOLO providers are the same set of apps.

Thoughs?

iainmcgin commented

I'm still unconvinced by the argument to require platform level support for OpenYOLO. We were able to build everything we needed based on what is provided by the platform in API 15, so we can support 99.4% of Android devices. Relying on framework changes would now mean waiting for Android P, with no actual relevance to app developers until 2021 based on Android's "three years to 50% market share" upgrade rate. I'm not willing to give up on broad device compatibility, so we would always end up with a bifurcated way of doing things until then.

The only things I really ever wanted from the platform were:

  1. Access to the identity of the current auto-fill provider, and to treat this setting as a general "credential provider" preference rather than just for auto-fill. It has been stated that the platform won't provide this for privacy reasons, which baffles me, as an app can already query all other installed apps on the device and infer (with high enough precision) what the provider is, which is currently what we do for OpenYOLO.

  2. An easy way to programmatically enable / disable form-fill. It has been stated that the platform won't provide this as it allows misguided apps like banks to prevent the usage of a password manager. I think the number apps which would actually disable autofill in a user hostile way are incredibly small, and they can already do this by evading autofill heuristics (e.g. use custom views). In order words, if you want to stop autofill, you already can, so the barrier to disabling it when it actually makes sense isn't useful.

Anyway, I'd be interested to hear what @nerdyverde and @StanKocken have to say. This is a community effort, I'm not actually a dictator 🤣

sganov commented
StanKocken commented

I agree that not sure if we can have any support on the platform level.

We can do nothing (so have the double save, but as Iain mentioned it may not be such a big issue) or have a basic helping support for provider (which I would prefer).

Now, if the OpenYolo provider can determine if the client app is openyolo compatible (with manifest meta data).

Let's say you have few password managers, all installed on the user device (for some reason…):

  • A which implements Autofill but not OpenYolo
  • B which implements OpenYolo but not Autofill
  • C which implements both OpenYolo and Autofill.

Some scenarios:

  • User select A as autofill system: the autofill save popup will show up for A, OpenYolo for B
  • User cannot select B as autofill system
  • User select C as autofill system: the autofill save popup won't show up (because C detected that he's the autofill system, so disable itself autofill for this app) but OpenYolo with B and C

So only the first scenario is an issue.
Today the 3 openYolo providers can prevent this to happen: SmartLock, 1Password and Dashlane have (or can have) both OpenYolo and Autofill.
So the question is more for new OpenYOLO provider?

In most cases the users will have either only Smartlock or Smartlock + a 3rd party Password Manager.

nerdyverde commented

While OpenYOLO and Autofill may eventually converge, I think that requires much longer term planning than the lifetime of this issue. Although the goals of each project are similar, there are enough differences currently that I think it makes sense to continue development in parallel.

As we've discussed above, this issue can essentially be divided into the following two scenarios:

  1. The Autofill provider is one of the OpenYOLO providers on device
  2. The Autofill provider is not not one of the OpenYOLO providers on device

I think that the first scenario is much more likely to occur than the second. Anecdotal evidence from our customer support team suggests that we don't often encounter situations where people are using multiple third-party password managers simultaneously. While we do occasionally hear from customers who use different password managers for home and work, these are the exception rather than the rule.

In addition to the first scenario being more likely to occur, it concerns me more because it seems particularly unforgivable that the same provider would prompt the user twice. The user reasonably expects that the provider should know that it has already prompted them and would find it ridiculous that they are being prompted again. The suggestion of adding meta-data to the manifest would address this problem by allowing the provider to know that its OpenYOLO implementation will be invoked and therefore choose to suppress its Autofill implementation.

We may want to use a permission rather than meta-data though. We could ensure the presence of this permission in the manifest by having the OpenYOLO API require it. This would ensure that providers can query whether OpenYOLO is supported in a calling app.

I'm convinced that the second scenario above is much less likely to occur, but I also concede that it's going to be the trickier of the two to address. No matter how I slice it, I can't get away from this requiring work on the part of the client app. The client will need to suppress Autofill, check if an OpenYOLO request can be handled, and if not, force a fill request with Autofill. I think that we're just going to need to document this well.

sganov commented
iainmcgin commented

So, there are some shorter term issues to resolve, like what we can do in the library to suppress auto-fill save prompts if OpenYOLO is used in the context of a given activity / fragment. @sganov, what do you suggest in this case? Ideally it would just be a new utility method we add to OpenYOLO that suppresses save for the lifetime of the current activity, which the developer could call any time after onCreate(). In talking to David about this, we couldn't find a reliable way to do this - it would be worth directing this issue towards a solution to that problem, and we can worry about the longer term implications of making OpenYOLO a support library for future platform support separately.

sganov commented
iainmcgin commented

OK, I'm covering this at DroidCon London (briefly) and I'll give your recommendation, which is typically in line with the UX guidance we've been giving for Smart Lock for Passwords. So, I'll say:

  • Skip the login screen if you using the OpenYOLO retrieve flow.
  • If retrieve fails, try getting an account hint and use this to sign in or jump straight to account creation.
  • If this fails, show the login form, but mark the ID and password fields as IMPORTANT_FOR_AUTOFILL_NO. Given that in 99.9% of cases the autofill provider and OpenYOLO provider are the same, it wouldn't have given you any data anyway, and this also avoids double save.

If this makes sense and resonates, I'll add this to our official docs.

dxslly commented

LGTM, except I would change the last bullet to mark all fields as not important and only if using OpenYOLO to save the credential. Something like:

  • If this fails, show the login form. If you are using OpenYOLO to save the credential mark root view with IMPORTANT_FOR_AUTOFILL_NO_EXCLUDE_DESCENDANTS. This ensures any login information is saved by either OpenYOLO provider or the active Autofill service.

Ideally it would just be a new utility method we add to OpenYOLO that suppresses save for the lifetime of the current activity, which the developer could call any time after onCreate(). In talking to David about this, we couldn't find a reliable way to do this.

AFAICT for clients to suppress only the save dialog they would need a way to configure Autofill sessions to never show a save dialog or a 'session-is-about-to-end' lifecycle hook where AM.cancel() could be called.

timcappalli commented

This repo is being archived. Closing issue.