Investigate intermittent issue with processing payments via webhook where UPE redirect never occurs

[james-allan]

Describe the bug
We've seen a couple of reports where merchants are experiencing orders left in a pending-payment status.

When you dive into the logs you see something like this:

This log entry indicates that the processing of the webhook was skipped because we expect that the customer should land on a UPE redirect URL and we should process the payment via the process_upe_redirect_payment() flow.

Something in these cases is either:

1. Preventing the UPE URL redirect from occurring at all.
2. Preventing the UPE URL redirect from validating. Noting that there are a lot of conditions that need to pass for the UPE redirect to actually lead to it processing the payment.

To Reproduce

I haven't been able to reproduce. By merchant reports the issue seems intermittent (not impacting all payments).

Expected behavior

The reason we skip these webhooks is because we needed to avoid double processing in cases where the webhook is processed at the same time that the redirect is (see #3078).

We will need to find a more robust way of handling this situation. Given we have a webhook capable of processing the payment, it isn't ideal that it is ignored and instead we wait for a potentially more flakey process to process it.

Some options include:

1. Move to prio webhook processing rather than the URL flow.
2. Move to a process of polling for status updates for orders that are unpaid, have an intent, and have the 'waiting-for-url-redirect'.
3. Move the webhook processing to an async task which will have less of a chance of duplicate/overlapping processing.
4. Introduce a more robust locking system where the order is locked before processing to avoid duplicate processing.

Additional context

Support thread: https://wordpress.org/support/topic/payment-pending-but-payment-succeeded/

The payment methods mentioned on that support thread include P24 and iDEAL which are both payment methods that redirect the customer offsite to complete the payment via their bank before being redirected back to the store.

[OmarFPG]

8287133-zen

[shameemreza]

Another report here: 8322353-zen

[yukikatayama]

Looks like we're seeing more reports on this bug this week. Do we have an update on the progress of the status @james-allan? 🙇‍♀️

[csnlima1]

8183081-zd-a8c

[stevenolsthoorn]

I have the same issue