iOS 17.5 background mode, app suspended now, call audio stops.

Question

iOS 17.5 background mode, app suspended now, call audio stops.

Closed this issue 4 months ago · 30 comments

YOU MUST read first!

Please use Community Forum for general technical discussions and questions.

[ *] I have used Google with the error message or bug in association with the library and Cordova words to make sure the issue I'm reporting is only related to iOSRTC.
I have provided steps to reproduce (e.g. using sample app code https://github.com/cordova-rtc/cordova-plugin-iosrtc-sample or updated extra/renderer-and-libwebrtc-tests.js file).
[ *] I have provided versions of third party library name, ios, Xcode and plugin version and adapter.js version if used.

Note: If the checkboxes above are not checked (which you do after the issue is posted), the issue will be closed, removing this checkbox will result in automatic closed issue.

Versions affected

Cordova version (e.g 7.1.0): 12.0.0
Cordova iOS version (e.g 5.1.0): 7.1.0
Plugin version (e.g 6.0.12): 8.0.4
iOS version (e.g 10.2): 17.5
Xcode version (e.g 11.1 - 11A1027): 15.4 (15F31d)
WebRTC-adapter version (e.g. 7.4.0): ?
WebRTC Framework version (e.g. JSSip 3.1.2): 3.10.1

Description

I'm not sure what iOS version this started, but with 17.5 our apps get "paused" when put in the background and at some point stops processing audio shutting down all network connections.

I have VoIP background mode enabled in XCode.

I can resolve the issue by completely removing RTCAudioController from the plugin, at which point there is no speakerphone/earpiece control, but it does keep the app running in the background like it used to.

I wrote my own audio controller for testing purposes, setting the correct category and mode, etc, to VoiceChat. Same results.

I even went as far as to integrate CallKit a bit to see if it would resolve the issue, no luck.

Basically, if I initialize any audio iOS 17.5 will pause the app in the background.

Any help or insight would be greatly appreciated.

Answer 1 · 2024-07-19T04:31:58.000Z

Do you have audio playing all the time while app is in background? if not - it's expected behavior. iOs tend to suspend app sooner or later and you should use push notifications to wake up it for audio output (VoIP or PTT pushs)

Answer 2 · 2024-07-19T09:40:02.000Z

Do you have audio playing all the time while app is in background? if not - it's expected behavior. iOs tend to suspend app sooner or later and you should use push notifications to wake up it for audio output (VoIP or PTT pushs)

Yes, a VoIP call in progress.

Like I mentioned, if I completely remove RTCAudioController from the plugin calls will continue normally when the app is placed in the background just by selecting the "Voice Over IP" background mode in XCode.

Answer 3 · 2024-07-19T09:49:03.000Z

Are you sure have AVAudioSession activated? I had to disable audio session management completely in WebRTC library, but don't remember why exactly...

Answer 4 · 2024-07-19T10:11:54.000Z

Are you sure have AVAudioSession activated? I had to disable audio session management completely in WebRTC library, but don't remember why exactly...

Did you retain speakerphone/earpiece toggle ability with the plugin?

Answer 5 · 2024-07-19T10:18:53.000Z

yes, but I have a little bit complicated configuration - I'm using iosrtc only for realtime video calls (and it's not supported to use it in backgrounded mode), and my own native implementation for audio messages (since I have to use PTT Framework). And only audio messages can be played/recorded in background in my case.

But the main idea is audio session have to be activated and some audio should play, or iOs suspends app. And it's not possible to workaround it. This answer I got some time ago directly from one of Apple support engineers.

Answer 6 · 2024-07-19T14:46:53.000Z

Maybe I should ask this ... has anyone been able to use the plugin with iOS 17.5 with a call staying up when placing the app in the background and be able to toggle earpiece/speaker in the foreground?

Answer 7 · 2024-07-21T02:18:35.000Z

Well, looks like there's no way to use the plugin any longer if you want voip calls to continue in the background. Used to work fine. Not sure when or how this broke, assuming an iOS 17.5 bug. If I remove the plugin it does work, but no way to send audio to earpiece, even with a custom plugin that just changes AVAudioSession's route, does nothing.

With the plugin, calls (and all websockets) shut down in 5-15 seconds. Without the plugin (and no earpiece/speakerphone control) calls and websockets will stay up indefinitely in the background.

Answer 8 · 2024-07-21T02:57:32.000Z

@bkervaski what do you mean by "If I remove the plugin it does work"? Do you mean you are switching to use WebRTC implementation from WebView?

Answer 9 · 2024-07-21T02:59:34.000Z

@bkervaski what do you mean by "If I remove the plugin it does work"? Do you mean you are switching to use WebRTC implementation from WebView?

Yes, but no way to redirect audio to the earpiece using native WebRTC.

Answer 10 · 2024-07-21T03:06:58.000Z

The reason is simple. WebView works in separate process. And obviously that process has it's own reference to AVAudioSession with it's own settings. And your app just doesn't have access to that AVAudioSession and can't do any changes to it.

Answer 11 · 2024-07-21T03:09:05.000Z

So if you need extended AVAudioSession management - you have to use WebRTC implementation inside your app (with iosrtc or your own). It's also related to use PTT Framework and CallKit.

Answer 12 · 2024-07-21T03:10:44.000Z

Do you have any minimal example with reproducible issue? For me it still looks like as not activated Audio Session or missing background mode.

Answer 13 · 2024-07-22T11:42:01.000Z

Hello @bkervaski, did you send 100 USD to me for this issue via Paypal ? If so please 🙏 provide email so I can give it back or I can give it to another good cause or to @RSATom that is really helping you instead.

I don't have the bandwidth to tackle this issue right now and @RSATom is more knowledgeable. That why I can't accept your donation.

Harold.

Answer 14 · 2024-07-22T11:44:38.000Z

@bkervaski you need to provide sample javascript to reproduce.

You can provide in this issue a modified version of this file if you don't know where to start:
https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/extra/renderer-and-libwebrtc-tests.js

Answer 15 · 2024-07-22T17:50:20.000Z

Hello @bkervaski, did you send 100 USD to me for this issue via Paypal ? If so please 🙏 provide email so I can give it back or I can give it to another good cause or to @RSATom that is really helping you instead.

I don't have the bandwidth to tackle this issue right now and @RSATom is more knowledgeable. That why I can't accept your donation.

Harold.

I did. Please pay it forward to @RSATom if you like or I'll donate to him separately.

Answer 16 · 2024-07-22T17:53:45.000Z

The issue appears to be that when using the plugin's WebRTC binaries, iOS will close WKWebView's JsSIP websocket after a few seconds, terminating the call.

Whereas, when using WKWebView's native WebRTC it will keep the websocket open, but no earpiece/speaker control.

Is there anyone pulling this off successfully?

@RSATom Thank you, I'm going to copy and strip the project down to just the plugin and JsSIP and post it for review. It's not a missing background mode.

Answer 17 · 2024-07-23T07:32:32.000Z

@bkervaski I have an idea what is going on in your case.

First of all, as I said before, WebView always works in separate process (read separate application). And if take this into account, when you use iosrtc, you have different parts of logic of your app spread across 2 separate processes (signalling part, and WebRTC Peer Connection part). And iOs manages state of that processes independently. And since WebView's process doesn't play any sound (and as consequence doesn't have activated Audio Session) - iOs have all reasons to to suspend that process on backgrounding. And no, it's not possible to have 2 separate processes with activated Audio Session at the same time (except mixWithOthers AVAudioSession option but it's totally different story).

So the only solution I see here - implement Cordova plugin with signalling part inside (I have this in my app). Then all parts will be in the same process and suspending WebView's process will not affect your logic.

Also, you should keep in mind, if app loses activation of AudioSession in backgrounded state - it's not possible to reactivate it again without foregrounding app by user. The only exception is - VoIP push notifications (+ CallKit) or PTT push notifications.

Answer 18 · 2024-07-23T13:18:12.000Z

I did. Please pay it forward to @RSATom if you like or I'll donate to him separately.

@bkervaski Done, @RSATom you should have received a 100 USD paypal payment on your github public email.
Thank you @RSATom for helping the community as always.

Answer 19 · 2024-07-23T13:20:00.000Z

Oh... It was bad idea. I have some troubles with using PayPal. I'll try to workaround or reject.

Answer 20 · 2024-07-23T13:21:29.000Z

Oh... It was bad idea. I have some troubles with using PayPal. I'll try to workaround or reject.

Ah shit, sorry i had my own issue with paypal account, feel free to reject and i will send elsewhere, you can reach me here hthetiot at sylaps.com if you want to share privatly the paypal account to use.

Answer 21 · 2024-08-02T15:50:35.000Z

Reached out to Apple through the dev program, basically they said "tough luck" on earpiece to speaker toggle for WKWebView.

So, we're having to work around it by suspending calls while the app is in the background and re-establish when the users brings it back.

Not ideal, really takes away from the experience.

Answer 22 · 2024-08-02T15:57:16.000Z

Yep, WKWebView is in separate process -> it has separate Audio Session -> there is no any control from your app over Audio Session in separate process...

Answer 23 · 2024-08-02T16:20:27.000Z

Yep, WKWebView is in separate process -> it has separate Audio Session -> there is no any control from your app over Audio Session in separate process...

Right, and ios rtc gets suspended now, so no more phone calls in the background.

Answer 24 · 2024-08-14T10:44:41.000Z

To make iosrtc works in background you need to add this plugin cordova-plugin-background-mode@0.7.3 and add the folllowing code:

document.addEventListener('deviceready', function () {
    if (cordova.plugins.backgroundMode) {
         cordova.plugins.backgroundMode.enable();
    }
}, false);

This can be tested using Sylaps on iOS https://apps.apple.com/us/app/sylaps/id954016392

Tested on iPhone and iPAd on, 17.5.1, notice that video from the device will go black while application in background, but audio input and ouput keep working.

Answer 25 · 2024-08-14T14:28:54.000Z

To make iosrtc works in background you need to add this plugin cordova-plugin-background-mode@0.7.3 and add the folllowing code:
document.addEventListener('deviceready', function () {
    if (cordova.plugins.backgroundMode) {
         cordova.plugins.backgroundMode.enable();
    }
}, false);
This can be tested using Sylaps on iOS https://apps.apple.com/us/app/sylaps/id954016392

Tested on iPhone and iPAd on, 17.5.1, notice that video from the device will go black while application in background, but audio input and ouput keep working.

Thank you very much for posting, but I've tried this. When I use cordova-plugin-background-mode@0.7.3 event "activate" to call isActive(), it reports "true". Just before this, I get the cordova "pause" event. I would think this even wouldn't fire if background mode is active.

No matter what I do the WKWebView won't keep the JsSIP websocket registration alive. My SIP server (Asterisk) just reports the connection closed a few seconds with a "Connection reset by peer" error.

If a call is active, I'll get the websocket closed error in the javascript console. If not, when I resume the app, the javascript console reports the websocket errors.

The only thing the debugger reports is a two "0x109024d40 - ProcessThrottlerTimedActivity::activityTimedOut" just before and after the app is paused and then a series of "Invalidating grant <invalid NS/CF object> failed" which may or may not be related.

Edit: Oddly, and to reiterate, if I don't use ios rtc, the JsSIP websocket does stay active in the background while on a call. Of course, no speaker/earpiece toggle without the plugin.

Answer 26 · 2024-08-14T17:32:26.000Z

I'm sorry but I disagree with your test result, Websocket stay alive using JSSip and freeswitch as server or P2P in my tests via Sylaps app (I'm the owner of it).

If not using iosrtc work for you then I guess you have your solution, iosrtc is becoming less relevant this days with built-in webrtc from WkWebview anyway.

Answer 27 · 2024-08-14T17:48:01.000Z

Note: iosrtc does not modify websocket behavior.

Answer 28 · 2024-08-14T18:36:00.000Z

I'm not suggestion that ios-rtc changes websocket behavior, I'm suggestion WKWebView will keep the sockets open when there is an active call initiated with native WebRTC.

I agree that the plugin is less relevant, however, I can find no way to toggle earpiece/speaker with WKWebView so I've been holding on to the plugin for this purpose only.

Thank you for your help.

Answer 29 · 2024-08-14T18:43:54.000Z

Websocket stay alive using JSSip and freeswitch as server or P2P in my tests via Sylaps app (I'm the owner of it).

While using ios-rtc plugin or just WKWebView's webrtc?

Answer 30 · 2024-08-20T14:25:51.000Z

While using ios-rtc plugin or just WKWebView's webrtc?

Both we test both mode and on ios <= 14.5 we use iosrtc when missing in webview.