CallMessageDelegate does not receive call messages

[amritashan]

Description

I have implemented CallMessageDelegate in my app to receive the call messages, but the handler function for receiving messages never gets triggered.

On the other hand, the Javascript counterpart of the application is able to successfully receive messages in the messageReceived event handler when using the Voice JS SDK.

I am using v6.11.1 Swift package

My delegate looks like as shown below:

import Foundation
import TwilioVoice

extension AppDelegate: CallMessageDelegate {
  func messageReceivedForCallSid(callSid: String, callMessage: CallMessage) {
    NSLog("message received: \(callMessage.content)")
  }
}

There was a little confusion about the function name because it looked like the function has been renamed several times throughout the various versions.

So, I have implemented the following variations as well, just to be sure:

1. func didReceiveMessage(call: Call, callMessage: CallMessage)
2. func callDidReceiveMessage(call: Call, message callMessage: CallMessage)
3. func messageReceived(callSid: String, message: CallMessage)

But none of them seem to get called.

Steps to Reproduce

(Same as shown above)

Code

(Same as shown above)

Expected Behavior

The function didReceiveMessage() or callDidReceiveMessage() or messageReceived() should get called when an call message is received.

Actual Behavior

Nothing happens.

Reproduces How Often

100% of time

Logs

N/A

Versions

TwilioVoice 6.11.1

Voice iOS SDK

N/A

Xcode

Version 15.2 (15C500b)

iOS Version

17.2

iOS Device

iPhone 15

[bobiechen-twilio]

Hi @amritashan

Thanks for reaching out. Quick question: how does the app provide the TVOCallMessageDelegate instance to the SDK? For calls (outgoing or incoming) the delegate is specified via the TVOConnectOptions/TVOAcceptOptions. Example:

    TVOConnectOptions *connectOptions = [TVOConnectOptions optionsWithAccessToken:accessToken
                                                                            block:^(TVOConnectOptionsBuilder *builder) {
        builder.params = twiMLParams;
        builder.uuid = uuid;
        builder.callMessageDelegate = self;
    }];
    self.call = [TwilioVoiceSDK connectWithOptions:connectOptions delegate:self];

For call invites, provide the delegate in the TwilioVoiceSDK.handleNotification() method:

[TwilioVoiceSDK handleNotification:payload.dictionaryPayload delegate:self delegateQueue:nil callMessageDelegate:self]

[amritashan]

HI @bobiechen-twilio ,

Thanks for replying.

I checked my code with your suggestion. I am indeed missing the delegate references.

Here are the relevant places where I have now added the delegate reference. Does this look ok?

Also, can you please let me know any document reference where I can find these little details?

1. CXProviderDelegate

    func performVoiceCall(uuid: UUID, client: String?, completionHandler: @escaping (Bool) -> Void) {
        let connectOptions = ConnectOptions(accessToken: accessToken) { builder in
            builder.params = [ "To": self.phoneNumber ]
            builder.uuid = uuid
            builder.callMessageDelegate = self
        }
        
        let call = TwilioVoiceSDK.connect(options: connectOptions, delegate: self)
        NSLog("Call placed to " + phoneNumber);
        activeCall = call
        activeCalls[call.uuid!.uuidString] = call
        callKitCompletionCallback = completionHandler
    }

2. PushEventDelegate

    func incomingPushReceived(payload: PKPushPayload) {
        // The Voice SDK will use main queue to invoke `cancelledCallInviteReceived:error:` when delegate queue is not passed
        TwilioVoiceSDK.handleNotification(payload: payload.dictionaryPayload, delegate: self, delegateQueue: nil, callMessageDelegate: self)
    }

    func incomingPushReceived(payload: PKPushPayload, completion: @escaping () -> Void) {
        // The Voice SDK will use main queue to invoke `cancelledCallInviteReceived:error:` when delegate queue is not passed
        TwilioVoiceSDK.handleNotification(payload: payload.dictionaryPayload, delegate: self, delegateQueue: nil, callMessageDelegate: self)

        if let version = Float(UIDevice.current.systemVersion), version < 13.0 {
            // Save for later when the notification is properly handled.
            incomingPushCompletionCallback = completion
        }
    }

[bobiechen-twilio]

Hi @amritashan

The implementation looks good to me and hopefully you are getting the inbound call-messages to the delegate methods. It's not shown everywhere but we do mention where to specify the TVOCallMessageDelegate instance in the TVOCall.sendMessage() and TVOCallInvite.sendMessage() API documentation.

By the way, props to you for setting up the SDK clients and the user-defined message subscription/callback endpoints in order to get the messages flowing end to end!

[amritashan]

Hi @bobiechen-twilio,

Thank you again for looking into it. I got really close to finding a solution.

Now, I get the call messages for outbound calls, but not for incoming calls.

I tested out the changes and noticed a few things:

1. Call Messages are working only for outbound call. This indicates that the following bit is working right:

        let connectOptions = ConnectOptions(accessToken: accessToken) { builder in
            builder.params = [ "To": self.phoneNumber ]
            builder.uuid = uuid
            builder.callMessageDelegate = self
        }
        
        let call = TwilioVoiceSDK.connect(options: connectOptions, delegate: self)

2. The CallMessageDelegate implementation called the function callDidReceiveMessage(call, callMessage) and not messageReceivedForCallSid() which I had implemented initially.

This, however, seems odd, because the protocol declaration clearly shows the function as:

- (void)messageReceivedForCallSid:(nonnull NSString *)callSid message:(nonnull TVOCallMessage *)callMessage
NS_SWIFT_NAME(messageReceived(callSid:message:));

Considering that the Swift name is messageReceived, it should have called that, however, it calls the function which has been marked deprecated:

- (void)call:(nonnull TVOCall *)call didReceiveMessage:(nonnull TVOCallMessage *)callMessage
NS_SWIFT_NAME(callDidReceiveMessage(call:message:))
__attribute__( (deprecated("Will not be supported in TwilioVoice v6.11+", "didReceiveMessage:message:")) );

3. Call messages are not working on inbound calls. Which means that the call to TwilioVoiceSDK.handleNotification() is probably not effective or is not called at the right time.

Here's how my implementation looks like (which is exactly how the quick start app is). Do you find anything that I might have missed?

class AppDelegate {
  ...
  var pushKitEventDelegate: PushKitEventDelegate?
  ...
  ...
  func initialise() {
    self.pushKitEventDelegate = self
  }
}

extension AppDelegate: PKPushRegistryDelegate {
  ...
  ...

    /**
     * This delegate method is available on iOS 11 and above. Call the completion handler once the
     * notification payload is passed to the `TwilioVoiceSDK.handleNotification()` method.
     */
    func pushRegistry(_ registry: PKPushRegistry, didReceiveIncomingPushWith payload: PKPushPayload, for type: PKPushType, completion: @escaping () -> Void) {
        NSLog("pushRegistry:didReceiveIncomingPushWithPayload:forType:completion:")
        NSLog("notification payload %@", payload.dictionaryPayload.description)

        if let delegate = self.pushKitEventDelegate {
            delegate.incomingPushReceived(payload: payload, completion: completion)
        }
        
        if let version = Float(UIDevice.current.systemVersion), version >= 13.0 {
            /**
             * The Voice SDK processes the call notification and returns the call invite synchronously. Report the incoming call to
             * CallKit and fulfill the completion before exiting this callback method.
             */
            completion()
        }
    }
}

extension AppDelegate: PushKitEventDelegate {
  ...
  ...
      func incomingPushReceived(payload: PKPushPayload, completion: @escaping () -> Void) {
        // The Voice SDK will use main queue to invoke `cancelledCallInviteReceived:error:` when delegate queue is not passed
        TwilioVoiceSDK.handleNotification(
            payload: payload.dictionaryPayload, 
            delegate: self, 
            delegateQueue: nil, 
            callMessageDelegate: self
        )

        if let version = Float(UIDevice.current.systemVersion), version < 13.0 {
            // Save for later when the notification is properly handled.
            incomingPushCompletionCallback = completion
        }
    }
}

Can you please look into this and advise?

Thank you.

[amritashan]

Hi @bobiechen-twilio, any thoughts on this one?

Does this look like a bug?

[bobiechen-twilio]

Hi @amritashan

When you said Call messages are not working on inbound calls. did you mean the TVOCallInvite object or the TVOCall object after you accepted the call invite? also are you receiving call-messages in this case (ie. getting the message-received callback)?

Also we will investigate the Swift callback

[amritashan]

@bobiechen-twilio ,

Sorry, I couldn't understand your question.

My understanding so far has been:

1. Outbound calls:

CallMessageDelegate is referred to in the ConnectOptions (builder.callMessageDelegate = self).

So, when you do TwilioVoiceSDK.connect(...), I will get the Call Messages in my CallMessageDelegate implementation class.

You are probably refering to this as TVOCall object.

2. Incoming Calls:

You need to refer to the CallMessageDelegate with the PKPushPayload

        TwilioVoiceSDK.handleNotification(
            payload: payload.dictionaryPayload, 
            delegate: self, 
            delegateQueue: nil, 
            callMessageDelegate: self
        )

Since this happens during the incoming Call invitation flow, i assume you refer to this as the TVOCallInvite object.

If my assumptions above are right, then I would say that for point 2 above (incoming calls) the CallMessageDelegate.callDidReceiveMessage() does not receive any call messages.

Is there any other additional configuration that needs to be done to make it work?

Please refer to the code outlines in my last post.

[amritashan]

@bobiechen-twilio , could you look into this?

[jamespick]

Hi not sure if you always get to see the outcomes of your sdk's so I thought I'd add a quick video walkthrough for you. @amritashan above part of our team. On the frontend we decided on a common framework of angular across web, desktop, ios and android then using Capacitor and Electron to build the respective apps. With Capacitor we built plugins to connect to ios and android and subsequently used the native SDKs to obtain a comprehensive and native feel for the telephony experience. In call push notifications mentioned in this issue above was specifically to support recording notifications but we have additional plans to push contact record information and perhaps even ai prompts to the phone users as they talk to customers. All backend activity (call routing, IVR, extension setup, queue managment is managed via serverless functions serviced by AWS Lambda & DynamoDB. SMS and WhatsApp messaging is currently only provided in our web based CRM app as part of a unified mailbox but IOS/Android apps are on the way. The following demo video highlights the IOS app alongside Desktop and iFrame Web: https://app.screencastify.com/v3/watch/GAdrRpcW6CfRVqkqmvV9

[bobiechen-twilio]

Hi @amritashan

Quick question - for the incoming call did you also make used-defined-message subscription using the Call SID? Please note that the Call SIDs of the parent and child call legs are different if the client incoming call is triggered by a dial. That being said, do you have the Call SID where you ran into problem getting the inbound call-message callbacks?

[jamespick]

Hi @bobiechen-twilio. @amritashan is out at the moment so I'll respond with some details. Our issue lies with in-call notifications send during a live call from server to client. For calls that a client makes (parent inbound) we are succesfully receiving the notifications on the ios device. For calls that a client receives (child call outbound) we do not receive the notification on ios or andriod device but can see it if answered by web client. In a scenario that a client call is made to another client call our server will send the notifications to both child and parent call sids. We can clearly see that the messages are arriving when we answer the call on the angular web client. It looks as follows:

{
    "content": {
        "message_body": {
            "status": "IN-PROGRESS"
        },
        "message_type": "recording"
    },
    "contentType": "application/json",
    "messageType": "user-defined-message",
    "voiceEventSid": "KX86de300aecc7452a037951edc6b87aac"
}

As requestsed we have just made a sample call from andriod client to ios client. The call sids are as follows. Both calls should have been issued (from the server) in call user-defined-messages. This is as a result of our client user toggling call recording start/stop commands.

Parent Incoming Call from Andriod Client: CA6382416c9c4e6f6ec6bec204739c42b1
Child Outgoing Call answered by IOS Client: CAa174328064fda2d5320b98dd018aa1d8

[bobiechen-twilio]

Thanks @jamespick

Couple of questions:

• Is the Android client (the caller) getting the call-message-received event?
• How is the user-defined-message sent, e.g. sent by using the user-defined-message API or the sendMessage() SDK method, when the iOS client (callee) is not getting the event?

[jamespick]

@bobiechen-twilio

Yes the android client (and infact also ios client when it is the caller) is successfully getting the call message event when acting as caller. We are using the following function to pickup the message callDidReceiveMessage(call, callMessage)

When the client is receiving a call (be it ios or android) we are unable to pickup the message. We know the message is being sent as we can see the message if the call is answered by our javascript web client.

In all cases messages are always sent from server to the clients using following method: https://www.twilio.com/docs/voice/api/userdefinedmessage-resource#create-a-userdefinedmessage-resource

We are not sending messages from our clients via in call messaging as we are using our own authenticated api calls to send messages.

[bobiechen-twilio]

Got it. Thanks.
One more thing I would your help to clarify - (on the call receiver side) are you trying to receive the call-message events while the call is still ringing, i.e. on the Call Invite object, or on the Call object when the call invite has been answered?

From the code snippet provided, providing the call-message delegate or event listener in the handleNotification() (or handleMessage()) will only allow the Call Invite object to receive the events. To enable the accepted Call object to receive the call-message events, the application needs to specify the delegate/listener in the AcceptOptions object like in the ConnectOptions when initiating a call from the mobile client.

[amritashan]

Hi @bobiechen-twilio,

My intention is to receive the messages during the call, and not while the call is ringing.

And you are right, the code could be listening for the messages during the call invite only. I was under the impression that assigning a the callMessageDelegate will allow the call to listen for the messages as well.

In that case, will the following code work, with the callMessageDelegate set as shown?

let acceptOptions = AcceptOptions(callInvite: invite.value) { builder in
  builder.uuid = invite.value.uuid
  builder.callMessageDelegate = self
}

let call = invite.value.accept(options: acceptOptions, delegate: self)

Can you point me to any relevant documentation for this?

Thanks.

[bobiechen-twilio]

By the way I checked the Swift callback and I was able to receive the callback:

    func messageReceived(callSid: String, message callMessage: CallMessage) {
        NSLog("message received \(callSid): \(callMessage.content)");
    }

[jamespick]

@bobiechen-twilio Many thanks for your help we are now processing in call notifications within inbound & outbound calls 😃. This solution also helped us resolve the similar issue we were having on the android client.

[bobiechen-twilio]

@jamespick @amritashan Glad to see things are working end to end now. Your feedback is much appreciated and will definitely help us improve the feature and documentation. Please reach out any time you need help with the SDK in the future.