Recordings suddenly stopped playing in salesforce (final player)
Closed this issue · 29 comments
Interim working player and final player that does not work screenshot attache
d
So AWS uses 2 different players in SF, and one works and the other does not. Seems no issue with recording itself.
The change happened when upgrading from 5.14 to 5.16 AWS managed package on SF side
If you look at the release notes for 5.15 here, we changed the way certain aspects of how we communicate with the AWS side.
Enhancement: Created the ExecuteAwsService service for simpler communication between Salesforce and AWS. WARNING: If you are using Contact Lens for audio recording you must replace your existing AwsGenerateAudioRecordingUrl named credential with with the ExecuteAwsService named credential.
@ramaniad We had created the new ExecuteAwsService Named Credentials when we upgraded. We never see the sfExecuteAWSService lambda being invoked. Once the ContactLensTranscripts.json attachment is downloaded to the CCA in salesforce, the player UI switches to the new player UI and no longer plays the audio.
When we hit the Play button, in the JS console we see:
mock media: play() index.ts:665
promise: play -- Promise {<fulfilled>: undefined} index.ts:916
Can you check the Salesforce debug logs to see the aura call that is triggered when you open a CCA page that has Contact Lens enabled? It might have more relevant debug logs. Attaching pictures showing what the log may look like
I enabled aura debug logs as-per https://developer.salesforce.com/docs/atlas.en-us.lightning.meta/lightning/aura_debug_mode.htm?search_text=a.log
is that what you were referring to?
I do see this error now:
If you open up the developer console,
you can look at the aura execution logs, which are shown in the bottom section
:
OK, when I reload the CCA page I don't see any errors in
that log.
Ahh, I do see a 403 invoking ExecuteAwsService in one of those logs
:
Also the AWS credentials configured on the ExecuteAwsService Named Credentials are marked as Last Used: N/A in AWS console - never been used
Ok yea, it looks like something is wrong with the named credential setup then - maybe a mistyped url or the access key/token is incorrect.
I generated new credentials and configured them on the Named Credential to ensure no typos, and still have the issue. The URL looks correct to me.
https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/Callcenter-sandbox-ACSFLambda--sfExecuteAWSService-Qsmv6lvqDcg2/invocations
My lambda is in us-west-2 and it's name is Callcenter-sandbox-ACSFLambda--sfExecuteAWSService-Qsmv6lvqDcg2
Is there any way to debug more specifically what the issue is? The error message in the aura logs is truncated.
15:23:40:107 VARIABLE_ASSIGNMENT [EXTERNAL]|message|"{\"message\":\"Credenti (52 more) ..."
Try adding a trailing / at the end of your endpoint - so https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/Callcenter-sandbox-ACSFLambda--sfExecuteAWSService-Qsmv6lvqDcg2/invocations/ (or if it already exists, remove the trailing /
I searched CloudTrail for the AWS access key associated with the ExecuteAwsService Named Credentials and don't see any logs. And still see no usage of the credentials in IAM console.
Looking at the NAMED_CREDENTIAL_REQUEST log, I see HTTP Header Authorization=Not Set - is that correct?
15:23:40:104 NAMED_CREDENTIAL_REQUEST NamedCallout[Named Credential Id=0XA03000000GnGF, Named Credential Name=ExecuteAwsService, Endpoint=https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/Callcenter-sandbox-ACSFLambda--sfExecuteAWSService-Qsmv6lvqDcg2/invocations, Method=POST, External Credential Type=EXTERNAL, HTTP Header Authorization=Not Set, Content-Type=null, Request Size bytes=219, Retry on 401=True]
Try adding a trailing
/at the end of your endpoint
We removed the trailing / long ago (from the old Named Credentials) because it was a bug in the CTI docs that showed it added. I just added it back and still fails.
Based on the NAMED_CREDENTIAL_REQUEST above, it looks like the authorization header is not even being sent and this is why it 403s, and this is also why CloudTrail shows nothing - there is not even enough information for CloudTrail to identify who is making this request since it contains no authorization info at all.
What do the callout options look like in your named credential?
Yea, based on that NAMED_CREDENTIAL_REQUEST log, I think you might have to reach out to Salesforce, since it seems to be a problem with the NamedCredential not setting the headers properly.
A little more explanation for the trailing / I have one test instance where I need the trailing / and one test instance where I need to remove the trailing /. I was hoping that would be an easy fix.
The last thing I could think of is that the IAM user the security credentials are attached to does not have the proper permissions. The user has this role, correct - invokeSfExecuteAWSServicePolicy
Yes it has that role. If it was an IAM permissions issue I would expect to see the failure in CloudTrail, but I don't see this users credentials ever even being used at all.
This is what my named credential callout looks like -
14:24:27:342 NAMED_CREDENTIAL_REQUEST NamedCallout[Named Credential Id=0XA1k000000CefI, Named Credential Name=ExecuteAwsService, Endpoint=https://lambda.eu-central-1.amazonaws.com/2015-03-31/functions/serverlessrepo-AmazonConnectSa-sfExecuteAWSService-Lp6dW1qruq11/invocations, Method=POST, External Credential Type=EXTERNAL, HTTP Header Authorization=Not Set, Content-Type=null, Request Size bytes=216, Retry on 401=True]
, and it works, so it seems not to be due to the `HTTP Header Authorization
One thing you can do to confirm its not due to the credentials configuration itself is to callout to the lambda function using postman
I invoked via Postman, I didn't POST any data so got a python error in response
{
"errorMessage": "'method'",
"errorType": "KeyError",
"stackTrace": [
" File \"/var/task/sfExecuteAWSService.py\", line 40, in lambda_handler\n method = event[\"method\"]\n"
]
}and can see the error in the lambda CloudWatch logs - so the credentials seem OK.
I also confirmed the trailing / on the URL again - by checking it is in the logs after adding it:
18:57:01:397 NAMED_CREDENTIAL_REQUEST NamedCallout[Named Credential Id=0XA03000000GnGF, Named Credential Name=ExecuteAwsService, Endpoint=https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/Callcenter-sandbox-ACSFLambda--sfExecuteAWSService-Qsmv6lvqDcg2/invocations/, Method=POST, External Credential Type=EXTERNAL, HTTP Header Authorization=Not Set, Content-Type=null, Request Size bytes=219, Retry on 401=True]
18:57:01:398 NAMED_CREDENTIAL_RESPONSE NamedCallout[Named Credential Id=0XA03000000GnGF, Named Credential Name=ExecuteAwsService, Status Code=403, Content-Type=application/json, Response Size bytes=72, Overall Callout Time ms=362, Connect Time ms=280
Is there anyway to troubleshoot this on the AWS side? There is nothing in CloudTrail, but AWS is returning the 403 I assume, so there must be some AWS side log indicating why it is doing that.
Or is there a way to see the full response in the Aura logs instead of the truncated version? Or is there a list of all AWS 403 response messages and we can see which ones have a message beginning with the string Credenti and try to figure out which message it is returning?
15:23:40:107 VARIABLE_ASSIGNMENT [EXTERNAL]|message|"{\"message\":\"Credenti (52 more) ..."
I think if you right click the log and select Open Raw Log it will show the full log. As for the AWS side, if it is not showing up in the lambda invocations, in cloud trail, or in IAM, I am unsure if there is anything else.
It's still truncated in the raw logs:
10:31:37.0 (370706130)|VARIABLE_ASSIGNMENT|[EXTERNAL]|message|"{\"message\":\"Credenti (52 more) ..."
Do you have a list of error messages AWS returns with 403? Do any of them begin with Credenti and 52 more characters?
I found some example error messages here that begin with Credenti
https://docs.aws.amazon.com/general/latest/gr/signature-v4-troubleshooting.html
Credential should be scoped to correct service: 'iam'.
Credential should be scoped to a valid Region, not 'us-west-1'.
The second one could be relevant here - is the region field in the named credential the same as what is in the endpoint url you typed us-west-2 and is the lambda actually in us-west-2 (which I assume it is). And is the service typed as lambda (maybe there is a typo)
oh man, the SF admin spelled it lamda! Changing to lambda fixed it - thanks!
Np! I'm glad it was just a simple fix hahaha.