Central Admin People Picker not working correctly

Question

Central Admin People Picker not working correctly

Closed this issue a year ago · 31 comments

I'm currently testing AzureCP on one of our SP2016 farm. I'm experiencing a strange behavior regarding the people picker from Central Administration. Below I describe all the steps I done to reproduce the scenario.

To start as clean I possible, I use the Reset AzureCP configuration button to start at initial phase.
Run a PowerShell script to configure AzureCP and add a new tenant.

Add-Type -AssemblyName "AzureCP, Version=1.0.0.0, Culture=neutral, PublicKeyToken=65dc6b5903b51636"
$config = [azurecp.AzureCPConfig]::GetConfiguration("AzureCPConfig")

$config.EnableAugmentation = $true
$config.FilterSecurityEnabledGroupsOnly = $true

$newAADTenant = New-Object azurecp.AzureTenant
$newAADTenant.Name = $j.AzureCP.AADTenantName
$newAADTenant.ApplicationId = $j.AzureCP.ApplicationId
$ExportableKeyStorageFlag = [System.Security.Cryptography.X509Certificates.X509KeyStorageFlags]::Exportable
$newAADTenant.ClientCertificatePrivateKey = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2($j.AzureCPCertificate.PFXPath, $j.AzureCPCertificate.PFXPassword,$ExportableKeyStorageFlag)
$newAADTenant.ExcludeGuests = $true
$config.AzureTenants.Add($newAADTenant)

$config.Update()

As you can see, I use an App Registration with Client certificate for authentication.

If I open the Global configuration page of AzureCP, the new tenant is added correctly. Note here: don't click on OK/Cancel button here, just leave the page by clicking on another Central Admin link.
Go to Application Management, and click on Change site collection administrators. Click on the Browse icon and type a user to find. Here I have systematically No results message.
From ULS log, I have the following:

[AzureCP] Unexpected error occurred while setting the certificate for tenant 'mapetiteentreprise.onmicrosoft.com'. Is the private key of the certificate exportable?: System.Security.Cryptography.CryptographicException: Keyset does not exist
[AzureCP] Unexpected error occurred in SearchOrValidate: System.Security.Cryptography.CryptographicException: Keyset does not exist

Now, go back to Global configuration page of AzureCP. Once open, just click on OK button.
Go back again to Change site collection administrators page, click on Browse, and type a user: it's works! (sometimes still not).
I proceed iisreset, reopen the Change site collection administrators page, and try to find again a user: it doesn't work anymore. No results at all, with the same errors in the ULS.

Directly on SharePoint site collection, no issue at all, the People Picker works well in all case.

To make it work again from Central Administration, the only way I've found for now is to remove the AAD tenant from the Global configuration page and recreate it. Not very fun...

I don't know if the ULS errors messages are relevant regarding the certificate private key, or it's a tree that hides the forest.
In all case, the certificate in question is on the Personal store, with permissions for the service account that manage the CA IIS Application.
Could you please check on your side if you can experience the same issue?
Or eventually, could you please advice?

Thanks!

Answer 1 · 2022-12-23T09:51:21.000Z

@julmsy thank you for reporting this.
I'll test your script to see if I can repro and keep you posted

Answer 2 · 2023-01-27T07:36:07.000Z

This issue is caused by the account running the application pool of central admin not being able to access the certificate's private key. This can be fixed by adding the account to the certificate's private key reader group.

Answer 3 · 2023-01-30T07:36:39.000Z

We are still facing this issue - even with the permissions on the private key. This kind of helped in the first run. It looks like a redeployment of the solutions breaks the access to the configuration.

Answer 4 · 2023-01-30T07:41:07.000Z

@Yvand Within the solution there are two hardcoded passwords that are used to encrypt the certificate within the solutions scope. Would there be any other option available (like storing the certificate in the certificate store and enabling access for the web application account)? At the moment, it feels like a potential security risk to gain access to AAD.

Answer 5 · 2023-01-30T09:57:53.000Z

@andikrueger AzureCP needs to store the certificate and its private key in a location that can be accessed by all SharePoint processes, in all SharePoint server of the farm (because SharePoint may want to call AzureCP from anywhere).
With this contraint in mind, I found no better solution than storing the certificate and its private key in the persisted object, in the SharePoint configuration database, and this is also the reason why there are hardcoded passwords.
I'm totally open to discussions for a better implementation, but I don't see the Windows certificate store as a possible/easy solution, considering the pain it will be to ensure it is availalbe (and up to date) on all SharePoint servers.

Answer 6 · 2023-01-30T10:17:16.000Z

Hello @Yvand,

Thanks for clarification.
However, I don't see any constraint to deploy the PFX certificate to all SharePoint server of the farm and give access to the private key to the necessary services accounts.
From CA then, you give the Thumbprint of the certificate and password.

Currently when you deploy AzureCP you need to add azurecp.dll and its dependent assemblies + also configure the proxy for Internet access on all servers of the farm.
So deploying the certificate on all servers and give permissions access to the private key is not really a surprise finally and make sense after all.

EDIT: when the certificate is renewed (and of course deployed on all servers), we can imagine a "edit" button on the AzureCP connection to update the Thumbprint and password (to avoid to remove the connection then recreate it).

Answer 7 · 2023-01-30T13:06:54.000Z

If I'm not mistaken, then there would be no need to store the password for the private key of the certificate. We simple would assign the permissions to the account to be able to access the private key.

And yes, it would be a manual step. On the other hand, this would allow for certain other management scenarios of renewing the certificate or pushing it from a central location to all servers that need access to this particular AAD application.

Answer 8 · 2023-01-30T13:43:17.000Z

Let me provide additional elements to feed this discussion:
The private key needs to be accessible to the following accounts (in all the servers of the SharePoint farm):

SharePoint farm account
All SharePoint application poool accounts
SharePoint crawler account

AzureCP also stores the client secret of the app registration in clear text in the SharePoint config db.
I totally agree it is not great in terms of security, which relies on how well this database is secure (and it should be because it is a critical one).

Initially I wanted to use SharePoint itself to secure this sensitive data but I found no API for that.
For example, it would be really nice to use the farm passphrase to encrypt the sensitive data, but I found no way to do that.
And I cannot use a dumb SecureString, because multiple servers neeed to be able to decode it (and as far as I remember, SecureString cannot do that).

Answer 9 · 2023-02-08T13:21:27.000Z

Here is another typical scenario when AzureCP Claim Provider stop working from Central Administration: add or remove a Claim type mapping from Claim types configuration page. It will generate the ULS errors described in my first post.

Most of the time iisreset solve the issue, but I had one case where I had to completely recreate the tenant connection.

Answer 10 · 2023-02-23T15:49:08.000Z

@julmsy I do not repro this issue. Is there any error recorded in SharePoint logs when you repro this?

Answer 11 · 2023-04-03T02:49:28.000Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Answer 12 · 2023-04-26T06:13:42.000Z

Keep this open. Still facing issues, and will back soon on this topic.

Answer 13 · 2023-04-28T14:47:18.000Z

So... I spent some time on it. And there is an interesting thing that need to be clarified.
Here are the ULS logs when I click on the Add tenant button from the AzureCP Global configuration page.

Azure AD tenant 'xxxyyyzzz.onmicrosoft.com' was successfully added to configuration 'AzureCPConfig'
[...]
Deserializing the type named azurecp.AzureCPConfig, AzureCP, Version=1.0.0.0, Culture=neutral, PublicKeyToken=xxxxxxxxxxxxxxxx and with id yyyyyyyy-zzzz-zzzz-zzzz-xxxxxxxxxxxx.
[...]
[AzureCP] Unexpected error occurred while deserializating the certificate for tenant 'xxxyyyzzz.onmicrosoft.com'.: System.Security.Cryptography.CryptographicException: The system cannot find the file specified.  , Callstack:   
 at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)    
 at System.Security.Cryptography.X509Certificates.X509Utils._LoadCertFromBlob(Byte[] rawData, IntPtr password, UInt32 dwFlags, Boolean persistKeySet, SafeCertContextHandle& pCertCtx)    
 at System.Security.Cryptography.X509Certificates.X509Utils.LoadCertFromBlob(Byte[] rawData, IntPtr password, UInt32 dwFlags, Boolean persistKeySet, SafeCertContextHandle pCertCtx)    
 at System.Security.Cryptography.X509Certificates.X509Certificate.LoadCertificateFromBlob(Byte[] rawData, Object password, X509KeyStorageFlags keyStorageFlags)    
 at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(Byte[] rawData, String password, X509KeyStorageFlags keyStorageFlags)    
 at azurecp.AzureTenant.OnDeserialization()

This Unexpected error appears generally multiple times (up to 9 times).
I confirm that I put the right certificate password.

As result:

People Picker from Central Administration is KO. I'm getting the following errors when I hit enter:

[AzureCP] Unexpected error occurred while getting access token for tenant 'xxxyyyzzz.onmicrosoft.com' on cloud instance 'AzurePublic': System.Security.Cryptography.CryptographicException: Keyset does not exist  , Callstack:   
 at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)    
 at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle& safeProvHandle, SafeKeyHandle& safeKeyHandle)    
 at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()    
 at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 dwKeySize, CspParameters parameters, Boolean useDefaultKeySize)    
 at System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()    
 at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.GetRSAPrivateKey(X509Certificate2 certificate)    
 at Microsoft.Identity.Client.PlatformsCommon.Shared.CryptographyManager.SignWithCertificate(String message, X509Certificate2 certificate)    
 at Microsoft.Identity.Client.Internal.JsonWebToken.Sign(X509Certificate2 certificate, String base64EncodedThumbprint, Boolean sendX5C)    
 at Microsoft.Identity.Client.Internal.ClientCredential.CertificateAndClaimsClientCredential.AddConfidentialClientParametersAsync(OAuth2Client oAuth2Client, ICoreLogger logger, ICryptographyManager cryptographyManager, String clientId, String tokenEndpoint, Boolean sendX5C, CancellationToken cancellationToken)    
 at Microsoft.Identity.Client.OAuth2.TokenClient.<AddBodyParamsAndHeadersAsync>d__7.MoveNext()  --- End of stack trace from previous location where exception was thrown ---    
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
 at Microsoft.Identity.Client.OAuth2.TokenClient.<SendTokenRequestAsync>d__5.MoveNext()  --- End of stack trace from previous location where exception was thrown ---    
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
 at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.<FetchNewAccessTokenAsync>d__3.MoveNext()  --- End of stack trace from previous location where exception was thrown ---    
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
 at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.<ExecuteAsync>d__2.MoveNext()  --- End of stack trace from previous location where exception was thrown ---    
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
 at Microsoft.Identity.Client.Internal.Requests.RequestBase.<RunAsync>d__12.MoveNext()  --- End of stack trace from previous location where exception was thrown ---    
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
 at Microsoft.Identity.Client.ApiConfig.Executors.ConfidentialClientExecutor.<ExecuteAsync>d__3.MoveNext()  --- End of stack trace from previous location where exception was thrown ---    
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
 at azurecp.TaskHelper.<TimeoutAfter>d__0`1.MoveNext()  --- End of stack trace from previous location where exception was thrown ---    
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
 at azurecp.AADAppOnlyAuthenticationProvider.<GetAccessToken>d__14.MoveNext()
[AzureCP] Unexpected error occurred Microsoft.Graph could not query tenant 'xxxyyyzzz.onmicrosoft.com': Microsoft.Graph.ServiceException: Code: InvalidAuthenticationToken  Message: Access token is empty.  Inner error:   AdditionalData:   date: 2023-04-28T14:23:25   request-id: yyyyyyyy-zzzz-zzzz-zzzz-xxxxxxxxxxxx   client-request-id: yyyyyyyy-zzzz-zzzz-zzzz-xxxxxxxxxxxx  ClientRequestId: yyyyyyyy-zzzz-zzzz-zzzz-xxxxxxxxxxxx  , Callstack:   
 at Microsoft.Graph.HttpProvider.<SendAsync>d__18.MoveNext()  --- End of stack trace from previous location where exception was thrown ---    
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
 at Microsoft.Graph.BaseRequest.<SendRequestAsync>d__38.MoveNext()  --- End of stack trace from previous location where exception was thrown ---    
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
 at Microsoft.Graph.BaseRequest.<SendAsync>d__34`1.MoveNext()  --- End of stack trace from previous location where exception was thrown ---    
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
 at Microsoft.Graph.GraphServiceUsersCollectionRequest.<GetAsync>d__4.MoveNext()  --- End of stack trace from previous location where exception was thrown ---    
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
 at azurecp.AzureCP.<>c__DisplayClass60_0.<<QueryAzureADTenantAsync>b__0>d.MoveNext()  --- End of stack trace from previous location where exception was thrown ---    
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
 at azurecp.AzureCP.<QueryAzureADTenantAsync>d__60.MoveNext()
[AzureCP] Got 0 users/groups in 122 ms from 'xxxyyyzzz.onmicrosoft.com' with input 'firstname.lastname'

But works nicely from SharePoint sites.

And I also tried on a vanilla SharePoint 2016 (just a CU applied, AzureCP (Nightly 20230426) and Nintex installed, and few SA provisioned), with a self-signing certificate for authentication with the App Registration: I also have this errors.

What are the potentials reasons that could cause issues during the deserializating process of the certificate?
It's however really strange that it don't work from CA but work from the sites. If there is a real issue at all on the certificate it would not work at all, not partially.

Let me know if you need more information.

Answer 14 · 2023-04-28T15:10:42.000Z

Could it depend on the account?
Do the web apps have a dedicated app pool account?
Does it work if you create a web app that uses the farm account?

Answer 15 · 2023-05-02T10:34:59.000Z

I've not finished all my tests, but could you try if you can repro this issue with the following configuration:

SharePoint topology/config: not really matter from what I've seen.
AzureCP tenant configuration: authentication with certificate.
Have a dedicated SharePoint admin account (like a-yvand account that you use to remote your server)
Open Central Admin with you a- account, and do some tests.

The complete test to done:

Create a fresh site collection with only one admin user (sometimes at this point the people picker is not working)
Once site collection created, still from CA, click Change site collection administrators
Add a second site collection admin (if issue, you are getting a "The user does not exist or is not unique." error)

Answer 16 · 2023-05-12T12:35:31.000Z

Hi @Yvand,

Here is a real typical issue I'm having:

I remote the SharePoint server with my personal admin account
I opened 2 IE browser: one with my personal admin account, the second with the SP Farm account

As you can see, on one case the Claim Provider is working, on the other, not.

I don't really explain this, but it's like if the first account/process that desarialize AzureCP have the "hand" on the certificate. Then if another account/process need to read the certificate and private key, there is an issue.

Answer 17 · 2023-05-12T12:46:02.000Z

@julmsy what is the error recorded in the SPS log when it does not work?
Edit: Does it change anything if you browse the central admin from the VM hosting it VS from a different machine? (I'm thinking about the UAC)

Answer 18 · 2023-05-15T07:43:43.000Z

Hi @Yvand,

The logs are the same as usual:

[AzureCP] Unexpected error occurred while setting the certificate for tenant 'xxx.onmicrosoft.com'. Is the private key of the certificate exportable?: System.Security.Cryptography.CryptographicException: Key not valid for use in specified state.  , Callstack:   
 at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)    
 at System.Security.Cryptography.X509Certificates.X509Utils._ExportCertificatesToBlob(SafeCertStoreHandle safeCertStoreHandle, X509ContentType contentType, IntPtr password)    
 at System.Security.Cryptography.X509Certificates.X509Certificate.ExportHelper(X509ContentType contentType, Object password)    
 at azurecp.AzureTenant.set_ClientCertificatePrivateKey(X509Certificate2 value)
 
[AzureCP] Unexpected error occurred in SearchOrValidate: System.Security.Cryptography.CryptographicException: Key not valid for use in specified state.  , Callstack:   
 at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)    
 at System.Security.Cryptography.X509Certificates.X509Utils._ExportCertificatesToBlob(SafeCertStoreHandle safeCertStoreHandle, X509ContentType contentType, IntPtr password)    
 at System.Security.Cryptography.X509Certificates.X509Certificate.ExportHelper(X509ContentType contentType, Object password)    
 at azurecp.AzureTenant.set_ClientCertificatePrivateKey(X509Certificate2 value)
 
[AzureCP] Validation failed: found 0 entities instead of 1 for incoming claim with value 'first.last@xxx.com' and type 'http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress'

Regarding permissions, my account is:

Server and Farm admin
dbo on SP Config DB

UAC is set on 'Never notify'.

For now I only did the tests from the SharePoint VM itself, I'll check if I can operate a test from another machine.

On your side, did you manage to repro this issue ?
So far, no problem with the secret authentication, this is really something related with certificate authentication.

Answer 19 · 2023-05-15T14:19:26.000Z

Interesting thing, but I was able to reverse the situation. With my account on the Central Admin I did:

Remove the actual tenant configuration on AzureCP
Add again the tenant
iisreset
-> In this scenario, the claim provider works with my account, but NOT on the browser opened with the Farm account. This situation is valid for a time (1h? - token valid time), then it's not work anymore on my account.

From what I saw from the logs, the issue is not related with the process (w3wp.exe), neither the thread.

Answer 20 · 2023-05-15T15:55:41.000Z

Based on your description, it looks like SharePoint does not impersonate the account in the central administration: It does all operations using the account credentials (instead of using the farm account credentials).
This is exactly the opposite in regular sites: Operations that involve (custom) claims providers are always impersonated to run as the application pool account.
But it does not explain what permission exactly the non-working account in the central admin lacks, and that the application pool account always have (assuming you never repro the issue in regular sites, regardless of the scenario you test).
For the sake of the test, I would still try to load central admin from a different VM (e.g. SQL VM) to double check that it makes no difference.
FYI I'm on vacation until June 1, I cannot say when I'll be able to replay the scenario myself.

Side note: In case you have an Azure subscription, could you consider to try to repro in either https://learn.microsoft.com/en-us/samples/azure/azure-quickstart-templates/sharepoint-adfs/ or https://registry.terraform.io/modules/Yvand/sharepoint/azurerm/latest ?
I maintain both and use them as my baseline for every test/dev/whatever I do with SPS

Answer 21 · 2023-05-15T16:06:21.000Z

I used your arm template to create my environment. I have seen the same issue as described here.

it is an interesting thought about the account elevation and the context change. Still the issues appeared to be in the handling of the certificate which would be obtained from the configuration object.

Answer 22 · 2023-05-16T13:00:55.000Z

Ok, I did the test on my VMs, as I'm encountering the same issue.
So I confirm that if I open the Central Administration remotely from my SQL VM, I have exactly the same behavior:

Works correctly if I open the browser with the Farm account
KO with another account

Thanks @andikrueger for the confirmation, at least it confirm that is not related to our environments.
Have a nice vacation @Yvand 🏖️ !

Answer 23 · 2023-06-05T10:02:38.000Z

@julmsy I reproduce a similar exception but not in the central admin or the main web app, somehow I get it on the applicationPool "SharePoint Service Applications".
Interestingly, "SharePoint Service Applications" runs as spsvc, which is neither the central admin account of the web app account.

For the little I tested, it works fine on both the central admin and the main web app.
I tested on SharePoint Subscription May 2023 CU.
I'll continue the investigations on this issue.

My exception:

[AzureCP] Unexpected error occurred while deserializating the certificate for tenant 'xxx.onmicrosoft.com'.: System.Security.Cryptography.CryptographicException: The system cannot find the file specified.  , Callstack:   
 at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)    
 at System.Security.Cryptography.X509Certificates.X509Utils._LoadCertFromBlob(Byte[] rawData, IntPtr password, UInt32 dwFlags, Boolean persistKeySet, SafeCertContextHandle& pCertCtx)    
 at System.Security.Cryptography.X509Certificates.X509Utils.LoadCertFromBlob(Byte[] rawData, IntPtr password, UInt32 dwFlags, Boolean persistKeySet, SafeCertContextHandle pCertCtx)    
 at System.Security.Cryptography.X509Certificates.X509Certificate.LoadCertificateFromBlob(Byte[] rawData, Object password, X509KeyStorageFlags keyStorageFlags)    
 at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(Byte[] rawData, String password, X509KeyStorageFlags keyStorageFlags)    
 at azurecp.AzureTenant.OnDeserialization()

Answer 24 · 2023-06-05T14:22:45.000Z

I made some progress but it is inconsistent:
Initially I installed Visual Studio and did live debug. At some point when I visit central admin > change site collection administrators, I see the following exception:

[AzureCP] Unexpected error occurred while getting access token for tenant 'M365x997769.onmicrosoft.com' on cloud instance 'AzurePublic': System.Security.Cryptography.CryptographicException: Access denied.  , Callstack:   
 at System.Security.Cryptography.NCryptNative.SetProperty(SafeNCryptHandle ncryptObject, String propertyName, Byte[] value, CngPropertyOptions propertyOptions)    
 at System.Security.Cryptography.CngKey.Open(SafeNCryptKeyHandle keyHandle, CngKeyHandleOpenOptions keyHandleOpenOptions)    
 at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.GetRSAPrivateKey(X509Certificate2 certificate)    
 at Microsoft.Identity.Client.PlatformsCommon.Shared.CryptographyManager.SignWithCertificate(String message, X509Certificate2 certificate)    
 at Microsoft.Identity.Client.Internal.JsonWebToken.Sign(X509Certificate2 certificate, String base64EncodedThumbprint, Boolean sendX5C)    
 at Microsoft.Identity.Client.Internal.ClientCredential.CertificateAndClaimsClientCredential.AddConfidentialClientParametersAsync(OAuth2Client oAuth2Client, ICoreLogger logger, ICryptographyManager cryptographyManager, String clientId, String tokenEndpoint, Boolean sendX5C, CancellationToken cancellationToken)    
 at Microsoft.Identity.Client.OAuth2.TokenClient.<AddBodyParamsAndHeadersAsync>d__7.MoveNext()  --- End of stack trace from previous location where exception was thrown ---    
[...] 
 at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()    
 at azurecp.AADAppOnlyAuthenticationProvider.<GetAccessToken>d__14.MoveNext() in D:\a\1\s\AzureCP\AADAppOnlyAuthenticationProvider.cs:line 123

Using procmon (log attached here cert-access-denied.zip) I saw many access denied for the central admin process (spfarm account is not local admin).
If I grant spfarm full control to either Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates or Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\EnterpriseCertificates and recycle, I get a different error:

[AzureCP] Unexpected error occurred while setting the certificate for tenant 'M365x997769.onmicrosoft.com'. Is the private key of the certificate exportable?: System.Security.Cryptography.CryptographicException: Keyset does not exist  , Callstack:   
 at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)    
 at System.Security.Cryptography.X509Certificates.X509Utils._ExportCertificatesToBlob(SafeCertStoreHandle safeCertStoreHandle, X509ContentType contentType, IntPtr password)    
 at System.Security.Cryptography.X509Certificates.X509Certificate.ExportHelper(X509ContentType contentType, Object password)    
 at azurecp.AzureTenant.set_ClientCertificatePrivateKey(X509Certificate2 value) in D:\a\1\s\AzureCP\AzureCPConfig.cs:line 759

EDIT:
If I delete and re-add the certificate in AzureCP configuration, whether spfarm has permissions in the registry keys, I get this error:

[AzureCP] Unexpected error occurred while deserializating the certificate for tenant 'M365x997769.onMicrosoft.com'.: System.Security.Cryptography.CryptographicException: The system cannot find the file specified.  , Callstack:   
 at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)    
 at System.Security.Cryptography.X509Certificates.X509Utils._LoadCertFromBlob(Byte[] rawData, IntPtr password, UInt32 dwFlags, Boolean persistKeySet, SafeCertContextHandle& pCertCtx)    
 at System.Security.Cryptography.X509Certificates.X509Utils.LoadCertFromBlob(Byte[] rawData, IntPtr password, UInt32 dwFlags, Boolean persistKeySet, SafeCertContextHandle pCertCtx)    
 at System.Security.Cryptography.X509Certificates.X509Certificate.LoadCertificateFromBlob(Byte[] rawData, Object password, X509KeyStorageFlags keyStorageFlags)    
 at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(Byte[] rawData, String password, X509KeyStorageFlags keyStorageFlags)    
 at azurecp.AzureTenant.OnDeserialization() in D:\a\1\s\AzureCP\AzureCPConfig.cs:line 812

But I get it only 1 time, then requests go to Azure AD as expected.
Does this remotely match some of your tests?

Answer 25 · 2023-06-14T12:38:50.000Z

@andikrueger @julmsy I think I have fixed the certificate issues, through 4f97d35 and fb6a28a.

Could you test the latest nightly release and confirm if it works also for you?

Answer 26 · 2023-06-15T10:06:25.000Z

Thanks for the update @Yvand.
Yes, for sure! Will test it soon.

Answer 27 · 2023-06-15T14:38:07.000Z

OK, I tested quickly on a DEV env. First impression: this is better.
I'll test it a bit deeper on the next days, and I will get back to you.

Answer 28 · 2023-06-20T14:56:09.000Z

So, for me it's all good. No more error at all regarding certificate.
Could you please explain a little bit the change/cause on this issue? You just change the way on how the certificate is added into the configDB ?

Last point: as this was the last of the last issues, do you intend to release a public version of AzureCP soon?

Answer 29 · 2023-06-21T07:19:32.000Z

@julmsy awesome, thank you for your feedback.
Basically I was inspired by https://support.microsoft.com/en-us/topic/kb5025823-change-in-how-net-applications-import-x-509-certificates-bf81c936-af2b-446e-9f7a-016f4713b46b
Then I read https://www.pkisolutions.com/handling-x509keystorageflags-in-applications/ and I much better understood what was happening in the code and the purpose of the flags.
So, in practice:

I import the pfx from the SPS config database using EphemeralKeySet : https://github.com/Yvand/AzureCP/blob/06d91a9976da47db67b4f8a089f8a82e99fc0249/AzureCP/AzureCPConfig.cs#LL826C1-L826C82. With this flag, the cert cannot be exported (persisted) anywhere, and that is perfectly fine since it is only intended to be used in-memory.
It is only when the certificate is selected by the administrator that it can be exported to SPS config database

Yes, publishing a release is high in my prio list. I'm currently actively working on updating my GitHub workflows to finish the migration from Azure DevOps, there is still work but I did the most difficult part

Answer 30 · 2023-08-07T06:19:51.000Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Answer 31 · 2023-09-16T22:34:59.000Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.