[Bug] Manual Token with azure ai search gives back forbidden

[ginopoitier]

Context / Scenario

We are currently working on implementing KM as a RAG service, we use the serverless version and use azure ai as vector DB, using the api token it all works fine but switching to use Entra Managed identity it stops working

Our code looks like this to add KM to the DI

var azureAiSearchConfig = new AzureAISearchConfig();
var openAiConfig = new OpenAIConfig();    
var memoryConfiguration = new KernelMemoryConfig();    
var searchClientConfig = new SearchClientConfig();    
var simpleFileStorage = new SimpleFileStorageConfig();    
var azureEntraConfig = new AzureEntraConfig();        

builder.Configuration       
        .BindSection("KernelMemory:Retrieval:SearchClient", searchClientConfig)
        .BindSection("KernelMemory:Services:AzureAISearch", azureAiSearchConfig)
        .BindSection("KernelMemory:Services:OpenAi", openAiConfig)
        .BindSection("KernelMemory", memoryConfiguration)
        .BindSection("KernelMemory:Services:SimpleFileStorage", simpleFileStorage)
        .BindSection("AzureEntraConfig", azureEntraConfig);   
 
if (!string.IsNullOrWhiteSpace(azureEntraConfig.ClientId))    
{
        var credential = new ClientSecretCredential(azureEntraConfig.TenantId, azureEntraConfig.ClientId, azureEntraConfig.ClientSecret);        
        azureAiSearchConfig.SetCredential(credential);    
}

var apiKey = Environment.GetEnvironmentVariable("OpenAiApiKey");
        
if (!string.IsNullOrWhiteSpace(apiKey))
{
        openAiConfig.APIKey = apiKey!;
}

var memory = new KernelMemoryBuilder()
        .Configure(kmb => kmb.Services
        .AddLogging(l => l.AddSerilog(Log.Logger)))
        .WithDefaultMimeTypeDetection()
        .AddSingleton(memoryConfiguration)
        .WithOpenAI(openAiConfig)
        .WithAzureAISearchMemoryDb(azureAiSearchConfig)
        .WithSimpleFileStorage(simpleFileStorage)
        .WithSearchClientConfig(searchClientConfig)
        .WithContentDecoder<PdfTronDecoder>()
        .Build<MemoryServerless>();
       
builder.Services.AddSingleton<IKernelMemory>(memory);

What happened?

Did i forget to configure something, i tried looking at the docs but didn't find anything relevant, does KM need very specific permissions?

Importance

edge case

Platform, Language, Versions

.NET Core 9
C#
KM v0.95.241216.2

Relevant log output

[dluc]

It might be a matter of IAM permissions assigned to the principal, did you grant these?

• Search Service Contributor: [...] create, delete, and list indexes, get index definitions [...]
• Search Index Data Contributor: [...] import, refresh, or query the documents collection of an index. This role doesn't support index creation or management. By default, this role is for all indexes on a search service.