One-button deploy APIM, Keyvault, and Log Ananlytics. Auto-configure APIM to work with your Azure OpenAI endpoint.
Using Azure's APIM orchestration provides a organizations with a powerful way to scale and manage their Azure OpenAI service without deploying Azure OpenAI endpoints everywhere. Administrators can issue subscription keys via APIM for accessing a single Azure OpenAI service instead of having teams share Azure OpenAI keys. APIM delivers usage metrics along with API monitoring to improve business intelligence. APIM policies control access, throttling, and a mechanism for chargeback cost models.
There are four solutions developed to meet the needs of the organization from a sandbox to model a production environment.
- Azure Commercial API Management to Azure OpenAI
- Azure Commercial API Management to Azure OpenAI with private endpoints
- Azure Government API Management to Azure OpenAI
- Azure Government API Management to Azure OpenAI with private endpoints
Once the service is deployed, use the following section to understand how to access your Azure OpenAI service via APIM.
- Chargeback
link to further in doc
- Retry policy to leverage two or more Azure OpenAI endpoints
- Expands capacity without impact to user experience or requesting increase to existing Azure OpenAI endpoints
TBD
link to further in doc
- Provide cost management per subscription
link to further in doc
- Preview feature for two or more Azure OpenAI endpoints using round-robin load balancing
- Pair with Resiliency for highly scalable solution
link to further in doc
- Managed Identity
- Subscription Key (aka SAS Key)
link to further in doc
-
Contributor permissions to subscription or resource group
-
Resource Group (or ability to create)
-
Azure OpenAI service deployed
-
Azure OpenAI model deployed
-
Azure OpenAI service URL
-
Azure OpenAI key
-
Azure OpenAI service public IP
- Azure Government w/ Private Endpoint Deploy Solution Only
- How-to: Get the public IP address of Azure OpenAI service
Each solution provides a simple one-button deployment. Select the "Deploy to Azure" button which will open the Azure portal and provide a form for details.
To use the command line deployment method, fork the library and use Codespaces or clone the forked library to your local computer.
- How to install the Azure CLI | Microsoft Learn
- Connect to Azure Government with Azure CLI - Azure Government | Microsoft Learn
- How to install Azure PowerShell | Microsoft Learn
- Connect to Azure Government with PowerShell - Azure Government | Microsoft Learn
The following architectural solutions support two use-cases in the Azure Commercial and Azure Government environments. Determining which solution to implement requires understanding of your current utilization of Azure.
-
Azure Commercial API Management to Azure OpenAI
- Azure Commercial is primary cloud environment used by the team or organization.
- Developing proof of concept or minimum viable production solution.
- Isolated from enterprise networking using internal networks, Express Routes, and site-2-site VPN connections from the cloud to on-premesis networks.
-
Azure Commercial API Management to Azure OpenAI with private endpoints
- Azure Commerical is primary cloud environment used by the team or organization
- Pilot or production solution.
- Connected to the enterprise networking using internal networks, Express Routes, and site-2-site VPN connections from the cloud to on-premesis networks.
-
Azure Government API Management to Azure OpenAI
- Azure Government is primary cloud environment used by the team or organization.
- Developing proof of concept or minimum viable production solution.
- Isolated from enterprise networking using internal networks, Express Routes, and site-2-site VPN connections from the cloud to on-premesis networks.
-
Azure Government API Management to Azure OpenAI with private endpoints
- Azure Government is primary cloud environment used by the team or organization
- Pilot or production solution.
- Connected to the enterprise networking using internal networks, Express Routes, and site-2-site VPN connections from the cloud to on-premesis networks.
Use API management deployed to the Azure Commercial cloud using public IP addresses for accessing APIM and for APIM to access the Azure OpenAI API. Access to the services is secured using keys and Defender for Cloud.
! NOTE ! - It can take up to 45 minutes for all services to deploy as API Management has many underlying Azure resources deployed running the service.
Simple one-button deployment, opens in Azure Portal
# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "East US"
$subscriptionName = "MySubscription"
az login
az account set --subscription $subscriptionName
az group create --name $resourceGroupName --location $location
az deployment group create --resource-group $resourceGroupName --template-file .\public-apim.bicep --mode Incremental
# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "East US"
$subscriptionName = "MySubscription"
Connect-AzAccount
Set-AzContext -Subscription $subscriptionName
New-AzResourceGroup -Name $resourceGroupName -Location $location
New-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile .\public-apim.bicep -Verbose -mode Incremental
- Now that APIM is deployed and automatically configured to work with your Azure OpenAI service
Use API management deployed to the Azure Commercial cloud using private IP addresses for accessing APIM and for APIM to access the Azure OpenAI API. Access to the services is secured using private network connectivity, keys and Defender for Cloud. Access to the private network is controlled by customer infrastructure and supports internal routing via Express Route or site-2-site VPN for broader enterprise network access like on-premises data centers or site-based users.
! NOTE ! - It can take up to 45 minutes for all services to deploy as API Management has many underlying Azure resources deployed running the service.
Simple one-button deployment, opens in Azure Portal
# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "East US"
$subscriptionName = "MySubscription"
az login
az account set --subscription $subscriptionName
az group create --name $resourceGroupName --location $location
az deployment group create --resource-group $resourceGroupName --template-file .\private-apim.bicep --mode Incremental
# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "East US"
$subscriptionName = "MySubscription"
Connect-AzAccount
Set-AzContext -Subscription $subscriptionName
New-AzResourceGroup -Name $resourceGroupName -Location $location
New-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile .\private-apim.bicep -Verbose -mode Incremental
- Now that APIM is deployed and automatically configured to work with your Azure OpenAI service
Use API management deployed to the Azure Government cloud using public IP addresses for accessing APIM and for APIM to access the Azure Commercial-based Azure OpenAI API. Access to the services is secured using keys and Defender for Cloud.
Network routing from the APIM to the Azure OpenAI address uses Microsoft's backbone, eliminating public routing.
! NOTE ! - It can take up to 45 minutes for all services to deploy as API Management has many underlying Azure resources deployed running the service.
Simple one-button deployment, opens in Azure Portal
# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "usgovvirginia"
$subscriptionName = "MySubscription"
az cloud set --name AzureUSGovernment
az login
az account set --subscription $subscriptionName
az group create --name $resourceGroupName --location $location
az deployment group create --resource-group $resourceGroupName --template-file .\public-apim.bicep --mode Incremental
# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "East US"
$subscriptionName = "MySubscription"
Connect-AzAccount -Environment AzureUSGovernment
Set-AzContext -Subscription $subscriptionName
New-AzResourceGroup -Name $resourceGroupName -Location $location
New-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile .\public-apim.bicep -Verbose -mode Incremental
- Now that APIM is deployed and automatically configured to work with your Azure OpenAI service
Use API management deployed to the Azure Government cloud using private IP addresses for accessing APIM and for APIM to access the Azure Commercial-based Azure OpenAI API.
Access to the services is secured using private network connectivity, keys and Defender for Cloud. Access to the private network is controlled by customer infrastructure and supports internal routing via Express Route or site-2-site VPN for broader enterprise network access like on-premises data centers or site-based users.
Network routing from the APIM to the Azure OpenAI address uses Microsoft's backbone, eliminating public routing.
! NOTE ! - It can take up to 45 minutes for all services to deploy as API Management has many underlying Azure resources deployed running the service.
Simple one-button deployment, opens in Azure Portal
# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "usgovvirginia"
$subscriptionName = "MySubscription"
az cloud set --name AzureUSGovernment
az login
az account set --subscription $subscriptionName
az group create --name $resourceGroupName --location $location
az deployment group create --resource-group $resourceGroupName --template-file .\private-apim-azure_government.bicep --mode Incremental
# Update the following variables to use the appropriate resource group and subscription.
$resourceGroupName = "RG-APIM-OpenAI"
$location = "East US"
$subscriptionName = "MySubscription"
Connect-AzAccount -Environment AzureUSGovernment
Set-AzContext -Subscription $subscriptionName
New-AzResourceGroup -Name $resourceGroupName -Location $location
New-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile .\private-apim-azure_government.bicep -Verbose -mode Incremental
- Now that APIM is deployed and automatically configured to work with your Azure OpenAI service
TBD
Policy for collecting tokens and user id
TBD
TBD
TBD
TBD
TBD
Azure API Management policy reference - retry | Microsoft Learn
TBD
Ensure reliability of your Azure API Management instance - Azure API Management | Microsoft LearnThrottling
TBD
Advanced request throttling with Azure API Management | Microsoft Learn
- Provide cost management per subscription
- Preview feature for two or more Azure OpenAI endpoints using round-robin load balancing
- Pair with Resiliency for highly scalable solution
TBD
TBD
TBD
Read through the following steps to setup interacting with APIM and how to use consoles or .net to programatically interact with Azure OpenAI via APIM.
To determine if you have one or more models deployed, visit the AI Studio. Here you can determine if you need to create a model or use an existing model. You will use the model name when quering the Azure OpenAI API via your APIM.
-
Navigate to your Azure OpenAI resource in Azure
-
Select Model deployments
-
Select Manage Deployments
-
Review your models and copy the Deployment name of the model you want to use
The subscription key for APIM is collected at the Subscription section of the APIM resource, regardless if you are in Azure Commercial or Government.
You can use this key for testing or as an example on how to create subscriptions to provide access to you Azure OpenAI service. Instead of sharing your Azure OpenAI Key, you create subscriptions in APIM and share this key, then you can analyze and monitor usage, provide guardrails for usage, and manage access.
- Navigate to your new APIM
- Select Subscriptions from the menu
- Select ...
- Select Show/Hide keys
- Select copy icon
The URL for APIM is collected at the Overview section of the APIM resource, regardless if you are in Azure Commercial or Government.
Using your Azure OpenAI model, API version, APIM URL, and APIM subscription key you can now execute Azure OpenAI queries against your APIM URL instead of your Azure OpenAI URL. This means you can create new subscription keys for anyone or any team who needs access to Azure OpenAI instead of deploying new Azure OpenAI services.
Copy and paste this script into a text editor or Visual Studio code.
Modify by including your values, then copy and paste all of it into PowerShell 7 terminal.
# Update these values to match your environment
$apimUrl = 'THE_HTTPS_URL_OF_YOUR_APIM_INSTANCE'
$modelName = 'GPT-3_5-Turbo' # Probaby what you named your model, but change if necessary
$apiVersion = '2023-03-15-preview' # Do not change this value, unless you are testing a different API version
$subscriptionKey = 'YOUR_APIM_SUBSCRIPTION_KEY'
# Do not touch these values
$url = $apimUrl + "/deployments/" + $modelName + "/chat/completions?api-version=" + $apiVersion
$key = "Ocp-Apim-Subscription-Key: " + $subscriptionKey
curl $url -k -H "Content-Type: application/json" -H $key -d '{
"messages": [
{
"role": "system",
"content": "You are an AI assistant that helps people find information."
},
{
"role": "user",
"content": "What are the differences between Azure Machine Learning and Azure AI services?"
}
]
}'
Copy and paste this script into a text editor or Visual Studio code.
Modify by including your values, then copy and paste all of it into bash terminal or create a ".sh" file to run.
#!/bin/bash
apimUrl="THE_HTTPS_URL_OF_YOUR_APIM_INSTANCE"
modelName="GPT-3_5-Turbo" # Probaby what you named your model, but change if necessary
apiVersion="2023-03-15-preview" # Do not change this value, unless you are testing a different API version
subscriptionKey="YOUR_APIM_SUBSCRIPTION_KEY"
url="${apimUrl}"/deployments/"${modelName}"/chat/completions?api-version="${apiVersion}"
key="Ocp-Apim-Subscription-Key: ${subscriptionKey}"
curl $url -k -H "Content-Type: application/json" -H $key -d '{
"messages": [
{
"role": "system",
"content": "You are an AI assistant that helps people find information."
},
{
"role": "user",
"content": "What are the differences between Azure Machine Learning and Azure AI services?"
}
]
}'
// Note: The Azure OpenAI client library for .NET is in preview.
// Install the .NET library via NuGet: dotnet add package Azure.AI.OpenAI --version 1.0.0-beta.5
using Azure;
using Azure.AI.OpenAI;
OpenAIClient client = new OpenAIClient(
new Uri("https://INSERT_APIM_URL_HERE/deployments/INSERT_MODEL_NAME_HERE/chat/completions?api-version=INSERT_API_VERSION_HERE"),
new AzureKeyCredential("INSERT_APIM_SUBSCRIPTION_KEY_HERE"));
// ### If streaming is not selected
Response<ChatCompletions> responseWithoutStream = await client.GetChatCompletionsAsync(
"INSERT_MODEL_NAME_HERE",
new ChatCompletionsOptions()
{
Messages =
{
new ChatMessage(ChatRole.System, @"You are an AI assistant that helps people find information."),
new ChatMessage(ChatRole.User, @"What are the differences between Azure Machine Learning and Azure AI services?"),
},
Temperature = (float)0,
MaxTokens = 800,
NucleusSamplingFactor = (float)1,
FrequencyPenalty = 0,
PresencePenalty = 0,
});
// The following code shows how to get to the content from Azure OpenAI's response
ChatCompletions completions = responseWithoutStream.Value;
ChatChoice choice = completions.Choices[0];
Console.WriteLine(choice.Message.Content);
When deploying to Azure Government with Private endpoints, the deployment process requires the public IP address of the Azure OpenAI service. Use ping or nslookup with fqdn of your Azure OpenAI url to determine it's public IP address.
- example:
- url is https://aoai.openai.azure.com
- fqdn is aoai.openai.azure.com
Then use nslookup in PowerShell terminal, or Linux console, along with the fqdn to find out the public IP address of your Azure OpenAI service.
There are two methods of securely storing the SAS key
Steps to automate updating the storage location of the SAS key following the renewal period of the SAS key.