Azure OpenAI Insights

The 'Azure OpenAI Insights' workbook offers deep insights into Azure OpenAI usage, helping you manage costs, optimize performance, and make strategic decisions for a robust AI infrastructure.

Introduction

In the ever-evolving world of Artificial Intelligence, organizations and entities across various sectors are on a quest to leverage advanced technologies efficiently. Azure OpenAI opens a realm of possibilities, offering both challenges and excitement, particularly for those at the early stages of AI adoption.

Read more in depth in this Tech Community blog: Azure OpenAI Insights: Monitoring AI with Confidence

This workbook offers deep insights into Azure OpenAI resources and usage (Platform Metrics and Logs) and can be powerful tool in analyzing & monitoring your AI initiatives.

Structure and Views

Structure

This workbook contains 3 main parts:

Overview - Holistic view of Azure OpenAI resources
Monitor - Holistic view of Azure OpenAI resources Metrics
Insights - Holistic view of Azure OpenAI resources Logs
- Requires by enabling Diagnostic Settings to Log Analytics Workspace.

Views

Types of views this workbook provides:

Overview
- Azure OpenAI Resources by
  - SubscriptionId
  - Resource Group
  - Location
  - Kind
  - Public Network Access
  - Private Network Access
- All Azure OpenAI Resources

The information displayed uses KQL queries to query the Azure Resource Graph.

Monitor
- Overview
  - Requests
  - Processed Inference Tokens
  - Processed Prompt Tokens
  - Generated Completions Tokens
  - Processed FineTuned Training Hours
  - Provisioned-managed Utilization
- HTTP Requests
  - by Model Name
  - by Model Version
  - by Model Deployment Name
  - by Status Code
  - by StreamType
  - by Operation Name
  - by API Name
  - by Region
- Token-Based Usage
  - Processed Inference Tokens
    - by Model Name
    - by Model Deployment Name
  - Processed Prompt Tokens
    - by Model Name
    - by Model Deployment Name
  - Generate Completitions Tokens
    - by Model Name
    - by Model Deployment Name
  - Active Tokens
    - by Model Name
    - by Model Deployment Name
- PTU Utilization
  - Provisioned-managed Utilization
    - by Model Name
    - Model Version
    - by Model Deployment Name
    - by StreamType
- Fine-tuning
  - Processed FineTuned Training Hours
    - by Model Name
    - by Model Deployment Name

The information displayed uses Azure OpenAI Platform Metrics and presented by multiple dimensions.

Insights
- Overview
  - Request / Response
    - by Resource
    - by Location
    - by StreamType
    - by Model Name + API Operation Name
    - by Model Name
    - by API Operation Name
    - by Model Deployment Name + API Operation Name
    - by Model Deployment Name
    - by Avg Duration (ms)
- By CallerIP
  - Request / Response
  - Avg Duration (ms)
  - Model Name + API Operation Name
  - Model Deployment Name + API Operation Name
  - API Operation Name
- All Logs

The information displayed uses KQL queries to query the Log Analytics Workspace that store the logs.

Note: The Logs will be available on resources that enabled Diagnostic Settings to Log Analytics Workspace.

How to use it?

Importing this Workbook to your Azure environment is quite simple.

Follow this steps to use the Workbook:

Login to Azure Portal
Go to 'Azure Workbooks'

Click on '+ Create'

Click on '+ New'

Open the Advanced Editor using the '</>' button on the toolbar

Select the 'Gallery Template' (step 1)
Replace the JSON code with this JSON code Azure OpenAI Insights JSON (step 2)
- We use the Gallery Templaty type (step 1), so we need to use the 'Azure OpenAI Insights.workbook' and not the 'Azure OpenAI Insights.json'.
Click 'Apply' (step 3)

Click in the ‘Save’ button on the toolbar

Select a name and where to save the Workbook:
- Title: 'Azure OpenAI Insights'
- Subscription: <Subscription Name>
- Resource group: <Resource Group Name>
- Location: <Region>
Click 'Save'

The Workbook is ready to use!

Click on 'Workbooks'
Click on 'Azure OpenAI Insights' Workbook.

Start using the Workbook and analyze your Azure OpenAI resources.