The rapid pace of AI advances demands experimentation-driven approaches for organizations to remain at the forefront of the industry. With AI steadily becoming a game-changer for an array of sectors, maintaining a fast-paced innovation trajectory is crucial for businesses aiming to leverage its full potential.
AI services are predominantly accessed via APIs, underscoring the essential need for a robust and efficient API management strategy. This strategy is instrumental for maintaining control and governance over the consumption of AI services.
With the expanding horizons of AI services and their seamless integration with APIs, there is a considerable demand for a comprehensive AI Gateway concept, which broadens the core principles of API management. Aiming to accelerate the experimentation of advanced use cases and pave the road for further innovation in this rapidly evolving field. The well-architected principles of the AI Gateway provides a framework for the confident deployment of Intelligent Apps into production..
This repo explores the AI Gateway concept through a series of experimental labs. Azure API Management plays a crucial role within these labs, handling AI services APIs, with security, reliability, performance, overall operational efficiency and cost controls. The primary focus is on Azure OpenAI, which sets the standard reference for Large Language Models (LLM). However, the same principles could potentially be applied to any LLM.
Acknowledging the rising dominance of Python, particularly in the realm of AI, along with the powerful experimental capabilities of Jupyter notebooks, the following labs are structured around Jupyter notebooks, with step-by-step instructions with Python scripts, Bicep files and APIM policies:
| Request forwarding |  |
Playground to try forwarding requests to either an Azure OpenAI endpoint or a mock server. APIM uses the system managed identity to authenticate into the Azure OpenAI service. |
| Backend circuit breaking |  |
Playground to try the built-in backend circuit breaker functionality of APIM to either an Azure OpenAI endpoints or a mock server. |
| Backend pool load balancing |  |
Playground to try the built-in load balancing backend pool functionality of APIM to either a list of Azure OpenAI endpoints or mock servers. |
| Advanced load balancing |  |
Playground to try the advanced load balancing (based on a custom APIM policy) to either a list of Azure OpenAI endpoints or mock servers. |
| Response streaming |  |
Playground to try response streaming with APIM and Azure OpenAI endpoints to explore the advantages and shortcomings associated with streaming. |
| Vector searching |  |
Playground to try the Retrieval Augmented Generation (RAG) pattern with Azure AI Search, Azure OpenAI embeddings and Azure OpenAI completions. All the endpoints are managed via APIM. |
| Built-in logging |  |
Playground to try the buil-in logging capabilities of API Management. The requests are logged into Application Insights and it's easy to track request/response details and token usage with provided notebook. |
| SLM self-hosting |  |
Playground to try the self-hosted phy-2 Small Language Model (SLM) trough the APIM self-hosted gateway with OpenAI API compatibility. |
- Developer tooling
- App building
- Token counting
- Semantic caching
- Token rate limiting
- Cost tracking
- Content filtering
- PII handling
- Prompt storing
- Function calling
- Prompt guarding
- Prompt model routing
- Python 3.8 or later version installed
- VS Code installed with the Jupyter notebook extension enabled
- Azure CLI installed
- An Azure Subscription with Contributor permissions
- Access granted to Azure OpenAI or just enable the mock service
- Sign in to Azure with Azure CLI
- Clone this repo and configure your local machine with the prerequisites. Or just create a GitHub Codespace and run it on the browser or in VS Code.
- Navigate through the available labs and select one that best suits your needs. For starters we recommend the request forwarding with just the Azure CLI or the backend pool load balancing with Bicep.
- Open the notebook and run the provided steps.
- Tailor the experiment according to your requirements. If you wish to contribute to our collective work, we would appreciate your submission of a pull request.
The Azure Well-Architected Framework is a design framework that can improve the quality of a workload. The following table maps labs with the Well-Architected Framework pillars to set you up for success through architectural experimentation.
| Lab | Security | Reliability | Performance | Operations | Costs |
|---|---|---|---|---|---|
| Request forwarding | ⭐ | ||||
| Backend circuit breaking | ⭐ | ⭐ | |||
| Backend pool load balancing | ⭐ | ⭐ | ⭐ | ||
| Advanced load balancing | ⭐ | ⭐ | ⭐ | ||
| Response streaming | ⭐ | ⭐ | |||
| Vector searching | ⭐ | ⭐ | ⭐ | ||
| Built-in logging | ⭐ | ⭐ | ⭐ | ⭐ | ⭐ |
| SLM self-hosting | ⭐ | ⭐ |
Tip
Check the Azure Well-Architected Framework perspective on Azure OpenAI Service for aditional guidance.
The AI-Gateway Mock server is designed to mimic the behavior and responses of the OpenAI API, thereby creating an efficient simulation environment suitable for testing and development purposes on the integration with APIM and other use cases. The app.py can be customized to tailor the Mock server to specific use cases.
Tip
Install the VS Code Reveal extension, open AI-GATEWAY.md and click on 'slides' at the botton to present the AI Gateway without leaving VS Code. Or just open the AI-GATEWAY.pptx for a plain old PowerPoint experience.
Numerous reference architectures, best practices and starter kits are available on this topic. Please refer to the resources provided if you need comprehensive solutions or a landing zone to initiate your project. We suggest leveraging the AI-Gateway labs to discover additional capabilities that can be integrated into the reference architectures.
- AI Hub Gateway Landing Zone
- Azure OpenAI Using PTUs/TPMs With API Management - Using the Scaling Special Sauce
- Manage Azure OpenAI using APIM
- Setting up Azure OpenAI as a central capability with Azure API Management
- Introduction to Building AI Apps
We believe that there may be valuable content that we are currently unaware of. We would greatly appreciate any suggestions or recommendations to enhance this list.
Important
This software is provided for demonstration purposes only. It is not intended to be relied upon for any purpose. The creators of this software make no representations or warranties of any kind, express or implied, about the completeness, accuracy, reliability, suitability or availability with respect to the software or the information, products, services, or related graphics contained in the software for any purpose. Any reliance you place on such information is therefore strictly at your own risk.