The goal of this article is to list the architectures and designs addressing the following AVS connectivity patterns in a Hub & Spoke topology:
- AVS <=> On-Prem
- AVS <=> Spokes
- AVS to Internet
As represented on the screenshot below, there are 3 options to provide internet breakout to AVS:
-
via an Azure Firewall or FW NVA already hosted in Azure
-
Via public IP at NSX-T level (allowing to deploy a 3P NVA in the AVS environment)
-
Via AVS managed SNAT solution (SNAT through an AVS NAT GW)
This repo is about option 1 and how to leverage an Azure Firewall or a FW NVA already used in an Azure VNet environment and extend it to AVS inspection and AVS internet connectivity.
Most of these designs are already documented across multiple sources and have been referenced accordingly in this article.
1. SINGLE HUB VNET ARCHITECTURE
1.1. SINGLE HUB VNET WITH FW NVA
1.2. SINGLE HUB VNET WITH AZURE FIREWALL
1.3. SINGLE HUB VNET WITHOUT GLOBAL REACH
2. HUB VNET AND AVS TRANSIT VNET ARCHITECTURE (WITHOUT GLOBAL REACH)
2.1. HUB VNET AND AVS TRANSIT VNET WITH NVAs
2.2. HUB VNET AND AVS TRANSIT VNET WITH AZURE FIREWALL
With this architecture, all the flows are sent through the FW for inspection: Spoke <=> Spoke, Spoke <=> On-Prem, Spoke <=> AVS, On-Prem <=> AVS (when Global Reach is not used).
Key deployment elements:
➡️ In the following 1.1 and 1.2 designs, the ARS only purpose is to push the default route to AVS.
➡️ The Spoke subnets are configured with GW route propagation disabled, hence not receiving the routes programmed by ARS. The default route to the FW needs to be enforced with a UDR on the Spokes.
➡️ VNet peering routes are preferred over ARS propagated routes (even if ARS routes are more specific): UDRs on the GW subnet are required to force the traffic to the Spoke VNets through the FW.
In section 1.3. we will see how this design can be adapted to offer On-Prem <=> AVS transit and filtering capabilities when Global Reach is not available or not aligned with the customer requirements (On-Prem <=> AVS traffic required to be inspected by a FW in an Azure VNet).
As the Azure Firewall doesn't speak BGP, a routing NVA is added to advertise the default route to the ARS for further propagation. This routing NVA is here only to generate the default route and won't be in the data path.
➡️ This design leverages the Next-hop IP feature supported by the ARS and that allows the NVA to set the Next-Hop of the default route advertised to be the AzFW.
Without Global Reach, as explained in this video by Adam, On-Prem <=> AVS transit can be achieved via the FW in the Hub VNet by configuring additional static routes and UDRs (highlighted on the diagrams below):
- NVA: static routes towards the AVS ranges pointing to the ARS facing NIC to be propagated On-Prem by the ARS. These static routes MUST be supernets of the specific AVS ranges to avoid a routing loop.
- (The On-Prem reachability from AVS is covered by the default route, or, like for AVS, an On-Prem supernet could be statically configured on the NVA and advertised to the ARS.)
- UDRs on the GW subnet to force both the AVS and On-Prem traffic through the FW. These UDRs MUST specifically match the On-Prem and AVS ranges.*
* The max number of UDRs in an Azure Route table is 400, ie: this architecture doesn't scale for scenarios where the (# of advertised On-Prem prefixes) > 400 - (# of AVS ranges) - (# of Spoke ranges).
➡️ Here the role of the ARS is to push the default route to AVS and the AVS ranges to the On-Prem to enable the On-Prem <=> AVS transit.
The internet breakout and transit with a supernet design topology designs are publicly documented on the referenced links.
Details of this design can be found in this video shared by Adam.
This approach is an alternative to design 1.1 bis and design 1.2 bis (when Global Reach is either not supported or not adapted to the customer requirements).
The 4 drivers for this architeture are the following:
- unblocking On-Prem <=> AVS transit scenarios (when GR is not available/adapted) in which there are thousands of On-Prem prefixes
- keeping the knowledge of the AVS/On-Prem route granularity, for example during the AVS migration phase
- avoiding configuration maintenance complexity
- having the ability if needed/preferred to bypass FW inspection for Spokes <=> On-Prem connectivity*, while mandatory in the single hub design.
* If Spoke <=> On-Prem filtering is not needed, disabling GW route propagation on the Spoke subnets is no longer required.
The public documentation for the transit design part is available on this link.
| resources | actions |
|---|---|
| FW NVA | 1/ originates and advertises the default route, 2/ forwards the AVS ranges to ARS1, 3/ forwards the On-Prem prefixes and H&S VNet ranges to the AVS NVA |
| AVS NVA | 1/ advertises the AVS ranges to the FW NVA, 2/ forwards the On-Prem and H&S VNet ranges to ARS2 |
| ARS1 | 1/ propagates the default route learnt from the FW NVA + the AVS ranges forwarded by the AVS NVA via the FW NVA to the On-Prem over ER and to the Spoke VNets, 2/ advertises the OnPrem + the hub and spoke VNet ranges to the FW NVA |
| ARS2 | 1/ propagates the default route + the On-Prem + the H&S VNet ranges to AVS, 2/ advertises the AVS ranges to the AVS NVA |
➡️ Internet connectivity can be provided either by the FW NIC facing the AVS NVA or a 3rd dedicated NIC on the FW NVA.
2 options for the FW NVA to AVS NVA transit:
- UDRs
- on the FW NVA NIC to AVS VNet: AVS ranges to the AVS NVA NIC,
- on the AVS NVA NIC to the Hub VNet: 0/0 to the FW NVA NIC
- disable GW route propagation on these 2 NICs
- BGP over IPSec/VxLAN if the granularity of the AVS and/or OnPrem routes are required
➡️ If stateful, the NVA instances should be configured as Active/Standby to avoid asymmetric routing.
For reasons already discussed in a previous article the eBGP session between the FW NVA and the AVS NVA is established within a VxLAN or IPSec tunnel:
- the FW NVA advertises the default route, the On-Prem prefixes and the Hub & spoke ranges to the AVS NVA.
- The AVS NVA advertises the AVS ranges to the FW NVA.
This approach is useful when the FW doesn't speak BGP, like the Azure Firewall.
A similar design is documented in detail here.
| resources | actions |
|---|---|
| AVS NVA | 1/ originates and advertises the default route, 2/ learns the AVS ranges from ARS2 and forwards them to ARS1, 3/ for these routes, the Next-Hop is updated to be the Azure Firewall*, 4/ forwards the On-Prem and H&S VNet ranges to ARS2 (the Next-Hop remains unchanged and will be the AVS NVA NIC facing ARS2). |
| ARS1 | 1/ propagates the default route + the AVS ranges learnt from the AVS NVA to the On-Prem over ER and to the Spoke VNets (with Next-Hop = AzFW), 2/ advertises the OnPrem + the H&S VNet ranges to the FW NVA |
| ARS2 | 1/ propagates the default route + the On-Prem + the H&S VNet ranges to AVS, 2/ advertises the AVS ranges to the AVS NVA |