Goal of this article is to explain As-path prepending behaviors when using ExpressRoute and Azure hubs with Azure VMware Solution and why Public ASNs should be used to prepend towards AVS.
Scenario1: Single ExpressRoute Circuit. Use one link as primary and second for failover using prepends with private ASN
I have a single ExpressRoute circuit with two links. On-prem prefix 10.61.0.0/24 is being advertised from both links of ExpressRoute with different As-path length. Path a advertises with shorter As-path and Path b advertises with longer As-path
In a single ExpressRoute circuit you have two physical connection. MSEE1 has BGP session with R1 and MSEE2 has BGP session with R2
What MSEE1 sees?
What MSEE2 sees?
How does DMSEE1 sees the route?
DMSEE1 sees the prefix 10.61.0.0/24 without prepend
How does DMSEE2 sees the route with prepend?
How does DMSEE advertise the route to AVS Edge?
All the private ASNs are stripped off by both DMSEEs before advertising them to AVS edge leaf. AVS Edge leaf will receive the prefix 10.61.0.0/24 from both DMSEEs with Aspath of 12076 Hence cannot differentiate which DMSEE is a better path as both have same Aspath length and will pick any . Hence with a single ExpressRoute circuit path preference is not possible.
- Use Public ASN for prepends as DMSEEs will not strip Public AS and now AVS leaf will be able to identify the best path based on the As-path prepend Here is an example advertised route where Public ASN is maintained
- Contact Azure Support who will help implement prepend on the inbound direction of AVS leaf to achieve path preference. This will not be visible to customer and every time any change needs to be done on prepends you need to contact support.
In this example one ExpressRoute circuit is in Dallas and another one in Chicago. We want to use Dallas as the primary path and Chicago as the Secondary/failover path.
All path a, b, c and d will advertise on-prem prefix 10.61.0.0/24. Path a and b will advertise 10.61.0.0/24 without any additional prepends. I will only put sample outputs from MSEE1. Both MSEE1 and MSEE2 routes would be identical.
What MSEE1 sees?
Path c and d will advertise 10.61.0.0/24 with prepends.
What MSEE3 sees?
What DMSEE1 sees?
The BGP routing table has both routes from path a and c. But Path a is preferred as it is shorter
What DMSEE2 sees?
The BGP routing table has both routes from path b and d. But Path b is preferred as it is shorter
How is the route advertised to AVS Edge?
The ASNs are still stripped of by the DMSEE. But since both DMSEE have identical routes and both identify Dallas(a and b) as shorter path. Hence the traffic from AVS to On-Prem will always route through Dallas circuit. Unless the routes are lost or the circuit is down.
While everything will route as expected when the all links are up and stable. But lets assume primary link between Onprem and MSEE1 is down and onprem routes are not advertised via MSEE1 to DMSEE1. In this scenario when AVS tries to reach Onprem since it only sees ASN 12076 the traffic might end up on DMSEE1 which has the path via Chicago and not Dallas. As you can see even though the secondary link from Dallas is up the possibility that Chicago path is used is still there.
- Use Public ASN for prepends as DMSEEs will not strip Public AS and now AVS leaf will be able to identify the best path based on the As-path prepend Here is an example advertised route where Public ASN is maintained
- Contact Azure Support who will help implement prepend on the inbound direction of AVS leaf to achieve path preference. This will not be visible to customer and every time any change needs to be done on prepends you need to contact support.
Scenario 3:Need internet breakout from AVS to be routed through an Azure hub. Need to have a backup Internet path through another hub.
There are two Hub with two NVAs each(I used CSR in my setup) advertising 0/0 using ARS and have next hop as the ILB IP1 and ILB IP2.
What DMSEE1 sees?
It receives 0/0 from both Vnet-prod and Vnet-dev. Vnet-prod is the preferred path due to shorter As-path length
What DMSEE2 sees?
It is identical to DMSEE1 routes
What DMSEE advertises to AVS edge?
The ASNs are still stripped of by the DMSEE. But since both DMSEE have identical routes and both identify path a as shorter path due to As-path length. Hence the traffic from AVS to Internet will always route through Vnet-Prod. Unless the routes are lost from Vnet-Prod.