In this lesson, we will explore an OMP configuration parameter called send-path-limit. Along the way, we will discuss in detail how the OMP best-path algorithm works and what are the key points in the selection process that are often overlooked.
What is OMP Send Path Limit?
OMP Send-Path-Limit is a configuration parameter that defines the maximum number of equal-cost vroutes that a vSmart controller or a vEdge router advertises to its OMP peers. By default, the omp send-path-limit is set to 4, which means that if a vSmart controller has ten equal-cost best paths to a destination, it will only advertise four routes to vEdges.
Changing the send-path-limit value is as simple as applying one configuration line under the omp configuration hierarchy, as shown in the output below:
vEdge/vSmart# conf t
Entering configuration mode terminal
vEdge/vSmart(config)#omp send-path-limit (1-16)
vEdge/vSmart(config)# commit and-quit
Commit complete.
vEdge/vSmart#
OMP Send-Path-Limit Configuration Example
The initial state
To demonstrate how the send-path-limit parameter works in the context of the OMP Best-Path algorithm, we are going to use the network devices shown in figure 1 below. We have three WAN edge routers - vEdges 1, 2, and 3 that are directly connected to network 10.1.1.0/24 in VPN 100 and one vSmart controller that oversees the sd-wan domain. We have three more vEdge routers (vEdges 4, 5, and 6) that will be referred to as "the other" vEdges.
There are two WAN transports that are not illustrated in the topology - an Internet cloud and an MPLS cloud. vEdge routers 1, 2, and 3 have the following transport attachments:
- vEdge-1 has two local TLOCs:
- T11 with IP address 39.3.0.1 and marked with the biz-internet color;
- T12 with IP address 10.10.0.1 and marked with the mpls color;
- vEdge-2 has two local TLOCs:
- T21 with IP address 39.3.0.2 and marked with the biz-internet color;
- T22 with IP address 10.10.0.2 and marked with the mpls color;
- vEdge-3 has two local TLOCs:
- T31 with IP address 39.3.0.3 and marked with the biz-internet color;
- T32 with IP address 10.10.0.3 and marked with the mpls color.
Each vEdge router advertises two routes for subnet 10.1.1.0/24 in VPN 100 to the vSmart controller - one via its biz-interface TLOC and one via its mpls TLOC. Therefore, the vSmart controller receives six OMP routes for 10.1.1.0/24. All routes have identical values for AD (250), OMP Preference (0), TLOC preference (0), Origin-type (Connected), and Origin-Metric (1). Given this information, try to answer the following question, having in mind that everything else is by default and there is no policy applied on vSmart:
- Which routes will the vSmart controller choose as best for prefix 10.1.1.0/24?
- Which routes will the vSmart controller advertise to the other vEdge routers in the SD-WAN domain?
- Which ones will the controller advertise to the other vEdges if we change the default send-path-limit value to 5?
- Which routes for 10.1.1.0/24 will the other vEdges install in their forwarding table (FIB)?
The answers
At first, answering the questions seem very straightforward. However, many engineers overlook a key part of the OMP best-path selection process. The Smart controller uses the steps in the best-path algorithm to select the best routes to a destination. This is pretty simple and well understood. However, the thing that many don't realize is that after the best paths are selected, the vSmart controller uses the tiebreakers to sort the routes in descending order (from best to worst)! The controller then stores and keeps the best paths in its VPN routing table sorted in descending order. Then the send-path-limit comes into the picture. Suppose that the parameter is set to X - this tells vSmart to "take the first X routes and advertise them out".
Ok, having said that, let's go ahead and sort the routes shown in figure 1. As we have said, all routes have the same AD (250), OMP Preference (0), TLOC preference (0), Origin-type (Connected), and Origin-Metric (1) which means that according to the best-path algorithm, they are all best equal-cost routes to destination 10.1.1.0/24. However, the controller must sort routes before inserting them into the VPN 100 routing table. As of Cisco SD-WAN version 20.7.1, there are 3 tiebreakers:
- Tiebreaker (Source Preference) - Prefer vEdge sourced routes over vSmart sourced.
- Tiebreaker (System IP) - Prefer routes that have lower System-IP.
- Tiebreaker (Private TLOC IP) - For routes coming from the same vEdge, prefer the ones with a lower private TLOC IP address.
Let's use these tiebreakers and sort the routes manually. From the perspective of the vSmart controller, all six routes come from vEdges. Therefore, according to tiebreaker 1, they are all equal. Then using tiebreaker 2, we can work out that the routes via vEdge-1 would be better than the ones via vEdge-2 because vEdge-1 has a lower system-IP than vEdge-2. Logically, the routes via vEdge-3 would be worst (vEdge1(1.1.1.1) < vEdge2(2.2.2.2) < vEdge3(3.3.3.3)). Ok, but which one of the two routes of vEdge-1 to put at the top? For this, we must also use tiebreaker 3. We can work out that route "10.1.1.0/24 via T2" is better than "10.1.1.0/24 via T1" because T2(10.10.0.1) has a lower private TLOC IP address than T1(39.3.01). Applying this logic to all routes, we will end up with the order shown in figure 2 below:
We can verify that this is the correct order by checking the routing table for VPN 100 on vSmart. You can see that vEdge1's mpls route is at the top, vEdge1's biz-internet route is second, and so on.
vSmart# show omp routes vpn 100 | t
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
PATH ATTRIBUTE
VPN PREFIX FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR ENCAP PREFERENCE
--------------------------------------------------------------------------------------------------------------------------------------
100 10.1.1.0/24 1.1.1.1 66 1011 C,R installed 1.1.1.1 mpls ipsec -
1.1.1.1 68 1011 C,R installed 1.1.1.1 biz-internet ipsec -
2.2.2.2 66 1018 C,R installed 2.2.2.2 mpls ipsec -
2.2.2.2 68 1018 C,R installed 2.2.2.2 biz-internet ipsec -
3.3.3.3 66 1009 C,R installed 3.3.3.3 mpls ipsec -
3.3.3.3 68 1009 C,R installed 3.3.3.3 biz-internet ipsec -
Ok, after the best routes are selected and sorted, the vSmart controller takes the first 4 route (by default) and advertise them to the overlay fabric. This can be verified on any other vEdge router.
vEdge-6# sh omp route vpn 100 | t
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
PATH ATTRIBUTE
VPN PREFIX FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR ENCAP PREFERENCE
--------------------------------------------------------------------------------------------------------------------------------------
100 10.1.1.0/24 1.1.1.30 71 1011 C,I,R installed 1.1.1.1 mpls ipsec -
1.1.1.30 72 1011 C,I,R installed 1.1.1.1 biz-internet ipsec -
1.1.1.30 87 1018 C,I,R installed 2.2.2.2 mpls ipsec -
1.1.1.30 88 1018 C,I,R installed 2.2.2.2 biz-internet ipsec -
What is the OMP ecmp-limit?
Once we have understood what the send-path limit parameter does, let's go ahead and change it to a non-default value, for example, let's make it 5.
As expected, the vSmart controller will advertise the first five best routes. This can easily be verified on any of the other vEdges. However, notice another important thing. Even though vEdge6 now receives five routes for 10.1.1.0/24 in VPN100, the router only installs the first four in the routing table. (see the flag I)
vEdge-6# sh omp route vpn 100 | t
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
PATH ATTRIBUTE
VPN PREFIX FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR ENCAP PREFERENCE
--------------------------------------------------------------------------------------------------------------------------------------
100 10.1.1.0/24 1.1.1.30 71 1011 C,I,R installed 1.1.1.1 mpls ipsec -
1.1.1.30 72 1011 C,I,R installed 1.1.1.1 biz-internet ipsec -
1.1.1.30 87 1018 C,I,R installed 2.2.2.2 mpls ipsec -
1.1.1.30 88 1018 C,I,R installed 2.2.2.2 biz-internet ipsec -
1.1.1.30 96 1009 C,R installed 3.3.3.3 mpls ipsec -
We can verify this by checking the routing table for VPN 100.
vEdge-6# sh ip route vpn 100 10.1.1.0/24
Codes Proto-sub-type:
IA -> ospf-intra-area, IE -> ospf-inter-area,
E1 -> ospf-external1, E2 -> ospf-external2,
N1 -> ospf-nssa-external1, N2 -> ospf-nssa-external2,
e -> bgp-external, i -> bgp-internal
Codes Status flags:
F -> fib, S -> selected, I -> inactive,
B -> blackhole, R -> recursive, L -> import
PROTOCOL NEXTHOP NEXTHOP NEXTHOP
VPN PREFIX PROTOCOL SUB TYPE IF NAME ADDR VPN TLOC IP COLOR ENCAP STATUS
---------------------------------------------------------------------------------------------------------------------------------------------
100 10.1.1.0/24 omp - - - - 1.1.1.1 mpls ipsec F,S
100 10.1.1.0/24 omp - - - - 1.1.1.1 biz-internet ipsec F,S
100 10.1.1.0/24 omp - - - - 2.2.2.2 mpls ipsec F,S
100 10.1.1.0/24 omp - - - - 2.2.2.2 biz-internet ipsec F,S
What actually happened is illustrated in figure 3 below. Even though vEdge6 now receives five routes for 10.1.1.0/24 in VPN100, the number of routes that get installed in the routing tables is subject to another OMP parameter called ecmp-limit. By default, this parameter is also set to 4, which means that whatever number of best routes a vEdge may have, it will only install the best four in the routing table.
Let's change the ecmp-limit parameter and set it to the non-default value of 5.
vEdge-6# conf t
Entering configuration mode terminal
vEdge-6(config)# omp ecmp-limit 5
vEdge-6(config)# commit and-quit
Commit complete.
Now if we check the routing table, we can see that vEdge6 installs all five routes it receives.
vEdge-6# sh ip route vpn 100 10.1.1.0/24 | t
ADDRESS PATH PROTOCOL NEXTHOP NEXTHOP NEXTHOP
VPN FAMILY PREFIX ID PROTOCOL SUB TYPE METRIC IFNAME ADDR TLOC IP COLOR ENCAP VPN STATUS
------------------------------------------------------------------------------------------------------------------------------
100 ipv4 10.1.1.0/24 0 omp - 0 - - 1.1.1.1 mpls ipsec - F,S
100 ipv4 10.1.1.0/24 1 omp - 0 - - 1.1.1.1 biz-internet ipsec - F,S
100 ipv4 10.1.1.0/24 2 omp - 0 - - 2.2.2.2 mpls ipsec - F,S
100 ipv4 10.1.1.0/24 3 omp - 0 - - 2.2.2.2 biz-internet ipsec - F,S
100 ipv4 10.1.1.0/24 4 omp - 0 - - 3.3.3.3 mpls ipsec - F,S
Key takeaways
- The OMP best-path algorithm does not only select the best routes but also sorts them in descending order (from best to worst).
- The Smart controller always inserts and keeps all routes sorted, with the best route at the top.
- The send-path-limit parameter defines the maximum number of best-paths that can be advertised.
- The ecmp-limit parameter defines the maximum number of best paths that can be installed in the routing table.
- Bonus - The controller-send-path-limit defines the maximum number of best-paths that a vSmart controller can advertise to another vSmart controller.