This lesson will demonstrate a scenario with several requirements that we will solve using an Application-aware Routing policy in combination with a few additional SD-WAN features.

Scenario Overview

An organization has a hub-and-spoke SD-WAN network where spokes have three local colors - mpls, biz-internet, and lte and the hub has two local colors - mpls and biz-internet. Figure 1 shows an example where vEdge-3 is a spoke and vEdge-1 is the hub.

The LTE color (in yellow) is via a radio module using metered SIM card with limited data usage. 

Scenario Requirements

You are tasked to configure the network in such a way so that it satisfies the following requirements:

• Since the LTE is on metered SIM card, in normal circumstances, spokes must not establish overlay tunnels over the LTE color.
• The LTE color must only be used when both mpls and biz-internet colors are unavailable.
• MySQL traffic (TCP/3306) in VPN5 must go through the overlay tunnel that meets the following SLA requirements:
  • Loss < 1%
  • Latency < 100ms
  • If both the mpls and biz-internet colors meet SLA, MySQL traffic must be ECMP-ed over both.
  • If both colors are up but do not meet SLA, MySQL traffic must go through the best-of-the-worst (BoW) color based on latency and latency-variance of 5ms.
• All other traffic in VPN5 must go through the overlay tunnel that meets the following SLA requirements:
  • Loss < 2%
  • Latency < 150ms
  • If both colors meet SLA, the traffic must be ECMP-ed over both colors.
• When both mpls and biz-internet colors are down and spokes establish an overlay tunnel through the LTE color, only MySQL traffic must be permitted to go via that tunnel.

Initial Topology

For this advanced lab lesson, we will use the topology and the initial configs shown on the section page. However, we will only work with two WAN edge routers - vEdge-3 representing a spoke and vEdge-1 representing a hub.

Notice that at the moment, vEdge-3 has established overlay tunnels over its three local colors (mpls, biz-internet and lte).

vEdge-3# show bfd sessions | t 

SYSTEM   SITE                                     DETECT      TX                  
IP       ID    LOCAL COLOR   COLOR         STATE  MULTIPLIER  INTERVAL  UPTIME    
----------------------------------------------------------------------------------
1.1.1.1  1     mpls          mpls          up     7           1000      0:00:05:44
1.1.1.1  1     biz-internet  biz-internet  up     7           1000      0:00:17:54
1.1.1.1  1     lte           biz-internet  up     7           1000      0:00:05:16

Everything else is by default. There is no policy applied on vSmart and the overlay routing is unchanged.

Okay, you can stop at this point and try to solve the requirements by yourself. Think of how you would approach each requirement and what technologies you would use. Try to imagine all failure scenarios and how your solution fits in. Then return to the lesson and check our solution.

Notice that multiple solutions may satisfy the scenario requirements. If you have found a better solution, please let us know in the comments, so we can update the lesson for others to know.

Scenario Solution

We can break down the requirements into three main tasks:

• Task #1 - Ensuring that LTE is only used when all other overlay tunnels are unavailable.
• Task #2 - Ensuring that the traffic in VPN5 goes through tunnels that meet SLA.
• Task #3 - Ensuring that only MySQL traffic goes through the LTE tunnel.

Task #1 - Last Resort Circuit

Typically, organizations use cellular networks (3G/4G/LTE/5G, etc.) as a backup WAN, to provide network resiliency in case all overlay tunnels over wired WAN links become unavailable.

Cisco SD-WAN has a feature called circuit of last resort that activates a local tloc only if all overlay tunnels fail. The local tloc does not maintain any control or data connections in this mode until it gets activated.

Configuring a local tloc as last-resort-circuit is as simple as applying one command via CLI or via Device Template. Let's configure the spoke's lte color as a circuit of last resort and then check the overlay tunnels.

vEdge-3# conf t
Entering configuration mode terminal
vEdge-3(config)# 
vEdge-3(config)# vpn 0 interface ge0/2 tunnel-interface 
vEdge-3(config-tunnel-interface)#last-resort-circuit 
vEdge-3(config-tunnel-interface)# commit and-quit
Commit complete.

Now if we check the spoke's overlay tunnels, we will see that there is no tunnel over the LTE color.

vEdge-3# show bfd sessions | t

SYSTEM   SITE                                     DETECT      TX                  
IP       ID    LOCAL COLOR   COLOR         STATE  MULTIPLIER  INTERVAL  UPTIME    
----------------------------------------------------------------------------------
1.1.1.1  1     mpls          mpls          up     7           1000      0:00:05:44
1.1.1.1  1     biz-internet  biz-internet  up     7           1000      0:00:17:54

Notice that when we configured the last-resort-circuit command on the LTE interface, it triggers "Control TLOC Down" and "BFD TLOC Down" alarms on the router. These two alarms will only be cleared when all other overlay tunnels become unavailable. This is the expected behavior.

Task #2 - Application-aware Routing (App-route)

The second task says that the traffic in VPN 5 must go through the tunnel that meets the following SLA requirements:

• MySQL traffic (TCP/3306) must go through the overlay tunnel meeting the following SLA:
  • Loss < 1%
  • Latency < 100ms
  • If both the mpls and biz-internet colors meet SLA, MySQL traffic must be ECMP-ed over both.
  • If both colors are up but do not meet SLA, MySQL traffic must go through the best-of-the-worst (BoW) color based on latency and latency-variance of 5ms.
• All other traffic in VPN5 must go through the overlay tunnel that meets the following SLA requirements:
  • Loss < 2%
  • Latency < 150ms
  • If both colors meet SLA, the traffic must be ECMP-ed over both colors.

Obviously, we will need to configure a new Application-aware Routing (App-route) Policy and apply it to both the spoke and the hub for VPN 5. It is also obvious that we will need two sla-classes - one for MySQL traffic and one for all other traffic. With this in mind, let's go ahead and define the policy lists and the sla-classes that we are going to need. 

SLA-class

Let's quickly define the policy lists that we will need when configuring the App-Route policy. We will need a vpn-list that matches VPN 5 and two site lists - one for the hub and one for the spokes.

policy
 lists
  vpn-list VPN5 
   vpn 5
  !
  site-list HUB
   site-id 1
  !
  site-list SPOKES
   site-id 3-6
  !
 !
!

Then we will need two different sla-classes - one for the MySQL traffic and one for all other traffic. Let's configure them as per the scenario requirement.

​policy
 sla-class MYSQL
  loss    1
  latency 100
  fallback-best-tunnel
   criteria         latency
   latency-variance 5
  !
 sla-class CLASS-DEFAULT
  loss    2
  latency 150
 !

Best-of-the-worst (BoW) tunnel selection

WAN edge routers can use a technique called best-of-the-worst (BoW) tunnel selection to find the best tunnel when none meets the SLA requirements. The technique is pretty straightforward, we specify criteria in the SLA-class configuration. When none of the tunnels meet SLA,  the router just compares all available local colors and chooses the one that has a lower value based on the specified criteria.

For example, we have specified that we want to use the BoW tunnel based on latency. When none meets SLA, the router will select the tunnel with the lowest latency.

What is Variance?

BoW could lead to instabilities when the overlay tunnels have very small margins in the specified criteria. For example, assume that the required SLA latency is 100 ms. Let's say Tunnel-1 has 121ms and Tunnel-2 has 120ms. As per the best-of-the-worst logic, Tunnel-2 is the best because it has lower latency. However, at the next quality measurement interval Tunnel-1 has 121ms and Tunnel-2 has 122ms. Then Tunnel-2 becomes best as per the BoW logic and so on. The BoW best tunnel changes at every quality measurement interval because the latency constantly changes with a slight deviation in ms, as shown in Figure 4 below. This could lead to reprogramming of the overlay path at every quality measurement interval.

To mitigate this problem, Cisco SD-WAN introduced the concept of Variance. Variance is a delta value that dampens slight deviations in the specified BoW criteria. If the difference between two tunnels' criteria is less than the variance value, the BoW algorithm considers them equal. For example, if Tunnel-1 has a latency of 120ms and Tunnel-2 has a latency of 121ms and the variance value is 5ms - the BoW logic considers them equal because the difference in latency of 1ms is less than the variance value of 5ms.

To understand the variance logic, consider the following examples.

Example 1 
T1 latency 120ms, T2 latency 121ms, Variance 5ms
121ms - 120ms < 5ms (difference btw T1 and T2 is less than the variance)
BoW best tunnel - T1 and T2

Example 2
T1 latency 120ms, T2 latency 128ms, Variance 5ms
128ms - 120ms > 5ms (difference btw T1 and T2 is more than the variance)
BoW best tunnel - T1

Example 3
T1 latency 130ms, T2 latency 120ms, Variance 5ms
130ms - 120ms > 5ms (difference btw T1 and T2 is more than the variance)
BoW best tunnel - T2

Example 4
T1 latency 122ms, T2 latency 121ms, Variance 5ms
122ms - 121ms < 5ms (difference btw T1 and T2 is less than the variance)
BoW best tunnel - T1 and T2

App-route Policy

Now that we have configured the policy lists and the sla-classes, let's go ahead and configure the application-aware routing policy for both traffic classes.