This lesson will explore a Cisco SD-WAN functionality called Dynamic On-demand Tunnels. We will see when this feature could come in handy when designing the overlay topology and how we configure on-demand tunnels.

The Business Need

It is very common for organizations to deploy IPsec tunnels in a full-mesh SD-WAN overlay to provide a low-latency path to voice and video services. In most cases, this leads to many idle tunnels between remote sites that rarely or never communicate with each other. Additionally, when an organization uses less-powerful WAN edge platforms such as Cisco ISR 1100 series on spoke sites, the full-mesh topology becomes a scaling limitation because, at some point, the routers can't handle the thousands of IPsec tunnels required.

There are three solutions to this scaling problem:

1. Changing the topology to hub-and-spoke: This is the easiest way to scale up the overlay fabric. However, spokes aren't able to communicate with other spokes via low-latency direct tunnels (all traffic goes through the hub).
2. Manually adjusting the topology to custom partial mesh: This option would require a complex control-plane policy and won't scale efficiently. Additionally, it is hard to predict which spokes communicate with each other in many scenarios, so it is very hard to determine which sites to set up direct tunnels between.
3. Using dynamic on-demand tunnels: The most efficient option to both scale up the overlay and keep having a direct low-latency path between remote sites is by using on-demand tunnels. The topology is hub-and-spoke, but spoke-to-spoke traffic triggers the WAN edge routers to set up a temporary direct IPsec tunnel between the two sites. (similarly to DMVPN Phase 3)

Figure 1 illustrates a typical use-case of the Cisco SD-WAN Dynamic On-demand Tunnels.

What are On-demand Tunnels?

On-demand Tunnels are a Cisco SD-WAN functionality that triggers the establishment of an IPsec overlay tunnel between two sites only when there is traffic between the WAN edger routers. After the flow of direct traffic stops and the pre-defined idle-timer expires, the tunnel is placed in an inactive state. In this state, the tunnel doesn't consume bandwidth or CPU and therefore does not affect the router's performance. However, the on-demand tunnel can quickly be brought back up again when direct traffic re-occurs.

In summary, dynamic spoke-to-spoke tunnels allow organizations to use less expensive, less powerful WAN edge routers and, at the same time, be able to scale up the overlay and use direct low-latency paths between remote sites.

How do On-demand Tunnels work?

We could break the process of enabling this functionality within an SD-WAN overlay into four distinct configuration steps:

The control-plane

The first step in setting up on-demand tunnels is to configure the control plane properly. There are two essential pre-requisites when enabling the functionality in Cisco SD-WAN:

1. All spoke sites must receive all TLOCs and vroutes of all other spokes. This practically means that the initial topology (before enabling on-demand functionality) must be a full-mesh and not hub-and-spoke.
2. All spoke vRoutes must have a backup path set up through tloc-action in a centralized control policy, as shown in the output below. The ultimate tloc represents the potential direct path to reach this prefix directly.

Figure 2 illustrates both pre-requisites in a simple example. Notice that the topology is full-mesh (there are tunnels between vEdge-4 and vEdge-5) and also that spokes have a backup path for the prefixes of other spokes (ultimate tloc with tloc-action backup).

The output below shows an example of a vRoute that has a backup next-hop enforced via a centralized control policy. The next-hop tloc points to the HUB. The ultimate tloc points directly to the remote spoke and represents the direct data plane path to the destination prefix.

---------------------------------------------------
omp route entries for vpn 4 route 10.5.4.0/24
---------------------------------------------------
            RECEIVED FROM:                   
peer            1.1.1.30
path-id         53
label           1010
status          C,I,R
loss-reason     not set
lost-to-peer    not set
lost-to-path-id not set
    Attributes:
     originator       5.5.5.5
     type             installed
     tloc             1.1.1.1, mpls, ipsec
     ultimate-tloc    5.5.5.5, mpls, ipsec -- backup
     domain-id        not set
     overlay-id        1
     site-id          5
     preference       not set
     tag              not set
     origin-proto     connected
     origin-metric    0
     as-path          not set
     community        not set
     unknown-attr-len not set

We will see how to configure the control plane in the configuration portion of this lesson.

The data-plane

Okay, we said that the initial topology before implementing the on-demand functionality must be full-mesh (all spokes knowing all TLOCs). But we want to have a hub-and-spoke topology with dynamic spoke-to-spoke tunnels, right? Well, how's that going to happen?

The magic happens in the data plane - once enabled for on-demand, WAN edge routers stop forming tunnels to other routers that are also enabled with the on-demand functionality, as shown in figure 3 above. Hence, if all spokes are enabled with on-demand and the hubs are not, the topology becomes hub-and-spoke because spokes won't form tunnels to other spokes (all are configured with on-demand) but at the same time spokes will form tunnels with the hub (hubs are not enabled for on-demand). 

Triggering the on-demand tunnel

The initial traffic between two spoke sites is routed through the static next-hop tloc in the vroute which is pointing to the hub. This forces the traffic to go through the hub.

When a vEdge router is enabled for on-demand tunnels and receives traffic from a prefix that has an ultimate-tloc-backup, it responds back via the hub and simultaneously triggers the provisioning of direct on-demand tunnels to the site sourcing the traffic (not to the individual vEdge!). This is very important - in dual-homed spoke sites, when traffic triggers the on-demand functionality, direct tunnels are formed to all WAN edge routers on the souring spoke site.

For spoke sites with multiple WAN edge routers, it is very important to configure the on-demand functionality on all routers; otherwise, the on-demand feature may behave unpredictably.

Tunnel States

Once traffic triggers the setting up on-demand tunnels on both spoke sites, the vEdge routers start to monitor the state of the tunnels.

An on-demand tunnel has two states:

• Active - Once an on-demand tunnel is established, it is placed in an active state. This means that traffic is actively traversing the direct tunnel and BFD is up. In this state, the tunnel activity is constantly observed. When the traffic stops flowing, a pre-defined idle-timer starts (default one is ten minutes). When this timer expires, the tunnel is placed in an Inactive state.
• Inactive - When an on-demand tunnel is placed in an inactive state, it basically means that the tunnel is removed and no BFD probes are being sent for detection. Inactive tunnels do not use any bandwidth or CPU resources of the router.

On-demand Tunnels Benefits

Establishing dynamic spoke-to-spoke tunnels offers many performance advantages over the pure-vanilla hub-and-spoke topology:

• Significantly improved performance in scenarios with less-powerful routers working in a full-mesh overlay.
• Significantly improved latency between spokes.
• Reduced bandwidth usage.
• Reduced CPU and memory usage.

Configuring On-demand Tunnels

Let's now jump into the configuration portion of this lesson and see how we can enable spoke-to-spoke tunnels in our lab topology.

The initial state

For this lab example, we will use the topology shown in figure 6 below. For the purpose of this example, I have reverted the topology back to full-mesh and deleted all centralized control policies applied in previous lessons. The overlay topology is just a default full-mesh SD-WAN fabric.

To verify this, let's check that each router has overlay tunnels to all other sites (full-mesh).

vEdge-4# show bfd sessions | t

SYSTEM   SITE                                    DETECT     TX                 
IP       ID   LOCAL COLOR   COLOR         STATE  MULTIPLIER INTERVAL UPTIME    
-------------------------------------------------------------------------------
1.1.1.1  1    mpls          mpls          up     7          1000     0:00:00:14
1.1.1.1  1    biz-internet  biz-internet  up     7          1000     0:00:00:19
1.1.1.2  1    mpls          mpls          up     7          1000     0:00:00:14
1.1.1.2  1    biz-internet  biz-internet  up     7          1000     0:00:00:19
3.3.3.3  3    mpls          mpls          up     7          1000     0:00:00:14
3.3.3.3  3    biz-internet  biz-internet  up     7          1000     0:00:00:19
5.5.5.5  5    mpls          mpls          up     7          1000     0:00:00:14
5.5.5.5  5    biz-internet  biz-internet  up     7          1000     0:00:00:18
6.6.6.6  6    mpls          mpls          up     7          2000     0:00:00:14
6.6.6.6  6    biz-internet  biz-internet  up     7          1000     0:00:00:19

Additionally, each prefix is reachable via the direct overlay path to the respective remote site. Therefore, the overlay routing is also "as it is by default".

vEdge-4# show ip routes vpn 4 | t

     ADDRESS               PATH             NEXTHOP                             
VPN  FAMILY   PREFIX       ID    PROTOCOL   IFNAME  TLOC IP  COLOR         ENCAP
--------------------------------------------------------------------------------
4    ipv4     10.1.4.0/24  0     omp        -       1.1.1.1  mpls          ipsec
4    ipv4     10.1.4.0/24  1     omp        -       1.1.1.1  biz-internet  ipsec
4    ipv4     10.1.4.0/24  2     omp        -       1.1.1.2  mpls          ipsec
4    ipv4     10.1.4.0/24  3     omp        -       1.1.1.2  biz-internet  ipsec
4    ipv4     10.3.4.0/24  0     omp        -       3.3.3.3  mpls          ipsec
4    ipv4     10.3.4.0/24  1     omp        -       3.3.3.3  biz-internet  ipsec
4    ipv4     10.4.4.0/24  0     connected  ge0/4   -        -             -    
4    ipv4     10.5.4.0/24  0     omp        -       5.5.5.5  mpls          ipsec
4    ipv4     10.5.4.0/24  1     omp        -       5.5.5.5  biz-internet  ipsec
4    ipv4     10.6.4.0/24  0     omp        -       6.6.6.6  mpls          ipsec
4    ipv4     10.6.4.0/24  1     omp        -       6.6.6.6  biz-internet  ipsec

There is nothing fancy about the initial topology, so let's go ahead and demonstrate the Cisco SD-WAB on-demand functionality.

Step 1: Configuring the control plane

The first step in enabling the feature is to configure the control plane. We will need to provision a new centralized control policy that includes the tloc-action backup. This will, later on, ensure that spokes know that a direct path to the destination prefix exists so they can trigger the establishment of on-demand tunnels.