Why do we need the overlay?

Traditional routing protocols build per-prefix routing tables with each routing entry pointing to a next-hop IP address. This typically means that each packet is forwarded hop-by-hop across the network according to the routing table of each individual router along the path to the destination. This hop-by-hop routing behavior has many inefficiencies such as:

• Network segmentation and network slicing are hard to achieve at a large scale:
  • Transporting segmentation tags hop-by-hop across the network requires complex control plane interactions between VRFs, MPLS, and MP-BGP.
  • Network slicing and multi-tenancy are practically unachievable.
• Scaling is hard. Horizontal scaling is even harder to achieve. Equal-cost multipathing (ECMP) over multiple types of WAN transports and multiple different routing protocols at a large scale is practically impossible. 
• Service-chaining does not efficiently scale because it requires manual configuration on multiple devices.
• Multicast does not natively traverse public transports such as the Internet.

The Cisco SD-WAN overlay fabric solves most of these inefficiencies by changing the traditional routing concept of a next-hop IP address with a next-hop TLOC. As vEdges build overlay tunnels between their WAN tunnel endpoints (TLOCs), they advertise the site-local networks as reachable via their local TLOCs. Using this technique, each destination prefix in the routing tables points to a remote tunnel endpoint (TLOC). When a packet is then forwarded according to the routing table, it is sent through an overlay tunnel to a remote TLOC, as shown in figure 1 above. The packet is encapsulated with new outer headers where the source and destination addresses are replaced with the tunnel endpoints. This way, the intermediate WAN networks between both vEdge routers do not need to know anything about the original packet’s source and destination IP. Additionally, the packet is encrypted with IPsec and labeled with a VPN tag that tells the remote router which network segment (VPN) this traffic belongs to. Compared to the traditional VRF/MPLS/MP-BGP approach to network segmentation, this is a considerable simplification and efficiency improvement. We will explore the overlay routing in great detail in the coming sections.

Cisco SD-WAN Underlay vs Overlay

Cisco SD-WAN architecture is divided into two very distinct parts: the underlay network and the overlay fabric.

The Underlay

The underlay network represents the hardware infrastructure - all network devices that connect to the available WAN transports and local site networks. The router interfaces that connect the WAN transport networks are always configured in VPN0 (the Transport VPN). The attachment points that connect to the transports are called TLOCs (colored in figure 2). TLOCs play a very important role in abstracting the underlay network away from the overlay fabric and the applications.  The main and only function of the underlay network is to provide IP reachability between TLOCs. 

A vEdge router must have at least one interface configured in the transport VPN 0 to establish control plane tunnels to the SD-WAN controllers and join the overlay fabric. Each interface that connects to the WAN must have an IP address, color, and encapsulation type configured. These parameters are then advertised to the controllers via OMP as part of the TLOC route advertisements. 

Most commonly, a default route is defined via each WAN interface, as illustrated in figure 2. Therefore, if a vEdge router has five WAN connections, we can configure five default routes 0.0.0.0/0 via each WAN link. This is a common point of confusion for network engineers. How does a router choose which default route to use at any time? 

In Cisco SD-WAN, when multiple default routes exist, the one that is chosen depends on the local TLOC that will be used. When the overlay routing decides to use a particular IPsec tunnel, the underlay routing uses the default route with a next-hop IP address in the same subnet as the tunnel source IP address. If we look at figure 3, for example, when vEgde1 decides to forward traffic over the orange IPsec tunnel, it uses the default route that points to a next-hop IP address in the same subnet as the source interface IP (ge0/0).

Additionally, all Cisco SD-WAN devices have a gateway tracking feature that is enabled by default and can’t be stopped or modified. Each device probes using ARP the next-hop IP of each underlay static route every 10 seconds. If the device receives an ARP response, it maintains the static route in the VPN0’s routing table. If the device misses ten consecutive ARP responses for a next-hop IP, the device removes the static route that points to this IP from the routing table. The device periodically probes the next-hop address, and as soon as it receives an ARP response again, the static route is installed back in the routing table.

The Overlay

Cisco's SD-WAN Overlay network is made of IPsec tunnels that traverse from site to site using the underlay network forming the so-called SD-WAN Fabric. Each overlay tunnel is formed between two TLOCs. The routing within the overlay is governed by the Overlay Management Protocol (OMP), a control-plane protocol very similar to BGP. The OMP protocol runs over secure DTLS or TLS connections between the WAN edge routers and the vSmart controllers. The process is very similar to the BGP operation, the vSmart controller acts as a BGP route reflector (RR), it receives, modifies, and re-advertises routes from the vEdge routers, but never participate in the data-plane (in the packet forwarding).

Network Segmentation

Abstracting the packet forwarding away from the network and application logic opens a world of possibilities. This allows for the use of VPNs that divide the overlay network into different network segments. Essentially, segmentation is done at the WAN edge routers, and the segmentation information is carried as a VPN label in the packets. However, the underlay network (Transport VPN0), that connects the WAN edge routers to the WAN transport, is completely unaware of the network segments (VPNs). Only the overlay knows about the VPNs; the underlay network follows the standard IP routing.

Key Takeaways

Let's try to summarize the difference between Cisco SD-WAN's Underlay vs Overlay in one table: