We have seen multiple times by now that the Cisco SD-WAN architecture decouples the control and management-plane functions away from WAN edge routers. The solution implements these functions into centralized software-based controllers that oversee the entire network environment. Therefore, from the perspective of a vEdge router, the control and management plane functions are remote services that are communicated to the device. That is why each vEdge router must establish and maintain secure control connections to each Cisco SD-WAN controller, as is visualized in figure 1. 

The SD-WAN controllers are software entities that run as virtual machines or containers, hosted either on-prem or in a public cloud provider. Each controller plays a different role in the solution and interacts with the vEdge routers through various protocols. For example, vSmart uses the Cisco Overlay Management Protocol (OMP) to communicate routing, TLOC, and service information among vEdges. vManage uses NETCONF to push configurations to devices, SNMP to collect data, and ICMP echo/reply to detect liveliness. 

To meet today’s high standards for availability and security, each SD-WAN controller interacts with the WAN edge devices in a secure, resilient, and timely manner. However, when multiple control protocols are running in a network, ensuring that each one is compliant with the latest security policies is a complex task that quickly gets out of hand. For example, let’s say we have a traditional WAN environment running BGP, OSPF, EIGRP, SNMP, and PIM. How do we make sure that each protocol is secured? Each one implements key exchanges differently, supports different encryption and authentication algorithms, and is configured differently. That is why Cisco SD-WAN has taken a different, more scalable approach when it comes to securing the control plane protocols. Every WAN edge router establishes one or more secured control tunnels to each SD-WAN controller. The tunnels are secured using a standard transport security protocol (DTLS or TLS). 

Then each control plane protocol, such as OMP, NETCONF, SNMP, etc., goes through one of the DTLS/TLS secured control tunnels, as is visualized in figure 2. This technique ensures that the native security of these protocols (OMP, SNMP, NETCONF, etc.) is no longer a concern to the overall security because they are communicated through secured tunnels that are encrypted and authenticated. 

DTLS vs TLS

Cisco SD-WAN supports two transport layer security protocols that vEdges use for control-plane tunnels: DTLS (Datagram Transport Layer Security) defined in RFC 6347 and TLS (Transport Layer Security) defined in RFC 5246. Both protocols are very similar, and from a high level, they both serve the same goal - to provide end-to-end transport security between a router and an SD-WAN controller. The main difference is that DTLS uses UDP and TLS runs over TCP. This means that they handle packet loss differently. TLS primarily relies on TCP for packet delivery. On the other hand, DTLS implements its own sequence numbers, fragment offsets, and retransmissions because UDP does not guarantee reliable delivery of packets. In the end, TLS is slightly more secure and reliable but slower. DTLS is marginally faster and more efficient but less secure. However, the differences are so slight that they are pretty much the same in practice.

By default, Cisco SD-WAN utilizes DTLS for all control plane communications because it is faster than TLS, and latency is essential when devices are managed remotely. However, the solution allows us to change the protocol to TLS easily.

Control Connections to vBond

We will explore the various control connections that a WAN edge router establishes, starting with the vBond controller representing the orchestration plane.

Once we power on a vEdge router for the first time in an unconfigured state, it tries to find the IP address of vBond. The vBond orchestrator is the controller that glues everything together - that’s why it is called the “orchestrator.” vBond validates the identity of vEdges, tells them whether they sit behind NAT in the underlay, how many vSmart controllers oversee the domain, and most importantly - what are the IP addresses of vManage and vSmart. 
In simple words, an unconfigured vEdge router needs to contact vBond to receive the necessary information required to join the SD-WAN domain. However, how could a brand new unconfigured router know the IP address of the vBond controller for a particular organization? Well, there are a few different deployment options that routers can utilize to discover the orchestrator’s IP:

• ZTP/PnP - A vEdge router can automatically discover the vBond IP address over the Internet using a technique called Cisco PnP (Plug-and-Play) for IOS-XE devices and ZTP (Zero-touch provisioning) for Viptela SD-WAN devices. At a very high level, the router simply makes a query to devicehelper.cisco.com or ztp.viptela.com over the Internet, presents its serial/chassis number and certificate, and asks for the vBond IP address associated with its serial number.
• Bootstrap configuration - As an alternative option, a vEdge router can load a configuration file from a USB stick. The config file includes the vBond IP address, organization name, and everything else required so that the vEdge can successfully join the overlay fabric.
• Manually - And of course, the time-proven method that always works - a network administrator can configure everything through the CLI of the router, including the vBond IP address and the organization name.

We will go through each deployment option in detail in the SD-WAN Deployment chapter of this course. For now, let’s see what happens after a vEdge router finds the vBond orchestrator’s IP address. 

The vEdge router tries to establish a DTLS control-plane tunnel to the vBond IP address over each available transport interface. In the example shown in figure 3, vEdge-1 will try to form a DTLS control connection to the orchestrator over the Internet and over the MPLS cloud. It will authenticate itself over each control tunnel and find whether it sits behind a NAT device along any of the paths. In the end, vBond will send back a list with the IP addresses and ports of all other SD-WAN controllers (vSmart and vManage). Both DTLS control connections are then terminated because the vEdge router does not need to keep permanent control connections to vBond once it receives all necessary information. 

At this time, the vBond orchestrator will inform vManage and vSmart to expect a control connection request from that authenticated WAN Edge router. 

Control Connections to vManage

Once a vEdge router receives the list with the IP addresses of all SD-WAN controllers from vBond, it initiates a single, persistent DTLS control connection to vManage over only one transport. The vEdge tries every transport network starting with the interface having the lowest port number, and it keeps the first successfully established connection permanently. 

If we look at figure 4, all vEdges have two transport connections, one to the Internet and one to the MPLS cloud. However, DTLS tunnel to vManage is established only via the Internet. If a vEdge lost connectivity to vManage over this DTLS connection, it would form a new one through the other transport interface connected to the MPLS cloud.

When a vEdge router establishes a control connection to vManage, the controller provisions its configuration based on the device template attached to that vEdge. 

Control Connections to vSmart

Following the successful DTLS tunnel to vManage, the vEdge router initiates a persistent control connection to the vSmart controller over each transport interface. You can see in the example shown in figure 5 that all vEdges establish a DTLS tunnel to vSmart through the Internet and another one through the MPLS cloud at the same time. Technically, a single control connection to the controller is sufficient for a vEdge to receive control plane information. Still, vEdges keep one permanent DTLS tunnel per transport interface for control-plane resiliency.