In this lesson, we will explore the process of onboarding cEdge routers (CSR1k) to the Cisco SD-WAN fabric using Enterprise CA and manual bootstrap configuration. This lesson complements our two previous articles on setting up a Cisco SD-WAN home lab for practicing.

Initial Topology

Figure 1 shows the physical topology that we will use in this example. All cEdge routers are CSR 1000v running Cisco IOS-XE version 16.12.04. We won't go into the process of bringing up the SD-WAN controllers in this lesson and will jump straight into the cEdge onboarding process. If you want to understand how to deploy the Cisco SD-WAN controllers using local Enterprise CA, check out this lesson.

Notice that the example will only show how to onboard cEdge-1. If you want to set up the entire topology, you will need to repeat this for each WAN edge device.

Onboarding cEdge routers

Prepare the software image

When the router boots up, we first stop the PnP service so that the SD-WAN software packages can install. We do this using the following command in exec mode.

cEdge# pnpa service discovery stop

Once the PnP service has been stopped, we tell the router to install all underlying SD-WAN packages if necessary. Depending on the CSR1k software image, this may not be necessary. However, in practice environments, it could only help.

cEdge# request platform software sdwan software reset

The last step is to verify the software image using the following command.

cEdge# request platform software sdwan software upgrade-confirm

You should see that the sdwan software is ACTIVE and CONFIRMED as highlighted below.

cEdge# show sdwan soft
VERSION         ACTIVE  DEFAULT  PREVIOUS  CONFIRMED  TIMESTAMP                  
---------------------------------------------------------------------------------
16.12.4.0.4480  true    true     false     user       2022-04-03T08:20:13-00:00  

Total Space:388M Used Space:87M Available Space:297M

Bootstrap the cEdge router

Once the router loads up with the SD-WAN software, we can go ahead and configure the minimal configuration required to join the SD-WAN overlay fabric. Notice that when the cEdge router runs in Controller mode (basically SD-WAN mode), we enter the configuration mode using the "config-transaction" command instead of the well-known "configure terminal" or simply "conf t".

In the following example, we configure the basic IP addressing and default routing alongside the essential system parameters such as system-IP, site-id, and vBond address. In this example, we will configure a DNS name for vBond, as is recommended by Cisco.

cEdge# config-transaction
 hostname cEdge
 !
  int GigabitEthernet1
  ip address 39.3.1.1 255.255.255.0
  no shut
 !
 int GigabitEthernet2
  ip address 10.10.1.1 255.255.255.0
  no shut
 !
  ip route 0.0.0.0 0.0.0.0 39.3.1.254
  ip route 0.0.0.0 0.0.0.0 10.10.1.254	
  ip host vbond.networkacademy.io 10.1.1.10
 !
 system
  system-ip 1.1.1.1
  site-id 1
  organization-name "networkacademy-io"
  vbond vbond.networkacademy.io
 commit

At this point, you should be able to ping all Cisco SD-WAN controllers from the cEdge router that is being onboarded. If there is no IP connectivity between the WAN edge router and the controllers, there is no point in continuing further. You should troubleshoot the problem first.

If there is IP reachability between the cEdge router and the controllers, we are ready to configure the SD-WAN overlay tunnels. Notice something very important - the Tunnel keyword in the "interface Tunnel" command should always be with a capital T. It is not like in a regular Cisco IOS where you can create a new tunnel using the "interface tunnel 1" command.

 sdwan
  int GigabitEthernet1
  tunnel-interface
   color biz-internet
   encapsulation ipsec
 !
  int GigabitEthernet2
  tunnel-interface
   color mpls restrict
   encapsulation ipsec
 !
 interface Tunnel 1
  ip unnumbered GigabitEthernet1
  tunnel source GigabitEthernet1
  tunnel mode sdwan
!
 interface Tunnel 2
  ip unnumbered GigabitEthernet2
  tunnel source GigabitEthernet2
  tunnel mode sdwan
 commit

At this point, the CSR1K router has the required configuration to join the SD-WAN fabric. The next step is to install the root certificate and activate the routers.

Installing the Root CA Certificate

Now it is time to install the root CA certificate to the cEdge router. In our previous lessons, we have deployed the Cisco SD-WAN controllers using the Enterprise CA method with the vBond controller acting as a Root CA. If you have deployed the controllers following one of our lessons, you should have the Root CA certificate on vBond named ROOTCA.pem. The easiest way to install the root certificate on a cEdge CSR1K router is by creating a local file directly on the router using TCLSH, as shown in the following example. In the highlighted section, you should paste the ROOTCA.pem certificate which you take from vBond using the "cat ROOTCA.pem" command in vshell mode.

cEdge# tclsh 
cEdge(tcl)# puts [open "flash:ROOTCA.pem" w+] {
+> paste root-cert-here
+> }
cEdge-1(tcl)# exit

In the end, you should have the root certificate in the cEdge router's bootflash, as shown below.

cEdge# more bootflash:ROOTCA.pem
-----BEGIN CERTIFICATE-----
MIIDjzCCAnegAwIBAgITHFuqFiCbYL/pMA0GCSqGSIb3DQEBCwUAMF4xCzAJBgNV
BAYTAlVTMQswCQYDVQQIDAJOWTELMAkGA1UEBwwCTlkxGjAYBgNVBAoMEW5ldHdv
cmthY2FkZW15LWlvMRkwFwYDVQQDDBByb290LmNlcnRpZmljYXRlMB4XDTIyMDQw
MzA3NTIzN1oXDTI1MTFHMTA3NTIzN1owXjELMAkGA1UEBhMCVVMxCzAJBgNVBAgM
Ak5ZMQswCQYDVQQHDAJOWTEaMBgGA1UECgwRbmV0d29ya2FjYWRlbXktaW8xGTAX
BgNVBAMMEHJvb3QuY2VydGlmaWNhdGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQDOw7rS5HFTyuXBrsPISA9i8f5yR+gFDFeBCWlnXy6q1ztKQT5T+KtK
WoSVmAJTU9kAug1ACq4n/KSRXBxZu/YXzKF6qH8E26UBt6xofSAJBf3pIg3eIxjk
KY1+sSTxjUIg601GFMaGO0fYBQyj9xBJmIuiHWJOd0GHf6P0M48HJt39YJtKiV48
J1K20Arke5qPGNw86coRhFs//v/wqQesHobEGHuObCSZU73qTIrj92CjYiJ2gXID
PO+nKHNGhO3osGe9GFBQe1KFRsVbp7w/Nt8nrQUT2eW4/uie2YCd6Aw6e5A9pAWY
ob6QOzstw7TZ3AlLJ6uAbWMmdIlWhFO3AgMBAAGjUDBOMB0GA1UdDgQWBBRdh01N
M09x6ZVahuBFw498caqeUTAfBgNVHSMEGDAWgBRdh01NM09x6ZVahuBFw498caqe
UTAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQB+/HruVYFQr9suYBX8
WtyS8/SWSkXrjz4jQSYPGLZSSuKFUhaVOkV/Vld3oewAzEvGXamXwWJKRNFiJrSm
qiL1t18DrYQZ3PkqMbsrmjSCoxmY8lTFWEnWUoETmevgDDGA1y68n91yPRyQBpIZ
h5o66e4ZH7LHRxgDWRWOWjNSIrvQzKPQrwYg9p9yfUKboo0su5tuJBnyYV5s6GTI
JxeMfP8GBtwB6JcecE7EdnGI7gZh4bWJwGCBO4lZgUK8CjxAnMfR1mNrDHZhOqES
pK9s66SjJM5PAD5tJ4Qz0NWPJu5F1IzHXa3skAK3vHkgTixDocaIERh+DFYeDwkm
/dfY
-----END CERTIFICATE-----

The last step is to install the root certificate using the following command.

cEdge# request platform software sdwan root-cert-chain install bootflash:ROOTCA.pem

Uploading root-ca-cert-chain via VPN 0
Copying ... /bootflash/ROOTCA.pem via VPN 0
Updating the root certificate chain..
Successfully installed the root certificate chain

If everything has gone smoothly, you should see our custom root certificate installed on the router.

cEdge#show sdwan certificate root-ca-cert | in network
        Issuer: C=US, ST=NY, L=NY, O=networkacademy-io, CN=root.certificate
        Subject: C=US, ST=NY, L=NY, O=networkacademy-io, CN=root.certificate
		

Activating the cEdge router

Before the cEdge router can be able to join the SD-WAN fabric, it must have a device certificate, signed and installed by vManage. To do this, we need to have an available unused chassis number and token in vManage under Configuration > Certificates. We use them to activate the cEdge router as shown in the example below.

cEdge# request platform software sdwan vedge_cloud activate chassis-number CSR-D4B9356B-XXXX-XXXX-XXXX-XXXXXXXXXXXX token af6bd685d8674996xxxxxxxxxxxxxxxx

Once you've done, you should see in the logs that vManage logs into the cEdge using NETCONF over SSH, generates a CSR, then signs it and install a device certificate. Then the cEdge router should establish an OMP peering with vSmart and start receiving TLOCs and OMP routes.

cEdge#
*Apr  3 09:38:11.896: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User 'vmanage-admin' authenticated successfully from 1.1.1.20:41146 and was authorized for netconf over ssh. External groups:
*Apr  3 09:38:19.308: %Cisco-SDWAN-cEdge-2-action_notifier-6-INFO-1400002: R0/0: VCONFD_NOTIFIER: Notification: 4/3/2022 9:38:19 security-install-csr severity-level:minor host-name:default system-ip:1.1.1.1
*Apr  3 09:38:20.525: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User 'vmanage-admin' authenticated successfully from 1.1.1.20:41158 and was authorized for netconf over ssh. External groups:
*Apr  3 09:38:26.902: %Cisco-SDWAN-cEdge-2-action_notifier-6-INFO-1400002: R0/0: VCONFD_NOTIFIER: Notification: 4/3/2022 9:38:26 security-install-rcc severity-level:minor host-name:default system-ip:1.1.1.1
*Apr  3 09:38:45.329: %Cisco-SDWAN-cEdge-2-action_notifier-6-INFO-1400002: R0/0: VCONFD_NOTIFIER: Notification: 4/3/2022 9:38:45 security-install-certificate severity-level:minor host-name:default system-ip:1.1.1.1

*Apr  3 09:39:22.177: %Cisco-SDWAN-RP_0-OMPD-6-INFO-400002: vSmart peer 1.1.1.30 state changed to Init
*Apr  3 09:39:24.257: %Cisco-SDWAN-RP_0-OMPD-6-INFO-400002: R0/0: OMPD: vSmart peer 1.1.1.30 state changed to Handshake
*Apr  3 09:39:24.261: %Cisco-SDWAN-RP_0-OMPD-5-NTCE-400002: R0/0: OMPD: vSmart peer 1.1.1.30 state changed to Up 
*Apr  3 09:39:24.263: %Cisco-SDWAN-RP_0-OMPD-6-INFO-400005: R0/0: OMPD: Number of vSmarts connected : 1

Verifications

If you've done everything and the onboarding process has been successful, the router should have both device and root certificates installed and a valid serial number, as highlighted below.

cEdge#show sdwan control local-properties 
personality                       vedge
sp-organization-name              networkacademy-io
organization-name                 networkacademy-io
root-ca-chain-status              Installed

certificate-status                Installed
certificate-validity              Valid
certificate-not-valid-before      Apr 03 09:38:19 2022 GMT
certificate-not-valid-after       Mar 31 09:38:19 2032 GMT

enterprise-cert-status            Not-Applicable
enterprise-cert-validity          Not Applicable
enterprise-cert-not-valid-before  Not Applicable
enterprise-cert-not-valid-after   Not Applicable

dns-name                          vbond.networkacademy.io
site-id                           1
domain-id                         1
protocol                          dtls
tls-port                          0
system-ip                         1.1.1.1
chassis-num/unique-id             CSR-D4B9356B-B36E-5EEC-XXXX-XXXXXXXXXXXX
serial-num                        CB17XXXX
token                             Invalid
keygen-interval                   1:00:00:00
retry-interval                    0:00:00:15
no-activity-exp-interval          0:00:00:20
dns-cache-ttl                     0:00:02:00
port-hopped                       TRUE
time-since-last-port-hop          0:00:12:01
embargo-check                     success
number-vbond-peers                0
number-active-wan-interfaces      1

If you wonder what is the role of the root certificate, or what is the process of building and verifying the identity of sdwan devices, you could check out our lesson for SD-WAN Certificates.

The ultimate verification will be to check the control connections to the sdwan controllers. 

cEdge#show sdwan control connections                                     
                                                              PEER                PEER                                          CONTROLLER 
PEER    PEER PEER            SITE       DOMAIN PEER           PRIV  PEER          PUB                                           GROUP      
TYPE    PROT SYSTEM IP       ID         ID     PRIVATE IP     PORT  PUBLIC IP     PORT  LOCAL COLOR     PROXY STATE UPTIME      ID         
-------------------------------------------------------------------------------------------------------------------------------------
vsmart  dtls 1.1.1.30        1          1      10.1.1.30      12346 10.1.1.30     12346 biz-internet    No    up     0:00:05:24  0           
vmanage dtls 1.1.1.20        1          0      10.1.1.20      12546 10.1.1.20     12546 biz-internet    No    up     0:00:05:27  0           

Key Takeaways

The following table shows the difference between the onboarding configuration fo a vEdge router running Viptela OS and a cEdge router running Cisco IOS-XE.