Onboarding cEdge Catalyst 8000V

In this lesson, we will show the procedure of onboarding the Catalyst 8000V virtual-form-factor router to the Cisco SD-WAN fabric using manual bootstrap configuration and Enterprise CA.

The lesson is an addition to the previous ones on setting up a Cisco Viptela SD-WAN home lab for testing and practicing for CCIE Enterprise or the 300-415 ENSDWI exam.

Initial Topology

Figure 1 shows the physical topology that we will use in this example. All cEdge devices are Catalyst 8000V routers running IOS-XE version c8000v-17.09.01a. We won't cover the process of bringing up the SD-WAN controllers in this lesson and will jump straight into the cEdge routers onboarding. If you want to understand how to deploy the Cisco SD-WAN controllers using local Enterprise CA, check out this lesson.

Onboarding cEdges CSR1k - Physical Topology — Figure 1. Onboarding cEdges 80000V - Physical Topology

Notice that the example will only show how to onboard one of the routers - cEdge-1. To set up the entire topology, you must repeat this for each WAN edge device.

Onboarding Catalyst 8kV

Prepare the software image

When a Catalyst 8000V router is powered on for the first time, it boots up in AUTONOMOUS mode, as seen in the output below.

%BOOT-5-OPMODE_LOG: R0/0: binos: System booted in AUTONOMOUS mode

The router asks if you would like to enter the initial config dialog. We answer no.

Then we provide a new enable password and save the configuration to NVRAM.

% Please answer 'yes' or 'no'.
Would you like to enter the initial configuration dialog? [yes/no]: no
  The enable secret is a password used to protect
  access to privileged EXEC and configuration modes.
  This password, after entered, becomes encrypted in
  the configuration.
  -------------------------------------------------
  secret should be of minimum 10 characters and maximum 32 characters with
  at least 1 upper case, 1 lower case, 1 digit and
  should not contain [cisco]
  -------------------------------------------------
  Enter enable secret: ************
  Confirm enable secret: ************
The following configuration command script was created:
enable secret 9 $9$uYATfwi9sBtruU$A4/FPncLMnru9Oo4oQjaF89yHqrCXDJBp**********
!
end
[0] Go to the IOS command prompt without saving this config.
[1] Return back to the setup without saving this config.
[2] Save this configuration to nvram and exit.
Enter your selection [2]: 2
Building configuration...
Guestshell destroyed successfully ommand to modify this configuration.
Press RETURN to get started!

Now, it is time to install the root CA certificate to the cEdge router. In our previous lessons, we deployed the SD-WAN controllers using the Enterprise CA approach, with the vBond controller acting as a Root CA. If you have deployed the SD-WAN controllers according to our lesson, you should have the Root CA certificate on vBond named ROOTCA.pem. The easiest way to install the root certificate on a Catalyst 8000v router is by creating a local file directly on the router using TCLSH, as shown in the following example. In the highlighted section, you should paste the ROOTCA.pem certificate you take from vBond, using the "cat ROOTCA.pem" command in vshell mode.

cEdge# tclsh 
cEdge(tcl)# puts [open "bootflash:ROOTCA.pem" w+] {
+> paste root-cert-here
+> }
cEdge-1(tcl)# exit

In the end, you should have the root certificate in the cEdge router's bootflash, as shown below.

Router# dir bootflash:
Directory of bootflash:/
31      -rw-             1315   Sep 3 2022 08:19:25 +00:00  ROOTCA.pem
131078  drwx             4096   Sep 3 2022 08:18:48 +00:00  tracelogs
131073  drwx             4096   Sep 3 2022 08:16:36 +00:00  .installer
28      -rw-              618   Sep 3 2022 08:16:25 +00:00  cvac.log
131112  drwx             4096   Sep 3 2022 08:16:24 +00:00  license_evlog
29      -rw-              157   Sep 3 2022 08:16:23 +00:00  csrlxc-cfg.log
...
...
5183766528 bytes total (3968655360 bytes free)

Now, it is time to reboot the router in CONTROLLER mode, which is required for SD-WAN. The router will notify you that a bootstrap configuration isn't available, but we will continue anyway.

Router# controller-mode enable 
Enabling controller mode will erase the nvram filesystem, remove all configuration files, and reload the box! 
Ensure the BOOT variable points to a valid image 
Continue? [confirm]
% Warning: Bootstrap config file needed for Day-0 boot is missing
Do you want to abort? (yes/[no]): no
 Mode change success

After the reboot, the router will boot up in CONTROLLER mode, as shown in the output below.

Oct 22 16:30:59.812: %BOOT-5-OPMODE_LOG: R0/0: binos: System booted in CONTROLLER mode

The last step is to install the root certificate using the following command.

cEdge# request platform software sdwan root-cert-chain install bootflash:ROOTCA.pem
Uploading root-ca-cert-chain via VPN 0
Copying ... /bootflash/ROOTCA.pem via VPN 0
Updating the root certificate chain..
Successfully installed the root certificate chain

If everything has gone smoothly, you should see our Enterprise CA Root certificate installed on the router.

cEdge# show sdwan certificate root-ca-cert | in network
        Issuer: C=US, ST=NY, L=NY, O=networkacademy-io, CN=root.certificate
        Subject: C=US, ST=NY, L=NY, O=networkacademy-io, CN=root.certificate

Bootstrap the cEdge router.

Once the router loads up with the SD-WAN software, we can go ahead and configure the minimal configuration required to join the SD-WAN overlay fabric. Notice that when the cEdge router runs in Controller mode (basically SD-WAN mode), we enter the configuration mode using the "config-transaction" command instead of the well-known "configure terminal" or simply "conf t."

In the following example, we configure the basic IP addressing and default routing alongside the essential system parameters such as system-IP, site-id, and vBond address. In this example, we will configure a DNS name for vBond, as recommended by Cisco.

cEdge# config-transaction
 hostname cEdge-C8K
 !
  int GigabitEthernet1
  ip address 39.3.1.1 255.255.255.0
  no shut
 !
 int GigabitEthernet2
  ip address 10.10.1.1 255.255.255.0
  no shut
 !
  ip route 0.0.0.0 0.0.0.0 39.3.1.254
  ip route 0.0.0.0 0.0.0.0 10.10.1.254	
  ip host vbond.networkacademy.io 10.1.1.10
 !
 system
  system-ip 1.1.1.1
  site-id 1
  organization-name "networkacademy-io"
  vbond vbond.networkacademy.io
 commit

At this point, you should be able to ping all Cisco SD-WAN controllers from the cEdge router that is being onboarded. If there is no IP connectivity between the WAN edge router and the controllers, there is no point in continuing further. You should troubleshoot the problem first.

We are ready to configure the SD-WAN data plane tunnels if IP reachability exists between the cEdge router and the controllers. Notice something very important - the Tunnel keyword in the "interface Tunnel" command should always be with a capital T. It is unlike in a regular Cisco IOS, where you can create a new tunnel using the "interface tunnel 1" command.

 sdwan
  int GigabitEthernet1
  tunnel-interface
   color biz-internet
   encapsulation ipsec
 !
  int GigabitEthernet2
  tunnel-interface
   color mpls restrict
   encapsulation ipsec
 !
 interface Tunnel 1
  ip unnumbered GigabitEthernet1
  tunnel source GigabitEthernet1
  tunnel mode sdwan
!
 interface Tunnel 2
  ip unnumbered GigabitEthernet2
  tunnel source GigabitEthernet2
  tunnel mode sdwan
 commit

At this point, the Catalyst 8000V router has the required configuration to join the SD-WAN fabric. The next step is to install the root certificate and activate the routers.

Activating the cEdge router

Before the cEdge router can be able to join the SD-WAN fabric, it must have a device certificate signed and installed by vManage. To do this, we need to have an unused chassis number and token in vManage under Configuration > Certificates. We use them to activate the cEdge router, as shown in the example below.

cEdge# request platform software sdwan vedge_cloud activate chassis-number C8K-D4B9356B-XXXX-XXXX-XXXX-XXXXXXXXXXXX token bf6bd685das45996xxxxxxxxxxxxxxxx

Once you've done, you should see in the logs that vManage logs into the cEdge using NETCONF over SSH, generates a CSR, then signs it and install a device certificate. Then the cEdge router should establish an OMP peering with vSmart and start receiving TLOCs and OMP routes.

cEdge#
*Apr  3 09:38:11.896: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User 'vmanage-admin' authenticated successfully from 1.1.1.20:41146 and was authorized for netconf over ssh. External groups:
*Apr  3 09:38:19.308: %Cisco-SDWAN-cEdge-2-action_notifier-6-INFO-1400002: R0/0: VCONFD_NOTIFIER: Notification: 4/3/2022 9:38:19 security-install-csr severity-level:minor host-name:default system-ip:1.1.1.1
*Apr  3 09:38:20.525: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User 'vmanage-admin' authenticated successfully from 1.1.1.20:41158 and was authorized for netconf over ssh. External groups:
*Apr  3 09:38:26.902: %Cisco-SDWAN-cEdge-2-action_notifier-6-INFO-1400002: R0/0: VCONFD_NOTIFIER: Notification: 4/3/2022 9:38:26 security-install-rcc severity-level:minor host-name:default system-ip:1.1.1.1
*Apr  3 09:38:45.329: %Cisco-SDWAN-cEdge-2-action_notifier-6-INFO-1400002: R0/0: VCONFD_NOTIFIER: Notification: 4/3/2022 9:38:45 security-install-certificate severity-level:minor host-name:default system-ip:1.1.1.1
*Apr  3 09:39:22.177: %Cisco-SDWAN-RP_0-OMPD-6-INFO-400002: vSmart peer 1.1.1.30 state changed to Init
*Apr  3 09:39:24.257: %Cisco-SDWAN-RP_0-OMPD-6-INFO-400002: R0/0: OMPD: vSmart peer 1.1.1.30 state changed to Handshake
*Apr  3 09:39:24.261: %Cisco-SDWAN-RP_0-OMPD-5-NTCE-400002: R0/0: OMPD: vSmart peer 1.1.1.30 state changed to Up 
*Apr  3 09:39:24.263: %Cisco-SDWAN-RP_0-OMPD-6-INFO-400005: R0/0: OMPD: Number of vSmarts connected : 1

Verifications

If you've done everything and the onboarding process has been successful, the router should have both device and root certificates installed and a valid serial number, as highlighted below.

cEdge# show sdwan control local-properties 
personality                       vedge
sp-organization-name              networkacademy-io
organization-name                 networkacademy-io
root-ca-chain-status              Installed
certificate-status                Installed
certificate-validity              Valid
certificate-not-valid-before      Apr 03 09:38:19 2022 GMT
certificate-not-valid-after       Mar 31 09:38:19 2032 GMT
enterprise-cert-status            Not-Applicable
enterprise-cert-validity          Not Applicable
enterprise-cert-not-valid-before  Not Applicable
enterprise-cert-not-valid-after   Not Applicable
dns-name                          vbond.networkacademy.io
site-id                           1
domain-id                         1
protocol                          dtls
tls-port                          0
system-ip                         1.1.1.1
chassis-num/unique-id             C8K-D4B9356B-B36E-5EEC-XXXX-XXXXXXXXXXXX
serial-num                        CB17XXXX
token                             Invalid
keygen-interval                   1:00:00:00
retry-interval                    0:00:00:15
no-activity-exp-interval          0:00:00:20
dns-cache-ttl                     0:00:02:00
port-hopped                       TRUE
time-since-last-port-hop          0:00:12:01
embargo-check                     success
number-vbond-peers                0
number-active-wan-interfaces      1

If you wonder what is the role of the root certificate or what is the process of building and verifying the identity of sdwan devices, you could check out our lesson for SD-WAN Certificates.

The ultimate verification will be to check the control connections to the sdwan controllers.

cEdge# show sdwan control connections                                     
                                                              PEER                PEER                                          CONTROLLER 
PEER    PEER PEER            SITE       DOMAIN PEER           PRIV  PEER          PUB                                           GROUP      
TYPE    PROT SYSTEM IP       ID         ID     PRIVATE IP     PORT  PUBLIC IP     PORT  LOCAL COLOR     PROXY STATE UPTIME      ID         
-------------------------------------------------------------------------------------------------------------------------------------
vsmart  dtls 1.1.1.30        1          1      10.1.1.30      12346 10.1.1.30     12346 biz-internet    No    up     0:00:05:24  0           
vmanage dtls 1.1.1.20        1          0      10.1.1.20      12546 10.1.1.20     12546 biz-internet    No    up     0:00:05:27  0

Platform Console

The last thing in running Catalyst 8000V in a virtual EVE-NG environment is to change the console method after attaching a device template.

Depending on your lab, you will most likely end up attaching a device template to the 8000V edge routers. What typically happens is that you lose access to the device via the console. This happens because, by default, the device boot up configured with the following command.

platform console serial

However, after you attach a template, vManage changes the console method to

platform console virtual

The "virtual" option defines that the 8000V router is accessed through the virtual VGA console of the hypervisor. To change the console method back to "serial," you must configure a CLI add-on feature template and add it to the respective device template the router is attached to.

Comments

kennguyennet3085

Fri, 02/10/2023 - 12:52

Are you using the software c8000v-universalk9.xx.xx.xx.iso file in a EVE-NG in this example Ivan? On the Cisco website it is little confusing, the c8000v IOS XE SD-WAN software is only around 20MB.

Ivan.Ivanov

Fri, 02/10/2023 - 14:39

Hi,
The image I use is "c8000v-universalk9_8G_serial.17.09.01a.qcow2".
Image size is 1770.63 MB
HTH, Ivan

Fri, 02/10/2023 - 15:36

Thanks!

suren130

Wed, 02/15/2023 - 10:53

For the vbond, vmanage and vsmart, we do a certificate signing request, send it off to the CA server and the CA server gives a ID cert. This is to be done for each of the controllers.

But same is not done for the vedge devices. Only the root cert is copied into the vedge and installed.

My question is, do the vedges need a ID cert? And if so, how does it get generated?

If we don’t need a ID cert, then what’s the purpose of the root cert in the vedge? I have an enterprise CA and I thought that only this CA can generate the ID cert on behalf of devices.

Wed, 02/15/2023 - 11:32

Hi Suren,
Do vEdges need a device certificate - Yes, they need a device certificate. The vManage controller automatically installs a vmanage-signed certificate (this is a setting and can be changed to manual certificate installation).
Why do we need to install the enterprise root cert on vEdges - Because all controllers have device certificates signed by the enterprise root CA (what you describe in the first sentence). For edge routers to be able to verify and trust these certificates, they need to have the root ca certificate installed.
HTH, Ivan

pehruby

Thu, 01/11/2024 - 15:05

Hi Ivan,
what private key is used by vManage to sign the certificate for cEdge (c8000v)?

durale

Fri, 03/17/2023 - 12:45

can we use sdwan interfaces and also interfaces to only route traffic ?

mohanmuthu178

Tue, 04/25/2023 - 02:01

Hello Ivan,
Great write up, as usual. One quick question.

After you issue the "activate" command on the CLI, how does this router learn the IP of the vManage? At this moment, the router only knows the IP of vBond. But router cannot authenticate to vBond because it does not have the certificate yet. So, my question is, how C8000v gets in touch with vManage for installing its certificate etc.?

My search on Cisco documentation for this did not yield any results. I am sure you can help.

Thanks,
Mohan Muthu

mfang

Mon, 05/15/2023 - 03:38

Hi Ivan,

Thank you for sharing. I bought your book from Amazon and I find its contents are update than your website. For example, in your book, version 20 is used in building lab in eve-ng. Good book and website.

I have a question and hope you can help.
"To do this, we need to have an unused chassis number and token in vManage under Configuration > Certificates. We use them to activate the cEdge router, as shown in the example below." , but I don't have any unused chassis number and token in my vManage as mentioned above. Is it because I need to have Cisco smart license account and should I need to login Cisco website to download provision file or sync my vManage with their database? thank you and looking forward to your answer.

Cedrick MEWO

Thu, 06/13/2024 - 15:10

If don't have Smart Account, you case install vEdge ID certificate to generate Certificate Serial so you can upload to vManage and vBond with command "request vedge add chassis-num xxxxxxxxxxxxxxx serial-num xxxxxxxxxxx" you can issue the command "show certificate serial" on Edge device

riveraf

Fri, 05/17/2024 - 01:20

Thank you Ivan for this wonderful site. I just want to comment that there is not access to vshell nor tclsh when the c8000s is moved to controller mode, so reinstalling the ROOTCA.pem certificate is a pain. I had to start all over the process to bring up the c8000v again as you indicate in this site until I found that the copy command is available in controller mode. You can copy files from -for instance the vmanage to the c8000v.
cEdge_4#copy scp:/home/admin bootflash:
Address or name of remote host [10.10.0.5]? -->vmange IP address
Source username [admin]? admin
Source filename [/home/admin]? ROOTCA.pem
Destination filename [ROOTCA.pem]?
%Warning:There is a file already existing with this name
Do you want to over write? [confirm]
viptela 20.9.4.1
Password:
Sending file modes: C0600 1322 ROOTCA.pem
!