TLOC Extension

In real-world deployments, there will always be cases where the WAN edge routers cannot be directly connected to each available WAN transport as is shown in figure 1. In the example on the left, only one WAN edge device is connected to a single transport. This significantly reduces the overall availability and creates inefficiencies in the overlay fabric. TLOC-extension feature is designed to overcome this problem by extending the WAN transports to both SD-WAN routers without requiring direct attachment to both service provider clouds.

Figure 1. WAN Edges with a Single WAN Trasnport

Of course, the SD-WAN routers can connect to each WAN through front-facing switches as shown on the right but this is not a recommended approach because it adds additional costs and results in having another device to operate.

What is TLOC Extension?

TLOC extension is a feature that allows a WAN Edge router to communicate over the WAN transport connected to the adjacent WAN Edge router through a TLOC-extension interface. In the example shown in figure 2, vEdge-1 is directly connected to the Internet and also uses the TLOC extension feature to connect to the MPLS transport via vEdge-2. In the end, vEdge-1 has an overlay tunnel built over both WAN clouds even though it is directly attached only to one of the WAN transports. On the opposite side, vEdge-2 is attached to the MPLS cloud and uses the TLOC extension feature to connect to the Internet via WAN Edge 1.

Figure 2. TLOC Extension with Directly Connected vEdges

The feature is set up in a per-interface manner and provides transparent connectivity from one interface (called a TLOC extension interface) to a particular WAN transport. If we take vEdge-1 for example, it is unaware that the blue tunnel to the MPLS cloud passes through another WAN edge device because vEdge-2 extends their directly connected interface transparently to the MPLS transport.

TLOC Extension Types

Cisco SD-WAN allows for multiple ways of implementing TLOC extensions on WAN edge routers. The most typical and straightforward way of extending the transports is by using directly connected interfaces between the vEdges. However, depending on the local site design, the SD-WAN routers can be connected through Layer 2 switches as is illustrated in figure 3. L2 TLOC-extensions describe extensions between vEdges that are Layer2-adjacent are situated in the same broadcast domain/the same subnet.

The SD-WAN routers can also be connected at Layer 3 via any sort of IP routing. L3 TLOC-extensions describe extensions between two vEdges that are separated by an L3 device and are situated in different IP subnets. L3 TLOC extensions are implemented using GRE tunnels.

I'd also like to point out that TLOC-extensions can be configured using physical interfaces such as Ge0/0, Eth0, etc., and also using L3 subinterfaces.

Notable Limitations

At this point, it is a good time to point out some notable limitation of this feature:

As you might already know, TLOCs are only supported on routing interfaces. Well, the same applies to TLOC-extension interfaces as well. They are only supported on L3 routed interfaces. Switchports and SVIs cannot be configured as WAN/Overlay ports and can only be used within the service VPNs.
LTE can not be used as a TLOC extension interface between WAN Edge routers.
L3 TLOC-extension isn't supported on Viptela vEdge routers but only on SD-WAN devices running IOS-XE.
The feature would not work on transport ports that are bound to the loopback tunnels.

Configuring TLOC Extensions

Routing Considerations

A TLOC extension interface is always configured in VPN 0 and has an IP address assigned. Then the WAN interface to which it is bound is specified. For example, the vEdge-1's extension interface is ge0/4 and is bound to the Internet through ge0/0 and vEdge-2's extension interface is ge0/5 and is bound to the MPLS cloud through ge0/1. Then static default routes are configured in VPN0 on each WAN Edge router, pointing to the adjacent vEdge's IP as a next-hop.

However, some IP reachability considerations must be done in order for the overlay tunnels and BFD sessions to come up with remote peers over the TLOC extension interfaces. For example, to reach the MPLS cloud, vEdge-1 must be configured with a default route pointing to Ge0/5 of vEdge-2. Another very important part is the reverse IP reachability. To make sure that packets can be routed back to the TLOC extension interface, WAN Edge 2 must advertise subnet B to the MPLS provider. This can be done using any dynamic routing protocol that the service provider is willing to run. In a typical production deployment, some sort of inbound route policy is applied to deny all incoming dynamic routes from the provider since there is a static route pointing to the MPLS cloud.

In the case of WAN edge router 2, to reach the Internet cloud, it must be configured with a default route pointing to vEdge-1’s ge0/4 IP address. In a typical production deployment, subnet-A would be from the RFC1918 address space and NAT must be utilized on WAN edge-1 towards the INET cloud in order to ensure that traffic can be routed back from the Internet to vEdge-2 over the extension interface.

Let's go through a configuration example and see how this feature is implemented.

Extending vEdge-1 to the MPLS

Let's first extend router-1 to the MPLS cloud. For this example, we are going to use the private subnet 10.51.1.0/30 between Ge0/5 interfaces of both routers. This prefix 10.51.1.0/30 must also be advertised to the MPLS provider in order to have IP reachability in the other direction. For this purpose, we are going to utilize BGP.

It is important to understand that in order to extend vEdge-1 to the MPLS cloud, the actual TLOC extension happens on the adjacent router. From the perspective of router-1, Ge0/5 is connected to the MPLS provider in the same way as it would have been with a direct attachment link. Therefore, there is nothing special in the configuration on the vEdge-1 as you can see below. The interface is configured with IP address, color, and encapsulation type and a default route is specified. That's about it.

!
vpn 0
 !
 interface ge0/5
  ip address 10.51.1.1/30
  tunnel-interface
   encapsulation ipsec
   color mpls restrict
   allow-service all
  !
  no shutdown
 !
 ip route 0.0.0.0/0 10.51.1.2
 ip route 0.0.0.0/0 50.1.1.2
!

The "magic" happens on the neighboring router-2. In there, we should configure two things. The first one is to tell interface Ge0/5 that it extends the MPLS cloud attached to Ge0/1. The other important thing is to advertise the subnet 10.51.1.0/30 toward the service provider using BGP.

!
vpn 0
 !
 interface ge0/1
  ip address 10.50.2.1/30
  tunnel-interface
   encapsulation ipsec
   color mpls restrict
   allow-service all
  !
  no shutdown
 !
 interface ge0/5
  ip address 10.51.1.2/30
  tloc-extension ge0/1
  no shutdown
 !
 ip route 0.0.0.0/0 10.50.2.2
!
 router
  bgp 64551
   address-family ipv4-unicast
    network 10.51.1.0/30
   !
   neighbor 10.50.2.2
    no shutdown
    remote-as 65051
    address-family ipv4-unicast
    !
   !
  !

Once both steps are configured, we can see that vEdge-1 has a valid TLOC marked with the mpls color that has a BFD session in UP state even though the router is not directly connected to the MPLS cloud.

vEdge-1# show omp tlocs
---------------------------------------------------
tloc entries for 50.50.50.50
                 mpls
                 ipsec
---------------------------------------------------
            RECEIVED FROM:                   
peer            0.0.0.0
status          C,Red,R
loss-reason     not set
lost-to-peer    not set
lost-to-path-id not set
    Attributes:
     attribute-type    installed
     encap-key         not set
     encap-proto       0
     encap-spi         256
     encap-auth        sha1-hmac,ah-sha1-hmac
     encap-encrypt     aes256
     public-ip         10.51.1.1
     public-port       12406
     private-ip        10.51.1.1
     private-port      12406
     public-ip         ::
     public-port       0
     private-ip        ::
     private-port      0
     bfd-status        up
     domain-id         not set
     site-id           50
     overlay-id        not set
     preference        0
     tag               not set
     stale             not set
     weight            1
     version           2
    gen-id             0x80000019
     carrier           default
     restrict          1
     groups            [ 0 ]
     border             not set
     unknown-attr-len  not set

Extending vEdge-2 to the Internet

Let's now extend WAN edge 2 to the Internet provider. The configuration is simpler than the MPLS one because we do not need to advertise the interconnecting subnet 192.168.51.0/30 to the Internet provider, but only to NAT it when leaving the transport interface toward the provider.

All required configuration is shown in the diagram above. Once the config is applied, we can see that vEdge-2 has a valid TLOC marked with the public-internet color. Note that in the TLOC route, we can see the private-public IP address pair.

vEdge-2# show omp tlocs 
---------------------------------------------------
tloc entries for 50.50.50.51
                 public-internet
                 ipsec
---------------------------------------------------
            RECEIVED FROM:                   
peer            0.0.0.0
status          C,Red,R
loss-reason     not set
lost-to-peer    not set
lost-to-path-id not set
    Attributes:
     attribute-type    installed
     encap-key         not set
     encap-proto       0
     encap-spi         256
     encap-auth        sha1-hmac,ah-sha1-hmac
     encap-encrypt     aes256
     public-ip         50.1.1.1
     public-port       12366
     private-ip        192.168.51.2
     private-port      12366
     public-ip         ::
     public-port       0
     private-ip        ::
     private-port      0
     bfd-status        up
     domain-id         not set
     site-id           50
     overlay-id        not set
     preference        0
     tag               not set
     stale             not set
     weight            1
     version           2
    gen-id             0x80000014
     carrier           default
     restrict          0
     groups            [ 0 ]
     border             not set
     unknown-attr-len  not set

If we now check whether there is a BFD session over that color, you can see that there is one is UP state sourced from the TLOC extension interface 192.168.51.2 to a public IP address 80.1.1.1.

vEdge-2# show bfd sessions 
                            SOURCE TLOC      REMOTE TLOC                   DST PUBLIC
SYSTEM IP    SITE ID STATE  COLOR            COLOR           SOURCE IP     IP        
-------------------------------------------------------------------------------------
80.80.80.80  80      up     mpls             mpls            10.50.2.1     10.80.1.1 
80.80.80.80  80      up     public-internet  public-internet 192.168.51.2  80.1.1.1

Comments

mohanmuthu178

Tue, 05/31/2022 - 04:21

Great article and crystal clear diagrams. Well done. But I am still not able to understand what is the benefit of TLOC extension. It does not seem to add any level of redundancy, because, even if one circuit, or transport or one WANEdge fails, you still have only one other transport available. Apart from allowing one WANEdge to use the other side transport during normal operations, what is the benefit of TLOC Extension during a circuit/transport/WANEdge failure?
Thanks

Rps-Cheers

Sun, 08/07/2022 - 13:58

Notable Limitations mention：“Switchports and SVIs cannot be configured as WAN/Overlay ports and can only be used within the service VPNs.”
But “LAB 1 - Connecting to the WAN using TLOC Extension” also written “we can use a physical connection between the routers configured as a dot1Q trunk and then use two separate directly connected VLANs”. In addtion, we have configured SVI interface as a tloc extension interface as below..

interface Vlan4000
tunnel-interface
encapsulation ipsec weight 1
group 2
color biz-internet
no allow-service all
no allow-service bgp
<omit>

Ivan.Ivanov

Mon, 08/08/2022 - 07:24

Hi,
According to Viptela's official docs, only Layer 3 interfaces can be configured as tloc interfaces. Therefore, you can use sub-ints (e.g ge0/0.100, ge0/0.200, and so on) but can't use SVI interfaces (such as int vlan 100, int vlan 200, etc.)
HTH, Ivan

bekinde

Tue, 10/25/2022 - 03:11

I'm trying to duplicate this example with a TLOC extension and a NAT, and I can get the control plane up but the BFD sessions stay down. When I do a "show omp tlocs" on the vedge-2 equivalent it looks good like your example where it shows the correct public and private IP. It also looks correct on a remote vedge so the TLOC is getting advertised from vsmart. However if I do a "show bfd sessions" on the remote vedge the DST PUBLIC IP is the private IP that the vedge doesn't have a route to. Any help would be appreciated. Thanks.

msizi.mthembu

Fri, 11/04/2022 - 07:26

Hi Ivan,
Would it make sense to have TLOC-extension if I have two cEdge routers and one internet link terminating on each cEdge? I could configure one transport as biz-internet(R1) and the other as custom1(R2) as an example. Does this make design sense?

gvvenkov

Sun, 01/08/2023 - 15:21

Hi Ivan,
This is probably typo - on the Fig: "Configuring vEdge-2" I believe the "nat" is interface level under interface ge0/0.

Thanks,
George

Packet Sniffer

Wed, 03/01/2023 - 15:23

Hmmm, what does the "tloc-extension" command actually do? Even without it, everything would be routable. In situations with NAT, does it recognise some connections will not terminate upon itself, even with its own IP as the destination?

Ivan.Ivanov

Wed, 03/01/2023 - 21:48

Hi,
The reason is that interfaces with TLOC configuration do not behave as regular layer-3 interfaces. They do not allow passthrough traffic and only serve as tunnel endpoints. Therefore, traffic coming from the Internet can’t pass through the local INET TLOC and reach the adjacent router. Respectively, for the MPLS cloud. That is why we need to explicitly tell the routers that they are extending the adjacent router’s WAN transport and need to allow the traffic destined to the extension interface.
HTH, Ivan

Packet Sniffer

Thu, 03/02/2023 - 14:06

Thanks Ivan, but I tested in a lab and I was able to onboard a vEdge using NAT only from another vEdge, and not using the "tloc-extension" command.

Ivan.Ivanov

Thu, 03/02/2023 - 16:31

Yes, I understand your point. You can make it work but it is not supposed to be configured like that. What if you must use a private color? Or you should not NAT the transitory subnet. Or you have to make it work alongside a ZBFW, UTD, etc. It won't work without the "tloc-extension" command ...
HTH, Ivan

bilalhussain

Thu, 11/23/2023 - 12:46

Hi, you have two static default routes. One pointing to service provider and other to the TLOC extension interface. Would it not cause a conflict when you put two static default routes in real world ? Thanks.

Kalyani

Thu, 02/15/2024 - 05:42

Hello Ivan,

Based on the #show omp tlocs command output on both edge1 and edge2, both of them are in same site site id:50.
however, under #show bfd sessions, it shows that they are in site : 80.

Could you please clarify on this? my understanding was that if edges are in same site, they do not form BFD sessions with each other.

thschaff

Mon, 11/04/2024 - 14:26

Hi Ivan,

great content. I have a question regarding the last output

```
vEdge-2# show bfd sessions
SOURCE TLOC REMOTE TLOC DST PUBLIC
SYSTEM IP SITE ID STATE COLOR COLOR SOURCE IP IP
-------------------------------------------------------------------------------------
80.80.80.80 80 up mpls mpls 10.50.2.1 10.80.1.1
80.80.80.80 80 up public-internet public-internet 192.168.51.2 80.1.1.1
```

This was taken from vEdge 2. Shouldn't the destination of the first line (color: mpls) be the ip of the vEdge 1 TLOC extension interface (10.51.1.1)?

Thank you for clarification.