This lesson discusses the most common issues encountered with IP multicast using PIM Dense mode. Along the way, we introduce some of the multicast-specific troubleshooting tools available on Cisco IOS-XE devices.

Lab Topology and Initial State

The initial lab topology is shown in the diagram below. Notice that we have a full IP reachability using OSPFv2. PIM Dense mode is configured on all router links. The source is transmitting multicast traffic on group 239.1.1.1 (pinging 239.1.1.1).

We will use this topology in each troubleshooting scenario by introducing and resolving different faults. You can download the EVE-NG file at the end of the lesson.

Scenario 1 - Multicast routing not enabled on a router.

One of the most common problems with IP multicast, especially in exam environments, is that a router along the path is not enabled to perform multicast routing. 

Recall that multicast routing is disabled by default on all Cisco IOS and IOS-XE routers. We enable the mcast routing functionality by configuring a single CLI command in the global configuration mode of the router as follows:

R1# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)# ip multicast-routing
R1(config)# end
R1#

A router won't forward multicast traffic between interfaces if it is not configured to do multicast routing. Since the process of enabling the functionality is manual,  it is often forgotten by mistake or intentionally left configured in exam/testing environments.

Let's see how we can systematically troubleshoot a topology where one of the routers is not configured with IP multicast routing.

Problem Symptoms

It has been reported that PC2 is not able to receive the multicast stream (10.1.1.1, 239.1.1.1). 

• PC2 is joining the group (sends IGMP Membership report for 239.1.1.1)
• PC2 can ping the source IP address (there is unicast reachability)

Troubleshooting

Remember that the multicast distribution tree (MDT) is receiver-initiated. It is the responsibility of the receiver to join the group IP address and the responsibility of the last hop router to initiate the building of the source path tree (SPT) to the source.

The multicast traffic flows from the source toward the receivers. This is the data plane and is referred to as the downstream direction. However, the control-plane operations happen in the other direction from the receiver toward the source. This is referred to as the upstream direction. Since most issues happen with the control plane rather than the data plane, it is recommended that troubleshooting from the receivers' local gateway router is always started upstream.

MUST REMEMBER:
The Data-plane works downstream.
The Control-plane works upstream.

Starting point: The LHR

As a general rule of thumb - we always start troubleshooting with the receiver and the last-hop router and then continue upstream toward the source. In our example, the last-hop router of PC2 is R8. Let's log in to R8 and see what we can understand.

First, we check whether the receiver has joined the multicast group address using IGMP.

R8# sh ip igmp groups 239.1.1.1
IGMP Connected Group Membership
Group Address    Interface                Uptime    Expires   Last Reporter   Group Accounted
239.1.1.1        Ethernet1/0              00:34:49  00:02:17  10.10.20.1   

We can see that PC2 has successfully joined the group address 239.1.1.1. 

The next step is to see if multicast routing is enabled on the router. There is a direct way to verify this and an indirect way. We can directly check it by using the following command.

R8# sh ip multicast 
  Multicast Routing: enabled
  Multicast Multipath: disabled
  Multicast Route limit: No limit
  Multicast Fallback group mode: Dense
  Number of multicast boundaries configured with filter-autorp option: 0
  MoFRR: Disabled
  Multicast Service-Reflect Capabilities:
        Unicast to Multicast

However, we can also check this by looking at the router's mroute table. If multicast routing is not enabled, the mroute table directly says, "IP Multicast Forwarding is not enabled."

The next step is to check the LHR's route table. If R8 receives the multicast from the source, we expect to see a source-group pair (S, G) entry in the mroute table.

R8# sh ip mroute 239.1.1.1
IP Multicast Routing Table
<lines omitted for brevity>
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
                          t - LISP transit group
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 239.1.1.1), 00:41:45/00:02:23, RP 0.0.0.0, flags: DC
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Ethernet0/1, Forward/Dense, 00:41:17/stopped, flags: 
    Ethernet1/0, Forward/Dense, 00:41:45/stopped, flags: 

We can see that there is no (S, G). Therefore, R8 does not receive the multicast stream from the source. The initial flooding of multicast traffic does not reach the last-hop router. Hence, there must be a problem in the path between the source and the LHR.

Moving Upstream

Okay, how do we continue? We follow the unicast route to the source IP address 10.1.1.1. If we execute the show IP route 10.1.1.1 on R8, we see that the next hop is R6, so we log into R6.

On R6, we first check the mroute table to see if there is an entry for the source-group pair (10.1.1.1,239.1.1.1).

R6# sh ip mroute          
IP Multicast Routing Table
<lines omitted for brevity>
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
                          t - LISP transit group
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode

IP Multicast Forwarding is not enabled.

However, we can see that the multicast routing is obviously not enabled. Let's enable it.

Solution

We enable the multicast routing on R6

R6# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R6(config)# ip multicast-routing 
R6(config)# end

Now, if we check the mroute table of the last hop router R8, we can see that it has an (S, G) entry for the source-group pair. We can also see that it receives the multicast on incoming interface E0/1 and sends it to the receiver via outgoing interface E1/0.

R8# sh ip mroute 239.1.1.1
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report, 
       Z - Multicast Tunnel, z - MDT-data group sender, 
       Y - Joined MDT-data group, y - Sending to MDT-data group, 
       G - Received BGP C-Mroute, g - Sent BGP C-Mroute, 
       N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed, 
       Q - Received BGP S-A Route, q - Sent BGP S-A Route, 
       V - RD & Vector, v - Vector, p - PIM Joins on route, 
       x - VxLAN group, c - PFP-SA cache created entry, 
       * - determined by Assert, # - iif-starg configured on rpf intf, 
       e - encap-helper tunnel flag, l - LISP decap ref count contributor
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
                          t - LISP transit group
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 239.1.1.1), 01:51:25/stopped, RP 0.0.0.0, flags: DC
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Ethernet0/1, Forward/Dense, 01:50:57/stopped, flags: 
    Ethernet1/0, Forward/Dense, 01:51:25/stopped, flags: 

(10.1.1.1, 239.1.1.1), 00:00:28/00:02:31, flags: T
  Incoming interface: Ethernet0/1, RPF nbr 10.1.8.1
  Outgoing interface list:
    Ethernet1/0, Forward/Dense, 00:00:28/stopped, flags: 

Scenario 2 - PIM not enabled on an interface.

The second most common problem people encounter with IP Multicast, especially in an exam environment, is an interface along the MDT that is not enabled with PIM. IP multicast must be deployed within a single unicast routing domain with PIM enabled on all links that run the IGP.

Scenario 2.1 - PIM not enabled on a transit interface.

Problem Symptoms

It has been reported that PC2 does not receive the multicast stream (10.1.1.1, 239.1.1.1). 

Troubleshooting Steps

Following the troubleshooting logic shown in Figure 2, we start by logging into the last-hop router, R8. The first thing we always do is check the mroute table to see the entries for group 239.1.1.1.

R8#sh ip mroute 239.1.1.1
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report, 
       Z - Multicast Tunnel, z - MDT-data group sender, 
       Y - Joined MDT-data group, y - Sending to MDT-data group, 
       G - Received BGP C-Mroute, g - Sent BGP C-Mroute, 
       N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed, 
       Q - Received BGP S-A Route, q - Sent BGP S-A Route, 
       V - RD & Vector, v - Vector, p - PIM Joins on route, 
       x - VxLAN group, c - PFP-SA cache created entry, 
       * - determined by Assert, # - iif-starg configured on rpf intf, 
       e - encap-helper tunnel flag, l - LISP decap ref count contributor
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
                          t - LISP transit group
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 239.1.1.1), 06:04:51/00:02:12, RP 0.0.0.0, flags: DC
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Ethernet0/1, Forward/Dense, 06:04:24/stopped, flags: 
    Ethernet1/0, Forward/Dense, 06:04:51/stopped, flags: 

There is an (*, G) entry that reflects the router's PIM neighbors and directly connected receivers - e0/1 has a PIM adjacency, while e1/0 has a directly connected receive. That's why those two interfaces are in the OIL. 

Obviously, there is no (S, G) entry which means the multicast traffic does not reach the LHR. Therefore, the problem is somewhere along the way of the traffic. In this example, we will directly trace the multicast path from the LHR to the source and see where the path breaks. Then we will directly check there.

Understanding the mtrace command

One of the most useful troubleshooting tools when it comes to IP multicast on Cisco devices is the mtrace command. It is a control-plane tool that traces the reverse path from a receiver toward the source of a multicast group. It is very useful for narrowing down the problem area in large topologies. Suppose a network with hundreds of routers. You cannot simply log into each one and check the multicast routing table. You need to narrow down the problem to a few devices and look there.

An example of a successful mtrace is shown in the output below. Notice that we provide the source IP, destination IP, and group IP to the mtrace command.

R8# mtrace 10.1.1.1 10.10.20.1 239.1.1.1
           (source, destination, group)
Type escape sequence to abort.
Mtrace from 10.1.1.1 to 10.10.20.1 via group 239.1.1.1
From source (?) to destination (?)
Querying full reverse path... 
 0  10.10.20.1
-1  10.10.20.254 ==> 10.1.8.3 PIM  [10.1.1.0/24]
-2  10.1.8.1 ==> 10.1.6.2 PIM  [10.1.1.0/24]
-3  10.1.6.1 ==> 10.1.2.4 PIM  [10.1.1.0/24]
-4  10.1.2.1 ==> 10.1.1.253 PIM_MT  [10.1.1.0/24]
-5  10.1.1.1

• The source is the source address of the multicast traffic. In our example, this would be 10.1.1.1.
• The destination is a receiver of the multicast. In our example, this would be PC2 10.10.20.1.
• The group is the multicast group address. In our example, this would be 239.1.1.1.

The example shows a successful mtrace from R8 to the source. Let's break down what the output shows.

• line 0 says that the process starts with the receiver 10.10.20.1
• line -1 represents router R8
• line -2 represents router R6
• line -3 represents router R4
• line -4 represents router R2
• line -5 says that we can successfully reach the source.

This is the control-plane path from the receiver to the source. We have repeatedly said that the data plane works opposite to the control plane. Therefore, we can decode this output as follows: 

• The multicast traffic from (10.1.1.1, 239.1.1.1) is received by 10.1.1.253 (R2's interface connected to the source). It forwards it out via 10.1.2.1.
• Then, the traffic is received by 10.1.2.4 (R4). It forwards it out via 10.1.6.1.
• Then, the traffic is received by 10.1.6.2 (R6). It forwards it out via 10.1.8.1.
• Then, 10.1.8.3 (R8) receives the traffic and forwards it out via 10.10.20.254 (the interface connected to the receiver).

Let's briefly discuss how the mtrace command works using the diagram below.

The first important aspect of the diagnostic tool is that it uses the IGMP protocol. It uses the following three unicast messages:

• IGMP Traceroute Query
• IGMP Traceroute Request
• IGMP Traceroute Response