CCIE Pursuit Blog

September 21, 2008

IOS error(?) with OSPF point-to-multipoint network in PPPoFR Hub-and-Spoke Topology

How nerdy is that for a blog post title?   🙂

I’ve run up against this issue twice now.  The issue seems to be an order-of-operations issue in IOS with PPPoFR in a hub-and-spoke network running OSPF network type point-to-multipoint.  This is most likely something that you’ll (hopefully) never see in a real network.  If you do, then kick the router (and the architect) really hard and run away quickly.

In this case we’ve configured PPPoFR with R5 as the hub and R1 and R2 as the spokes.  On R5 we have two PPP circuits assigned – one assigned to each hub DLCI – to a single virtual-template.  Something like this:

r5:
int s0/0.1 multipoint <- just to make it more “interesting”   🙂
 frame-relay interface-dlci 501 ppp virtual-template 1
 frame-relay interface-dlci 502 ppp virtual-template 1

Everything was cool until I configured OSPF and noticed that I wasn’t seeing routes from r2:

RSRack12R5#sh ip route os
     128.8.0.0/16 is variably subnetted, 9 subnets, 2 masks
O IA    128.8.136.0/24 [110/2] via 128.8.125.1, 00:39:59, Virtual-Access2
O IA    128.8.63.0/24 [110/11113] via 128.8.125.1, 00:22:43, Virtual-Access2

Looking at the individual routes it looks like the router is sending traffic to r2 which should be going to r1:

RSRack12R5#sh ip route 128.8.136.0
Routing entry for 128.8.136.0/24
  Known via “ospf 100”, distance 110, metric 2, type inter area
  Last update from 128.8.125.1 on Virtual-Access2, 00:40:10 ago
  Routing Descriptor Blocks:
  * 128.8.125.1, from 150.8.1.1, 00:40:10 ago, via Virtual-Access2
      Route metric is 2, traffic share count is 1

RSRack12R5#sh ip route 128.8.125.1
Routing entry for 128.8.125.1/32
  Known via “connected”, distance 0, metric 0 (connected, via interface)
  Routing Descriptor Blocks:
  * directly connected, via Virtual-Access1
      Route metric is 0, traffic share count is 1

RSRack12R5#sh int virtual-access 2
Virtual-Access2 is up, line protocol is up
  Hardware is Virtual Access interface
  Internet address is 128.8.125.5/24
  MTU 1500 bytes, BW 100000 Kbit, DLY 100000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Open
  Open: IPCP
  PPPoFR vaccess, cloned from Virtual-Template1
  Vaccess status 0x44
  Bound to Serial0/0.1 DLCI 502, Cloned from Virtual-Template1, loopback not set <-502 is r2
  Keepalive set (10 sec)
  DTR is pulsed for 5 seconds on reset
  Last input 00:00:12, output never, output hang never
  Last clearing of “show interface” counters 02:09:09
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     7938 packets input, 609028 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     7949 packets output, 604952 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions

RSRack12R5#p 128.8.136.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 128.8.136.3, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)

RSRack12R5#trace 128.8.136.3

Type escape sequence to abort.
Tracing the route to 128.8.136.3

  1 128.8.125.228 msec 28 msec 32 msec  <-should go to r1
  2  *

An IP packet debug on r5 shows that the IOS does see any problems with shipping the wrong packets to r2:

Sep  4 07:38:04.065: IP: tableid=0, s=128.8.125.5 (local), d=128.8.136.3 (Virtual-Access2), routed via FIB
Sep  4 07:38:04.069: IP: s=128.8.125.5 (local), d=128.8.136.3 (Virtual-Access2), len 100, sending
Sep  4 07:38:04.069:     ICMP type=8, code=0

It looks like the virtual-access interfaces are associated with the wrong DLCIs.  I tried a reload – no change.

I ripped out PPPoFr and readded it:

RSRack12R5#sh ip int br | i Virtu
Virtual-Access1            128.8.125.5     YES TFTP   up                    up

Virtual-Template1          128.8.125.5     YES NVRAM  down                  down

Virtual-Access2            128.8.125.5     YES TFTP   up                    up

Virtual-Access3            unassigned      YES unset  down                  down

RSRack12R5#sh int virtual-acc 1 | i Bound
  Bound to Serial0/0.1 DLCI 501, Cloned from Virtual-Template1, loopback not set
RSRack12R5#sh int virtual-acc 2 | i Bound
  Bound to Serial0/0.1 DLCI 502, Cloned from Virtual-Template1, loopback not set
RSRack12R5#sh int virtual-acc 3 | i Bound
RSRack12R5#

RSRack12R5#sh ip route 128.8.136.3
Routing entry for 128.8.136.0/24
  Known via “ospf 100”, distance 110, metric 2, type inter area
  Last update from 128.8.125.1 on Virtual-Access2, 00:02:18 ago
  Routing Descriptor Blocks:
  * 128.8.125.1, from 150.8.1.1, 00:02:18 ago, via Virtual-Access2  <- should use virtual-access1
      Route metric is 2, traffic share count is 1

Same issue:

RSRack12R5#trace 128.8.136.3

Type escape sequence to abort.
Tracing the route to 128.8.136.3

  1 128.8.125.228 msec 28 msec 32 msec  <-should go to r1
  2  *  *

The IOS is up to date:

RSRack12R5#sh ver | i IOS|emo
Cisco IOS Software, C2600 Software (C2600-ADVENTERPRISEK9-M), Version 12.4(10), RELEASE SOFTWARE (fc1)
Cisco 2611XM (MPC860P) processor (revision 3.1) with 127308K/3764K bytes of memory.

I gave up.  I did go back and check my config line by line and everything was in agreement with the IE solution guide.  I looked in my notes and I did not encounter this issue on my last attempt at this lab.  If this were the actual lab I guess that I would rip out the PPPoFR configuration and just use a conventional hub-and-spoke FR network and eat the PPPoFR points.

It appears that I’m not the only person to run up against this issue:

It looks like the fixes are:

1) Reorder the frame-relay maps:

I did that, but IOS puts them back in order:

interface Serial0/0.1 multipoint
 frame-relay interface-dlci 501 ppp Virtual-Template1
 frame-relay interface-dlci 502 ppp Virtual-Template1

But for some reason it fixed the issue:

RSRack12R5#p 128.8.136.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 128.8.136.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 60/60/60 ms

RSRack12R5#trace 128.8.136.3

Type escape sequence to abort.
Tracing the route to 128.8.136.3

  1 128.8.125.1 28 msec 32 msec 29 msec
  2 128.8.136.3 32 msec *  40 msec

This fix does my fucking head in.  I actually stripped out the PPPoFR configuration before and then readded it.  The actual ORDER of configuration is important?  Even though IOS reorders the configuration lines automatically?

Unfortunately this “fix” does not survive a reload:

After reload:
RSRack12R5#p  128.8.136.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 128.8.136.3, timeout is 2 seconds:

Success rate is 0 percent (0/3)

RSRack12R5#trace  128.8.136.3

Type escape sequence to abort.
Tracing the route to 128.8.136.3

  1 128.8.125.228 msec 32 msec 28 msec
  2
RSRack12R5#

2) Create a separate virtual-template for each DLCI (I think that this is what I did the first time that I ran through this lab).

interface Serial0/0.1 multipoint
frame-relay interface-dlci 501ppp Virtual-Template1
frame-relay interface-dlci 502ppp Virtual-Template2

interface Virtual-Template1
ip address 128.1.125.5 255.255.255.0
ip ospf network point-to-multipoint
!
interface Virtual-Template2
ip address 128.1.125.5 255.255.255.0
ip ospf network point-to-multipoint

RSRack12R5#sh ip os neigh

Neighbor ID     Pri   State           Dead Time   Address         Interface
150.8.2.2         0   FULL/  –        00:01:59    128.8.125.2     Virtual-Access2
150.8.1.1
         0   FULL/  –        00:01:59    128.8.125.1     Virtual-Access1

Did not work:

RSRack12R5#p  128.8.136.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 128.8.136.3, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)

RSRack12R5#trace  128.8.136.3

Type escape sequence to abort.
Tracing the route to 128.8.136.3

  1 128.8.125.228 msec 32 msec 33 msec
  2
RSRack12R5#

But if I switch the virtual-template assignment (as suggested in the forum)……still more failure:

interface Serial0/0.1 multipoint
 frame-relay interface-dlci 501ppp Virtual-Template2
 frame-relay interface-dlci 502ppp Virtual-Template1

RSRack12R5#p 128.8.136.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 128.8.136.3, timeout is 2 seconds:
..
Success rate is 0 percent (0/2)

RSRack12R5#trace 128.8.136.3

Type escape sequence to abort.
Tracing the route to 128.8.136.3

  1 128.8.125.229 msec 28 msec 32 msec
  2

Brian Dennis stated that the proctors do not reload your routers before grading, so I guess that I have to count on that being the case if I run across this issue in the lab.

I ran across the same issue on an IPexpert lab (mock lab 3).  I was only getting routes from one spoke in a PPPoFR PPP multilink point-to-multipoint OSPF network…with a fucking virtual link thrown in just to increase the nerdtastic design.  I had to flip the multilink assignment of (many!) frame-relay interfaces and that got is cooking once again:

interface Serial0/1/0
 no ip address
 encapsulation frame-relay
 frame-relay interface-dlci 205 ppp Virtual-Template2
 frame-relay interface-dlci 206 ppp Virtual-Template1
 frame-relay interface-dlci 215 ppp Virtual-Template2
 frame-relay interface-dlci 216 ppp Virtual-Template1
 frame-relay interface-dlci 225 ppp Virtual-Template2
 frame-relay interface-dlci 226 ppp Virtual-Template1

no frame-relay interface-dlci 205 ppp Virtual-Template2
no frame-relay interface-dlci 206 ppp Virtual-Template1
no frame-relay interface-dlci 215 ppp Virtual-Template2
no frame-relay interface-dlci 216 ppp Virtual-Template1
no frame-relay interface-dlci 225 ppp Virtual-Template2
no frame-relay interface-dlci 226 ppp Virtual-Template1

frame-relay traffic-shaping
frame-relay interface-dlci 205 ppp Virtual-Template1
frame-relay interface-dlci 206 ppp Virtual-Template2
frame-relay interface-dlci 215 ppp Virtual-Template1
frame-relay interface-dlci 216 ppp Virtual-Template2
frame-relay interface-dlci 225 ppp Virtual-Template1
frame-relay interface-dlci 226 ppp Virtual-Template2

January 8, 2008

A Couple Of Must Read Posts

While blog surfing today, I came across two really excellent posts explaining a couple of “obscure” technologies.  These are must reads for CCIE candidates:

Internetwork Expert Blog:

Understanding PPP over Frame Relay (PPPoFR)– a great tutorial on PPPoFR, including when and why you would use it in the “real world.”

CCIE Candidate

area nssa translate type7 suppress-fa– Ethan Banks shares a great scenario concerning what happens when two area 0 ABRs border an NSSA area.  Who handles the 7-to-5 LSA translation?  If an ABR learns a route as an N1 from an NSSA, and the same route as an E1 from another neighbor – which does it prefer?  This is a great scenario to mock up in Dynamips.

Blog at WordPress.com.