CCIE Pursuit Blog

September 21, 2008

IOS error(?) with OSPF point-to-multipoint network in PPPoFR Hub-and-Spoke Topology

How nerdy is that for a blog post title?   🙂

I’ve run up against this issue twice now.  The issue seems to be an order-of-operations issue in IOS with PPPoFR in a hub-and-spoke network running OSPF network type point-to-multipoint.  This is most likely something that you’ll (hopefully) never see in a real network.  If you do, then kick the router (and the architect) really hard and run away quickly.

In this case we’ve configured PPPoFR with R5 as the hub and R1 and R2 as the spokes.  On R5 we have two PPP circuits assigned – one assigned to each hub DLCI – to a single virtual-template.  Something like this:

r5:
int s0/0.1 multipoint <- just to make it more “interesting”   🙂
 frame-relay interface-dlci 501 ppp virtual-template 1
 frame-relay interface-dlci 502 ppp virtual-template 1

Everything was cool until I configured OSPF and noticed that I wasn’t seeing routes from r2:

RSRack12R5#sh ip route os
     128.8.0.0/16 is variably subnetted, 9 subnets, 2 masks
O IA    128.8.136.0/24 [110/2] via 128.8.125.1, 00:39:59, Virtual-Access2
O IA    128.8.63.0/24 [110/11113] via 128.8.125.1, 00:22:43, Virtual-Access2

Looking at the individual routes it looks like the router is sending traffic to r2 which should be going to r1:

RSRack12R5#sh ip route 128.8.136.0
Routing entry for 128.8.136.0/24
  Known via “ospf 100”, distance 110, metric 2, type inter area
  Last update from 128.8.125.1 on Virtual-Access2, 00:40:10 ago
  Routing Descriptor Blocks:
  * 128.8.125.1, from 150.8.1.1, 00:40:10 ago, via Virtual-Access2
      Route metric is 2, traffic share count is 1

RSRack12R5#sh ip route 128.8.125.1
Routing entry for 128.8.125.1/32
  Known via “connected”, distance 0, metric 0 (connected, via interface)
  Routing Descriptor Blocks:
  * directly connected, via Virtual-Access1
      Route metric is 0, traffic share count is 1

RSRack12R5#sh int virtual-access 2
Virtual-Access2 is up, line protocol is up
  Hardware is Virtual Access interface
  Internet address is 128.8.125.5/24
  MTU 1500 bytes, BW 100000 Kbit, DLY 100000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Open
  Open: IPCP
  PPPoFR vaccess, cloned from Virtual-Template1
  Vaccess status 0x44
  Bound to Serial0/0.1 DLCI 502, Cloned from Virtual-Template1, loopback not set <-502 is r2
  Keepalive set (10 sec)
  DTR is pulsed for 5 seconds on reset
  Last input 00:00:12, output never, output hang never
  Last clearing of “show interface” counters 02:09:09
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     7938 packets input, 609028 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     7949 packets output, 604952 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions

RSRack12R5#p 128.8.136.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 128.8.136.3, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)

RSRack12R5#trace 128.8.136.3

Type escape sequence to abort.
Tracing the route to 128.8.136.3

  1 128.8.125.228 msec 28 msec 32 msec  <-should go to r1
  2  *

An IP packet debug on r5 shows that the IOS does see any problems with shipping the wrong packets to r2:

Sep  4 07:38:04.065: IP: tableid=0, s=128.8.125.5 (local), d=128.8.136.3 (Virtual-Access2), routed via FIB
Sep  4 07:38:04.069: IP: s=128.8.125.5 (local), d=128.8.136.3 (Virtual-Access2), len 100, sending
Sep  4 07:38:04.069:     ICMP type=8, code=0

It looks like the virtual-access interfaces are associated with the wrong DLCIs.  I tried a reload – no change.

I ripped out PPPoFr and readded it:

RSRack12R5#sh ip int br | i Virtu
Virtual-Access1            128.8.125.5     YES TFTP   up                    up

Virtual-Template1          128.8.125.5     YES NVRAM  down                  down

Virtual-Access2            128.8.125.5     YES TFTP   up                    up

Virtual-Access3            unassigned      YES unset  down                  down

RSRack12R5#sh int virtual-acc 1 | i Bound
  Bound to Serial0/0.1 DLCI 501, Cloned from Virtual-Template1, loopback not set
RSRack12R5#sh int virtual-acc 2 | i Bound
  Bound to Serial0/0.1 DLCI 502, Cloned from Virtual-Template1, loopback not set
RSRack12R5#sh int virtual-acc 3 | i Bound
RSRack12R5#

RSRack12R5#sh ip route 128.8.136.3
Routing entry for 128.8.136.0/24
  Known via “ospf 100”, distance 110, metric 2, type inter area
  Last update from 128.8.125.1 on Virtual-Access2, 00:02:18 ago
  Routing Descriptor Blocks:
  * 128.8.125.1, from 150.8.1.1, 00:02:18 ago, via Virtual-Access2  <- should use virtual-access1
      Route metric is 2, traffic share count is 1

Same issue:

RSRack12R5#trace 128.8.136.3

Type escape sequence to abort.
Tracing the route to 128.8.136.3

  1 128.8.125.228 msec 28 msec 32 msec  <-should go to r1
  2  *  *

The IOS is up to date:

RSRack12R5#sh ver | i IOS|emo
Cisco IOS Software, C2600 Software (C2600-ADVENTERPRISEK9-M), Version 12.4(10), RELEASE SOFTWARE (fc1)
Cisco 2611XM (MPC860P) processor (revision 3.1) with 127308K/3764K bytes of memory.

I gave up.  I did go back and check my config line by line and everything was in agreement with the IE solution guide.  I looked in my notes and I did not encounter this issue on my last attempt at this lab.  If this were the actual lab I guess that I would rip out the PPPoFR configuration and just use a conventional hub-and-spoke FR network and eat the PPPoFR points.

It appears that I’m not the only person to run up against this issue:

It looks like the fixes are:

1) Reorder the frame-relay maps:

I did that, but IOS puts them back in order:

interface Serial0/0.1 multipoint
 frame-relay interface-dlci 501 ppp Virtual-Template1
 frame-relay interface-dlci 502 ppp Virtual-Template1

But for some reason it fixed the issue:

RSRack12R5#p 128.8.136.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 128.8.136.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 60/60/60 ms

RSRack12R5#trace 128.8.136.3

Type escape sequence to abort.
Tracing the route to 128.8.136.3

  1 128.8.125.1 28 msec 32 msec 29 msec
  2 128.8.136.3 32 msec *  40 msec

This fix does my fucking head in.  I actually stripped out the PPPoFR configuration before and then readded it.  The actual ORDER of configuration is important?  Even though IOS reorders the configuration lines automatically?

Unfortunately this “fix” does not survive a reload:

After reload:
RSRack12R5#p  128.8.136.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 128.8.136.3, timeout is 2 seconds:

Success rate is 0 percent (0/3)

RSRack12R5#trace  128.8.136.3

Type escape sequence to abort.
Tracing the route to 128.8.136.3

  1 128.8.125.228 msec 32 msec 28 msec
  2
RSRack12R5#

2) Create a separate virtual-template for each DLCI (I think that this is what I did the first time that I ran through this lab).

interface Serial0/0.1 multipoint
frame-relay interface-dlci 501ppp Virtual-Template1
frame-relay interface-dlci 502ppp Virtual-Template2

interface Virtual-Template1
ip address 128.1.125.5 255.255.255.0
ip ospf network point-to-multipoint
!
interface Virtual-Template2
ip address 128.1.125.5 255.255.255.0
ip ospf network point-to-multipoint

RSRack12R5#sh ip os neigh

Neighbor ID     Pri   State           Dead Time   Address         Interface
150.8.2.2         0   FULL/  –        00:01:59    128.8.125.2     Virtual-Access2
150.8.1.1
         0   FULL/  –        00:01:59    128.8.125.1     Virtual-Access1

Did not work:

RSRack12R5#p  128.8.136.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 128.8.136.3, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)

RSRack12R5#trace  128.8.136.3

Type escape sequence to abort.
Tracing the route to 128.8.136.3

  1 128.8.125.228 msec 32 msec 33 msec
  2
RSRack12R5#

But if I switch the virtual-template assignment (as suggested in the forum)……still more failure:

interface Serial0/0.1 multipoint
 frame-relay interface-dlci 501ppp Virtual-Template2
 frame-relay interface-dlci 502ppp Virtual-Template1

RSRack12R5#p 128.8.136.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 128.8.136.3, timeout is 2 seconds:
..
Success rate is 0 percent (0/2)

RSRack12R5#trace 128.8.136.3

Type escape sequence to abort.
Tracing the route to 128.8.136.3

  1 128.8.125.229 msec 28 msec 32 msec
  2

Brian Dennis stated that the proctors do not reload your routers before grading, so I guess that I have to count on that being the case if I run across this issue in the lab.

I ran across the same issue on an IPexpert lab (mock lab 3).  I was only getting routes from one spoke in a PPPoFR PPP multilink point-to-multipoint OSPF network…with a fucking virtual link thrown in just to increase the nerdtastic design.  I had to flip the multilink assignment of (many!) frame-relay interfaces and that got is cooking once again:

interface Serial0/1/0
 no ip address
 encapsulation frame-relay
 frame-relay interface-dlci 205 ppp Virtual-Template2
 frame-relay interface-dlci 206 ppp Virtual-Template1
 frame-relay interface-dlci 215 ppp Virtual-Template2
 frame-relay interface-dlci 216 ppp Virtual-Template1
 frame-relay interface-dlci 225 ppp Virtual-Template2
 frame-relay interface-dlci 226 ppp Virtual-Template1

no frame-relay interface-dlci 205 ppp Virtual-Template2
no frame-relay interface-dlci 206 ppp Virtual-Template1
no frame-relay interface-dlci 215 ppp Virtual-Template2
no frame-relay interface-dlci 216 ppp Virtual-Template1
no frame-relay interface-dlci 225 ppp Virtual-Template2
no frame-relay interface-dlci 226 ppp Virtual-Template1

frame-relay traffic-shaping
frame-relay interface-dlci 205 ppp Virtual-Template1
frame-relay interface-dlci 206 ppp Virtual-Template2
frame-relay interface-dlci 215 ppp Virtual-Template1
frame-relay interface-dlci 216 ppp Virtual-Template2
frame-relay interface-dlci 225 ppp Virtual-Template1
frame-relay interface-dlci 226 ppp Virtual-Template2

Advertisements

7 Comments »

  1. While it would definitely help if you’d include more information, the problem is probably due to the fact that you have the same IP address on multiple Vi interfaces, resulting in quite a routing mess.

    Virtual access interfaces should be unnumbered. You could do other “interesting” things apart from using “ip unnumbered”, but assigning a fixed IP address to a template that is used by multiple physical interfaces leading in different directions is a recipe for disaster.

    Comment by Ivan Pepelnjak — September 21, 2008 @ 1:10 pm | Reply

  2. Have you tried configuring the Virtual templates before the serial interface?I seem to avoid most issues with PPPoFR when i configure it this way.
    Cheers

    Comment by Julian — September 22, 2008 @ 1:56 am | Reply

  3. it would be a big help if you’d include more information.. it might be due to a big jumble because you are working on 1 IP adress on a few Vi interfaces.

    Comment by Increase Your Web Site Traffic — September 27, 2008 @ 2:59 pm | Reply

  4. I faced the same problem with one of the labs once, while I had no problem applying the same configuration “as I remember” in another one. exactly the same symptoms and tried your same solutions with no help.

    Comment by wael osama — September 30, 2008 @ 9:50 pm | Reply

  5. Thank god I thought it was only me having this issue. I have the same problems in one of the ip expert labs. No matter the order i put the dlci’s in i get the same issue it will always want to route over multilink 2 and you loose reachability to half the network. Worst lab ever

    Comment by alden — December 19, 2008 @ 9:13 am | Reply

  6. I also try this, when i was playing with PPPoFR, it just doesn’t work if hub multipoint interface has an ip address. It only work if multipoint interface has ip unnumbered.

    Comment by Jaime Botello — January 7, 2009 @ 6:04 pm | Reply

  7. I thank anyone who wrote on this matter as it really helped me and saved me from banging my head against the wall.

    The problem solver is Ivan Pepelnjak Tip
    indeed unnumbered interfaces will solve the problem

    So I guess even if the ios might allow it sometimes the best rule is never to configure same IP address under 2 different interfaces even if you can

    If you need to use “ip unnumbered”

    Comment by Aly Awad — April 20, 2009 @ 8:19 am | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: