CCIE Pursuit Blog

December 28, 2008

Lab Tip: Start Your Troubleshooting With The Basics

I was racking my brains (what little I have…as you’ll soon see) as to why I couldn’t get my EIGRP adjacencies to come up.  Everything was fine up until I configured rotating keys:

key chain KEY_ROTATION
 key 10
  accept-lifetime 00:00:00 Jan 1 2008 00:15:00 Jan 1 2030
  send-lifetime 00:00:00 Jan 1 2008 00:05:00 Dec 31 2030
 key 20
  accept-lifetime 00:00:00 Jan 1 2030 infinite
  send-lifetime 00:00:00 Jan 1 2030 infinite

I removed and re-added the keys.  I made sure that the clocks on the routers were all set to the same time.  I shut/no shut interfaces.  I added and removed the ‘ip authentication’ commands on the interfaces.  I checked for spaces after the key chain name.  I even took out key 20 so that there was only one key.  Nothing worked.  WTF?!?  This shouldn’t be so difficult.

Then I finally ran a simple verification command that I should have run much, much earlier:

R2#sh key chain KEY_ROTATION
Key-chain KEY_ROTATION:
   * key 10 — text “(unset)”
        accept lifetime (00:00:00 UTC Jan 1 2008) – (00:15:00 UTC Jan 1 2030) [valid now]
        send lifetime (00:00:00 UTC Jan 1 2008) – (00:05:00 UTC Dec 31 2030) [valid now]
  * key 20 — text “(unset)”
        accept lifetime (00:00:00 UTC Jan 1 2030) – (infinite)
        send lifetime (00:00:00 UTC Jan 1 2030) – (infinite)

Notice the “(unset)” bit?  Yup.  I forgot to CONFIGURE THE PASSWORD!!!!  I was concentrating so hard on getting the accept and send lifetimes correct that I forgot to configure the most important bit.

The reason that I’m advertising my idiocy to the world is because it’s important in the lab (and in ‘real life’) to make sure that you verify the basics before you start ripping apart configurations and pulling out your hair.  I seriously wasted about 20 minutes on this mess.  In the lab I would have been under much, much more stress and those 20 minutes (besides being about 5% of my total available lab time) could easily have stretched out longer…or I could have moved on and lost some “easy” points.  😦

January 21, 2008

LFU 14: Menu Lockout

I was working on a task where you need to build a router menu for the NOC.  I added this line to my (existing) menu:

r2(config)#menu NOCMENU clear-screen

I invoked my menu…

r2#menu NOCMENU

…and ended up with a blank screen (which I could not break out of).

I jumped on a different router and telnetted to r2 and found out that I had overwritten my menu with the single line:

r2(config)#do sh run | sec menu
menu NOCMENU clear-screen

No biggie.  I’ll just remove that configuration:

r2(config)#no menu NOCMENU clear-screen
% Menu NOCMENU not deleted. In use

CRAP!!!

What to do?  I thought of a few options:

1) Reload – that would work as I had not written the configuration yet.  But it would waste time.
2) Kill my session on the console port.  That would work, but then I would need to reestablish my reverse telnet from the access server.  This would be a little quicker than reloading.

r2(config)#do sh user
    Line       User       Host(s)              Idle       Location
   0 con 0                idle                 00:06:05
*514 vty 0                idle                 00:00:00 141.1.36.6

I ended up doing this instead:

r2(config)#menu NOCMENU text 1 1 END!!!
r2(config)#menu NOCMENU command 1 exit

I reverse-telnetted back in to r2 via the access server and hit “1”

r2 con0 is now available

Press RETURN to get started.

Sweet!!!! I pressed RETURN as instructed and the curse of the evil menu was lifted!!!

I removed the menu:

r2(config)#no menu NOCMENU
r2(config)#do sh run | sec menu
r2(config)#

December 15, 2007

LFU 13: Know Your Defaults

If you reverse out something, make sure that it’s not a default.

I accidentally configured “logging monitor”.  No problem, I’ll just remove the command with “no logging monitor”.  No problem…except that “logging monitor” is a default command. 

r3(config)#logging mon
r3(config)#no logg mon
r3(config)#do sh run | i logg
no logging monitor   <-oops!!!!
logging trap debugging

By not knowing that “logging monitor” is on by default, I just changed the configuration of my router and I could lose points when the lab is graded.

If you accidentally turn on a function, do a quick “show run | i [function]”.  If it does not show up, then it is a default and you can go on your merry way.  If it does show up, then you will need to back it out.

December 14, 2007

Cisco 3725: How To Boot From External CompactFlash (slot0:)

This post was initially going to be an LFU and a rant along the lines of “WTF does Cisco have external CompactFlash on the 3725 if you can’t boot an image from it?”  I had already spent an embarrassing amount of time and rage figuring out that the external CompactFlash on the 3725 was slot0: and not flash:.  Then I tried to boot the image from slot0: (because I did not have room for that image in flash:), but I ran across the following problem:

r6(config)#boot system ?
  WORD   TFTP filename or URL
  flash  Boot from flash memory
  ftp    Boot from a server via ftp
  mop    Boot from a Decnet MOP server
  rcp    Boot from a server via rcp
  rom    Boot from rom
  tftp   Boot from a tftp server

It seems that you can’t boot from slot0:  I even tried to see if there was a slot0: option under “flash”: 

r6(config)#boot system flash ?
  WORD  System image filename
  <cr>

No such luck.  A quick trip to the InterWebs seemed to indicate that this was probably not possible.

Great.  I dropped a Base IP image into flash: and called it a weekend.  I was planning on experimenting with having the 3725 boot it’s image though TFTP (from a 2800’s flash:).  Luckily for me, Ethan Banks came to my rescue:

I did a “boot system flash slot0:/imagename.bin”, it was all good – the router would boot from the image on the external CF card. I think you should be able to do that, too. Too many things are missing in the base IP image, although maybe you can sneak by with it on this lab, like you said.

Here’s another trick with the 3745s I’m using. I do a “write erase” and “reload” to get the router ready for the next lab. That wipes out everything including the boot variable, and so during that first boot up after I wiped its little mind, the 3745 would autoboot the first image it found, a stale one left over on the system flash. To work around this “no bootvar on initial reload” nuisance, I deleted all other bins off of the 3745 system flash. Now when I boot the 3745 after a write erase, the boot variable is still wiped, but there’s only one image it can possibly autoboot – the one I need on the external CF card.

I followed his advice:

r6(config)#boot system slot0:c3725-adventerprisek9-mz.124-10.bin

and after the reload:

r6#sh ver | i IOS|image
Cisco IOS Software, 3700 Software (C3725-ADVENTERPRISEK9-M), Version 12.4(10), RELEASE SOFTWARE (fc1)
System image file is “slot0:c3725-adventerprisek9-mz.124-10.bin

Hellz yeah!!!

Thanks to Ethan my weekend will begin with lab 3 and not swearing and tinkering – although there probably will be plenty of that during IGP redistribution.  🙂

December 12, 2007

LFU 12: Know Your Flash From A Hole In The Ground

I was using a 2600 router for r6.  It had a single FastEthernet port (with a single serial port).  The Internetwork Expert lab topography calls for 2 Ethernet ports for r6.  So far this has not been a problem as all of the scenarios I have done have only used one Ethernet interface.  Volume II Lab 3 requires 2 Ethernet interfaces, so it was time for me to replace r6. 

I couldn’t find a 2600 with more than one Ethernet interface and using a 2800 seemed like overkill.  I found a bunch of 3725s (a model that I am not familiar with) and one actually still had a Compact Flash card in it.  It was 2 rack units, but at least no one would miss it so I nabbed it.

I downloaded the appropriate IOS image (c3725-adventerprisek9-mz.124-10.bin) from CCO and started my TFTP server:

r6#sh flash: all
Partition   Size    Used      Free      Bank-Size  State          Copy Mode
  1        31167K      0K    31167K        0K      Read/Write     Direct

System CompactFlash directory:
No files in System CompactFlash
[0 bytes used, 31916028 available, 31916028 total]
31168K bytes of ATA System CompactFlash (Read/Write)

 Chip information NOT available.

r6#
*Mar  1 00:29:29.587: %FILESYS-5-CF: External CompactFlash removed
Flash card inserted in slot0. Reading filesystem on the device…
Wait for the completion message before accessing device
Filesystem read completed in slot0.
Device in slot0 available for use

r6#copy tftp flash:
Address or name of remote host [10.1.1.100]?
Source filename [c3725-adventerprisek9-mz.124-10.bin]?
Destination filename [c3725-adventerprisek9-mz.124-10.bin]?
Accessing tftp://10.1.1.100/c3725-adventerprisek9-mz.124-10.bin…
Erase flash: before copying? [confirm]
Erasing the flash filesystem will remove all files! Continue? [confirm]
Erasing device… eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
eeeeeeeeeeeeeeeeeeeeeeeee …erased
Erase of flash: complete
Loading c3725-adventerprisek9-mz.124-10.bin from 10.1.1.100 (via FastEthernet0/0): !
%Error copying tftp://10.1.1.100/c3725-adventerprisek9-mz.124-10.bin (Not enough space on device)

WTF???  I have a blank 64M flash card and it can’t fit a 36M bin file?

I went and found a 128M card and I got the same error.

r6#sh flash: all
Partition   Size    Used      Free      Bank-Size  State          Copy Mode
  1        31167K      0K    31167K        0K      Read/Write     Direct

System CompactFlash directory:
No files in System CompactFlash
[0 bytes used, 31916028 available, 31916028 total]
31168K bytes of ATA System CompactFlash (Read/Write)

 Chip information NOT available.

Question:  Why is there only 31M available on an empty 128M flash card? 
Answer:Cuz I is dumb…dumb as dirt.

The router did its best to inform my stupid ass that the card was in SLOT0, but I just assumed that flash: meant external CompactFlash.  If I would have bothered to look a little closer at the IOS output I would have seen that the external CompactFlash card was in slot0 and that the CompactFlash in flash: was systemCompactFlash.  I would have saved myself some time and wiped that dumb look off my face if I would have taken just a little time to read.  Well, I would have saved some time…the dumb look is permanent.  🙂

r6#sh slot0:
-#- –length– ———date/time——— path
1     21228464 Mar 01 1993 02:49:42 +00:00 c3725-js-mz.123-9a.bin
2     22889736 Mar 01 1993 18:20:48 +00:00 c3725-adventerprisek9-mz.123-9a.bin
3      3207168 Mar 01 1993 00:16:52 +00:00 sdm.tar
4      1452032 Mar 01 1993 00:33:16 +00:00 qdm.tar

r6#copy tftp slot0:
Address or name of remote host [10.1.1.100]?
Source filename [c3725-adventerprisek9-mz.124-10.bin]?
Destination filename [c3725-adventerprisek9-mz.124-10.bin]?
Accessing tftp://10.1.1.100/c3725-adventerprisek9-mz.124-10.bin…
Erase slot0: before copying? [confirm]
Erasing the slot0 filesystem will remove all files! Continue? [confirm]
Current DOS File System flash card in slot0: will be formatted into Low End File
 System flash card!  Continue? [confirm]
Erasing device… eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee …erased
Erase of slot0: complete
Loading c3725-adventerprisek9-mz.124-10.bin from 10.1.1.100 (via FastEthernet0/0
): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[OK – 36934872 bytes]

r6#sh slot0

Slot0 CompactFlash directory:
File  Length   Name/status
  1   36934872  c3725-adventerprisek9-mz.124-10.bin
[36934936 bytes used, 27159268 available, 64094204 total]
62592K bytes of ATA Slot0 CompactFlash (Read/Write)

Remember to change your boot statement…which leads to my next LFU.

Make sure your router can boot from compact flash.  I could not get this stupid 3725 to boot from slot0.  I finally gave up and dropped an IP base image into flash: and booted to that.  Hopefully there’s no IPv6 on r6.  :0

December 5, 2007

LFU 11: Interfaces Can Have Multiple IPv6 Addresses

I ran into an issue this weekend with an IPv6 address that I thought that I had removed showing up in my routing table. 

r1(config)#ipv6 unicast-routing
r1(config)#int s1/0
r1(config-if)#ipv6 address 2001::11/64
r1(config-if)#do sh ipv6 int br | e down|unass
Serial1/0                  [up/up]
    FE80::CE00:12FF:FE10:0
    2001::11

Oh crap!  I wanted to configure 2001::1/64 not 2001::11/64.

r1(config-if)#ipv6 address 2001::1/64
r1(config-if)#do sh ipv6 int br | e down|unass
Serial1/0                  [up/up]
    FE80::CE00:12FF:FE10:0
    2001::1
    2001::11

r1(config-if)#do sh run int s1/0
Building configuration…

Current configuration : 262 bytes
!
interface Serial1/0
 ip address 10.0.0.1 255.0.0.0
 encapsulation frame-relay
 ipv6 address 2001::1/64
 ipv6 address 2001::11/64
 serial restart-delay 0
 no dce-terminal-timing-enable
 frame-relay map ip 10.0.0.2 102 broadcast
 no frame-relay inverse-arp
end

Unlike IPv4, configuring a different IPv6 address on an interface will NOT overwrite the existing IPv6 address with the new IPv6 address.  This is because an interface can have multiple IPv6 addresses.

You can use the “no ipv6 address” to remove all IPv6 addreses on an interface.  Then you can configure the new IPv6 address.

interface Serial1/0
 ip address 10.0.0.1 255.0.0.0
 encapsulation frame-relay
 ipv6 address 2001::1/64
 ipv6 address 2001::11/64
 ipv6 address 2001::111/64
 ipv6 address 2001::1111/64
 serial restart-delay 0
 no dce-terminal-timing-enable
 frame-relay map ip 10.0.0.2 102 broadcast
 no frame-relay inverse-arp
end

r1(config-if)#no ipv6 address
r1(config-if)#do sh run int s1/0
Building configuration…

Current configuration : 211 bytes
!
interface Serial1/0
 ip address 10.0.0.1 255.0.0.0
 encapsulation frame-relay
 serial restart-delay 0
 no dce-terminal-timing-enable
 frame-relay map ip 10.0.0.2 102 broadcast
 no frame-relay inverse-arp
end

December 1, 2007

LFU 10: No VLAN…No CDP!!!

I was working on a practice lab today and was using CDP to verify that all of the connections were correct.  I cruised through sw1 and sw2 but when I hit sw3 I started to see strange issues with CDP.
 
sw3 fa0/3 is connected to r3 fa0/1 but I cannot see a CDP neighbor:
 
sw3#sh run int fa0/3
Building configuration…
 
Current configuration : 95 bytes
!
interface FastEthernet0/3
 switchport access vlan 33
 switchport mode dynamic desirable
end
 
Port      Name               Status       Vlan       Duplex  Speed Type
Fa0/3                        connected    33         a-full  a-100 10/100BaseTX
 
sw3#sh int fa0/3
FastEthernet0/3 is up, line protocol is up (connected)
 
sw3#sh cdp neigh fa0/3
Capability Codes: R – Router, T – Trans Bridge, B – Source Route Bridge
                  S – Switch, H – Host, I – IGMP, r – Repeater, P – Phone
 
Device ID            Local Intrfce         Holdtme   Capability    Platform   Port ID
sw3#

The interface is up/up and CDP is running (it would have given me an error if it was not) but CDP does not see r3’s fa0/1 interface. 

Let’s look at r3:

r3#sh run int fa0/1
Building configuration…
 
Current configuration : 95 bytes
!
interface FastEthernet0/1
 ip address 132.1.33.3 255.255.255.0
 duplex auto
 speed auto
end
 
r3#sh ip int br | i net0/1
FastEthernet0/1
            132.1.33.3      YES manual up                    up
 
r3#sh cdp neigh fa0/1
Capability Codes: R – Router, T – Trans Bridge, B – Source Route Bridge
                  S – Switch, H – Host, I – IGMP, r – Repeater
 
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
r3#

r3’s fa0/1 interface is up and up and CDP is running.  I can actually physically trace that cable to sw3’s fa0/3 interface.  So why can’t sw3 see r3 (and vice versa) via CDP? 

Here’s the problem:
 
sw3#sh int fa0/3 switch
Name: Fa0/3
Switchport: Enabled
Administrative Mode: dynamic desirable
Operational Mode: static access
Administrative Trunking Encapsulation: negotiate
Operational Trunking Encapsulation: native
Negotiation of Trunking: On
Access Mode VLAN: 33 (Inactive)
Trunking Native Mode VLAN: 1 (default)
Administrative Native VLAN tagging: enabled
Voice VLAN: none

I configured the port as an accesss port assigned to VLAN 33, but VLAN 33 does not exit on the switch (it will once VTP does it’s magic).  Once VLAN 33 is configured on the switch CDP will work:

sw3#sh vlan id 33

VLAN Name                             Status    Ports
—- ——————————– ——— ——————————-
33   VLAN0033                         active    Fa0/3, Po2

VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
—- —– ———- —– —— —— ——– —- ——– —— ——
33   enet  100033     1500  –      –      –        –    –        0      0

Remote SPAN VLAN
—————-
Disabled

Primary Secondary Type              Ports
——- ——— —————– ——————————————

sw3#sh int fa0/3 switch
Name: Fa0/3
Switchport: Enabled
Administrative Mode: dynamic desirable
Operational Mode: static access
Administrative Trunking Encapsulation: negotiate
Operational Trunking Encapsulation: native
Negotiation of Trunking: On
Access Mode VLAN: 33 (VLAN0033)
Trunking Native Mode VLAN: 1 (default)
Administrative Native VLAN tagging: enabled
Voice VLAN: none

sw3#sh cdp neigh fa0/3
Capability Codes: R – Router, T – Trans Bridge, B – Source Route Bridge
                  S – Switch, H – Host, I – IGMP, r – Repeater, P – Phone

Device ID            Local Intrfce         Holdtme   Capability    Platform   Port ID
r3                  Fas 0/3               179           R S I     2651XM    Fas0/1

November 26, 2007

LFU 9: Beware The Implied OSPF Process!!!

I ran into an interesting issue on the Internetwork Expert Volume II lab 2 this weekend.  I noticed that I had a rogue OSPF process showing up on one of my switches (3560).  I had configured OSPF process 100, but not OSPF process 17 (17 wasthe area that the sw1 OSPF interfaces were in):

sw1#sh ip proto sum
Index Process Name
0     connected
1     static
2     ospf 100
3     rip
4     ospf 17   <-where did this come from?
*** IP Routing is NSF aware ***

sw1#sh ip proto | b ospf
Routing Protocol is “ospf 100”
  Outgoing update filter list for all interfaces is not set
  Incoming update filter list for all interfaces is not set
  Router ID 150.1.7.7
  It is an autonomous system boundary router
  Redistributing External Routes from,
    rip, includes subnets in redistribution
  Number of areas in this router is 1. 1 normal 0 stub 0 nssa
  Maximum path: 4
  Routing for Networks:
    132.1.17.7 0.0.0.0 area 17
  Routing Information Sources:
    Gateway         Distance      Last Update
    150.1.3.3            110      00:18:43
    150.1.2.2            110      00:18:43
    150.1.1.1            110      01:44:44
    150.1.7.7            110      01:44:44
  Distance: (default is 110)

There was no other OSPF process configure.  I thought that maybe I accidentally configured “router os 17” at some point, but when I looked at the configuration all that was there was OSPF 100.  I thought that there must be some bug in the switch IOS.  The phantom OSPF process was showing up in the “show ip protocol summary” output but not in the “show ip protocol” output.  I reloaded the switch just in case.

That didn’t solve the issue.  I looked through the configuration and found the issue…under the RIP process!  This solves the mystery of the “ospf 17”:

sw1#sh run | b router rip
router rip
 version 2
 redistribute ospf 17 metric 1   <-whoops!  
 offset-list EVEN in 16 Vlan783
 network 150.1.0.0
 network 204.12.1.0
 no auto-summary

This explains my route redistribution issue as well!  I must have typed in the area number instead of the correct OSPF process (100) when redistributing OSPF into RIP.  This is the type of fat-finger issue that can make you fail your lab.  I banged my head on my desk and corrected the process ID.

sw1#sh run | b router rip
router rip
 version 2
 redistribute ospf 100 metric 1    
 offset-list EVEN in 16 Vlan783
 network 150.1.0.0
 network 204.12.1.0
 no auto-summary

Strange, IOS must assume the existence (or imminent existence) of a routing process if you reference it in a redistribution statement.  What’s even stranger is that even after I removed (changed) the reference…it still showed up:

sw1#sh ip proto sum
Index Process Name
0     connected
1     static
2     ospf 100
3     rip
4     ospf 17
*** IP Routing is NSF aware ***

WTF?  Was I being haunted by an undead OSPF process?  I verified that there was nothing left in the configuration that referred to OSPF process 17:

sw1#sh run | i 17
 ip address 132.1.17.7 255.255.255.0
interface FastEthernet0/17
 area 17 authentication
 network 132.1.17.7 0.0.0.0 area 17

So the process was not configured and there was no longer any reference to it, so why was it still showing up?

sw1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
sw1(config)#no router ospf 17
sw1(config)#
sw1(config)#do sh ip proto sum
Index Process Name
0     connected
1     static
2     ospf 100
3     rip
*** IP Routing is NSF aware ***

Lesson learned: if you reference an OSPD process that does not exist (in RIP at least) then the router assumes that this process is actually running.  You’ll need to explicitly remove the process (“no router ospf x”) in order to remove it from the “show ip protocol summary” output.

November 17, 2007

LFU 8: Read The Prompts!!!

ARRGHH!!!!!  Today I was saving out a running configuration on a 2651XM to flash.  Very simple thing to do.  I had plenty of space in flash:

r1#sh flash:

System flash directory:
File  Length   Name/status
  1   29631128  c2600-adventerprisek9-mz.124-10.bin
[29631192 bytes used, 20176164 available, 49807356 total]
49152K bytes of processor board System flash (Read/Write)

I went ahead and copied the running configuration to flash, but I neglected to read the prompts carefully before hitting the “enter” key:

r1#copy run flash:r1_basic_cfg
Destination filename [r1_basic_cfg]?
Erase flash: before copying? [confirm]
Erasing the flash filesystem will remove all files! Continue? [confirm] <-DOH!!!!
Erasing device… eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

I didn’t figure on melting the entire box.  There was plenty of room in flash for the config.  I just hit enter one too many times!!!

The correct way:
r3#copy run flash:r3_basic_cfg
Destination filename [r3_basic_cfg]?
Erase flash: before copying? [confirm]n
Verifying checksum…  OK (0x1EC9)
1802 bytes copied in 4.707 secs (383 bytes/sec)

r3#sh flash:

System flash directory:
File  Length   Name/status
  1   29631128  c2600-adventerprisek9-mz.124-10.bin
  2   1802     r3_basic_cfg
[29633060 bytes used, 20174296 available, 49807356 total]
49152K bytes of processor board System flash (Read/Write)

Back to R1:

r1#sh ver | i IOS
Cisco IOS Software, C2600 Software (C2600-ADVENTERPRISEK9-M), Version 12.4(10), RELEASE SOFTWARE (fc1)

r1#sh flash:

System flash directory:
File  Length   Name/status
  1   1224     r1_basic_cfg
[1288 bytes used, 49806068 available, 49807356 total]
49152K bytes of processor board System flash (Read/Write)

So I have my config file, but no IOS.  At least this should be an easy fix.

I have the appropriate IOS running on r3:

r3#sh ver | i IOS
Cisco IOS Software, C2600 Software (C2600-ADVENTERPRISEK9-M), Version 12.4(10), RELEASE SOFTWARE (fc1)

I have connectivity to r3 via a PTP connection. I just need to set r3 up to be a tftp server for that image:

r3(config)#tftp-server flash:c2600-adventerprisek9-mz.124-10.bin

Now I need to copy that image to flash on r1:

r1#copy tftp flash:
Address or name of remote host []? 155.1.13.3
Source filename []? c2600-adventerprisek9-mz.124-10.bin
Destination filename [c2600-adventerprisek9-mz.124-10.bin]?
Accessing tftp://155.1.13.3/c2600-adventerprisek9-mz.124-10.bin…
Erase flash: before copying? [confirm]n <-learn from your mistakes!!!
Loading c2600-adventerprisek9-mz.124-10.bin from 155.1.13.3 (via Serial0/1):
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[OK – 29631128 bytes]

Verifying checksum…  OK (0xC7EA)
29631128 bytes copied in 753.755 secs (39311 bytes/sec)

After getting a cup of coffee (or 3 cups – I have the PTP link set at 64K)  , everything is  wonderful again on r1:

r1#sh flash:

System flash directory:
File  Length   Name/status
  1   1224     r1_basic_cfg
  2   29631128  c2600-adventerprisek9-mz.124-10.bin
[29632480 bytes used, 20174876 available, 49807356 total]
49152K bytes of processor board System flash (Read/Write)

Configure the boot statement and reload:

r1(config)#boot system flash:c2600-adventerprisek9-mz.124-10.bin

 

November 8, 2007

LFU 7: RTFM

Last night (actually early this morning due to my inability to translate time zones) I rented some time on Internetwork Expert’s racks.  I was assigned to rack 6.  I logged on and after doing some of the technology labs, I decided to do as much of Lab 1 from the Volume III workbook as time allowed.  I had already done the layer 2 part of this lab on my rack and figured that it was good practice to do it again and see how far I could get through the IGP configuration before I fell asleep.

One of the great things about renting a vendor’s rack is that they are set up precisely for their workbooks, right down to the interface.  On my rack I need to constantly tweak vendor provided initial configurations to match my actual interfaces.  It’s not a huge deal, but it does burn up some time when setting up for a lab.  On IE’s rack I was able to quickly load the initial configurations and start on the lab.

I tore through the layer 2 stuff.  A lot of that had to do with this being the second time that I’ve done this lab, but also because the tasks are pretty basic and unambiguous. 

I was zipping along until I hit task 3.2  In this task you’re asked to create a point-to-point Frame Relay connection from r6 to bb1 (backbone router).  Pop on an IP address and configure Frame Relay.  Piece of cake.   That’s when the fun began:

I could not ping bb1 from r6.  I configured it correctly and I was getting a good frame map:

r6#p 54.1.2.254

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 54.1.2.254, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5) 

r6#sh run | sec Serial0/0
interface Serial0/0
 no ip address
 encapsulation frame-relay
interface Serial0/0.1 point-to-point
 ip address 54.1.2.6 255.255.255.0
 frame-relay interface-dlci 100

r6#sh frame map
Serial0/0.1 (up): point-to-point dlci, dlci 100(0x64,0x1840), broadcast
          status defined, active

Weird.  I did a shut/no shut on the physical interface and was still unable to ping the bb1.  I jumped on the access server and tried to reverse telnet to bb1 but was not allowed to access it.  At this point I grabbed the answer key to verify my configuration.  I felt stupid doing this because the task was so simple and I had successfully completed it before on my own rack.  My configuration matched the answer key.  So what the hell was going on?

I verified that Frame Relay was working.  LMI, Frame Maps, Frame PVCs, interfaces were up and up…everything looked good.  Routing protocols had not been configured at this point, so it HAD to be something with the layer 2 configuration.  But what?

I had a couple of troubleshooting routes that I could take a this point.  I was leaning towards something being configured wrong on the bb1, so I decided to default r6’s s0/0 interface and then use Frame Relay Inverse-ARP on the physical interface to create a dynamic mapping.  If successful, that should allow me to verify the DLCI and IP address of bb1:

This is after I defaulted and rebuilt the s0/0 interface:
r6#sh run int s0/0
Building configuration…

Current configuration : 89 bytes
!
interface Serial0/0
 ip address 54.1.2.6 255.255.255.0
 encapsulation frame-relay
end

r6#sh ip int br | i Serial0/0
Serial0/0                  54.1.2.6        YES manual up                    up

Serial0/0.1                unassigned      YES manual deleted               down

r6#sh frame map
Serial0/0 (up): ip 54.6.1.254 dlci 101(0x65,0x1850), dynamic,
              broadcast,, status defined, active
Serial0/0 (up): ip 54.6.2.254 dlci 100(0x64,0x1840), dynamic,
              broadcast,, status defined, active
Serial0/0 (up): ip 54.6.3.254 dlci 51(0x33,0xC30), dynamic,
              broadcast,, status defined, active

Fuck me running!  The IP address on bb1 was wrong.  The second octet is 6 and should be 1:

Serial0/0 (up): ip 54.6.2.254 dlci 100(0x64,0x1840), dynamic,
              broadcast,, status defined, active

r6#sh ip int br | i Serial0/0
Serial0/0                  54.1.2.6        YES manual up                    up

My configuration matched the topology and the answer key.  I opened the initial configuration for the bb1 from Internetwork Expert’s own documentation (remember that I could not access the bb1):

interface Serial0.100 point-to-point
 description PVC 100 to Rack8
 ip address 54.1.2.254 255.255.255.0 
 ipv6 address 2001:54:1:2::254/64
 ipv6 address FE80::254 link-local
 frame-relay interface-dlci 100

Why hath thou forsaken me Internetwork Expert?  I was crushed.  I was up way past my bedtime and looking at only a couple hours of sleep before the roar of my alarm woke me to face a day at work with minimal sleep.  I would spend my few hours of sleep plotting the slow and painful demise of the Brians.  🙂

I went ahead and altered R6’s s0/0 interface IP and everything worked just fine.

I have learned a couple of things about myself over the years:

  1. I am often wrong.
  2. If I wait a bit and keep my big mouth shut, I’ll usually see the error of my ways.

Unfortunately, I usually skip step two and end up with my foot in my mouth.  In this case I thankfully saw the error of my ways before emailing IE complaining about the incorrect configuration of the bb1.

If you’ve used IE’s labs before, you’ll know that the IP address on the topology are written in the form of “10.x.1.0/24” where x = your rack number.  I have always used x = 1.  This has never been a problem because I use my own rack most of the time.  When I have rented racks before, it’s never been an issue either because I’ve never used the bb routers before. 

In this case I was on rack 6 and had pasted in the initial configurations which used x=1 (this negates my “initial configs need no editing” statement from earlier).  This meant that IE had the bbs configured correctly (x = 6) but that ALL of my interfaces were configured incorrectly (x = 1).

ARRGHH!!!!!!!

I went ahead and changed the octet only on the connections to the bb routers.  I was getting pretty tired and it was obvious that I was not going to complete the entire lab before passing out, so I did not worry about any issues caused by the bb-injected routes using x=6.

In the end I was disappointed that I spent so much time troubleshooting this issue, but I am happy that I was able to troubleshoot the issue and eventually find the underlying problem (my idiocy).

The lesson: Read the instructions and don’t get so comfortable with a topology or routine that you don’t think/question why you’re doing something a certain way.  If this had happened to me on the actual lab, it would have sunk me.

Next Page »

Blog at WordPress.com.