Rack Hell

Want to know what I absolutely hate?  Trying to cobble together a lab.

I decided to finally install the BB routers (backbone routers used to inject routes into the lab) in my home lab.  I had planned on adding a 2600 and using the existing Frame Relay switch (Cisco 2851) and access server (2511) for the 3 BB routers.  I quickly had to chuck the idea of using the 2511 as I don’t have an AUI adapter so I couldn’t use its Ethernet port.  Also – as I would painfully learn later – 14 megs of RAM and 8 megs of flash was not going to allow me to load an IOS image capable of running IPv6.

The good news is that the Frame Relay switch (2851) worked perfectly as BB3 as I had two open Ethernet (gig ports) ports.  I did have to upgrade the IOS to the Advanced Enterprise code.  The 2851 has shitloads of RAM and flash so that was painless as far as upgrades go. 

As I said, my original plan was to use my existing Frame Relay switch (check), my access server (no go), and to use an existing 2620 that I had set up as R7 (I’ve never actually used it).  If everything went right, I would not need to add any more devices to my stack.

The access server was out, so I needed to add at least one more device.  I started tackling the existing 2620.  No luck.  It didn’t have enough memory to run code capable of IPv6 routing protocols.  Internetwork Expert does have a tutorial on how to  boot XM images from a non-XM 2600, but I decided to raid our storeroom and see if I could find a 2651XM instead.

No luck on the 2651XM, but I did find a 2620XM.  Sweet!!!  I was able to load 12.4(10) Advanced Services code.  I pasted in the IE config and everything was going great until I tried to configure BGP.

Oh fucking joy!!!  An hour in and I get this mess:

14:28:00: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0×80067BB4, alignment 0
Pool: Processor  Free: 64496  Cause: Not enough free memory
Alternate Pool: None  Free: 0  Cause: No Alternate pool
 -Process= “Exec”, ipl= 0, pid= 3 -Traceback= 0×8114609C 0×80055F1C 0×80058150 0×80058994 0×80067BB8 0×80067784 0×80D967E8 0×8281364C 0×80D97998 0×818B248C 0×818B3B58 0×81186C94 0×81185F70 0×811A5C50 0×802405F8 0×80243CE4
bb1(config-router)# !
14:28:03: %SYS-2-CFORKMEM: Process creation of BGP Open failed (no memory).-Process= “BGP Router”, ipl= 0, pid= 215 -Traceback= 0×8114609C 0×821FAE24 0×82215260 0×80D92EC4 0×802405F8 0×80243CE4

bb1(config-router)#do sh mem
                          Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor   848C91A0    16389232    16388652         580         408         264
      I/O    5700000     9437184     1736328     7700856     7700856     7700828

bb1(config)#do sh ver | i emo
Cisco 2620XM (MPC860P) processor (revision 1.0) with 89088K/9216K bytes of memory.

Great. Back to the drawing board.

There were no more suitable 2600s available (just a few 2620s).  We don’t use those devices except as an occasionally dial up router for out of band management at remote sites.  We do have tons of 2800 routers available, but they are larger, heavier, and louder than the 2600s so I try to avoid them.  Plus they could be used in our actual network, so I run the (unlikely) risk of someone pulling one from my stack for production use.  PLUS, it is hard as hell to find one with the flash card still in the chasis.  That said, I was looking at adding two 2800s to my stack.  Crap.

We have tons of extra 3750 switches (especially the non-PoE ones).  Since BB2 really only needed to have a single Ethernet port, I decided to try using a 3750.  I found an old 12 port non-PoE 3750 and loaded the 12.2(SEE) Advanced Services code.  Voila!  BB2 was up and running and only using one rack unit!  Score one for the home team.

I finally had to capitulate and use a 2800 for BB1.

The 2800 that I grabbed did not have a flash card in it, so I hunted down a flash card and my old buddy rommon greeted me:

rommon 1 > boot
program load complete, entry point: 0×8000f000, size: 0xc5a0

Initializing ATA monitor library…….
program load complete, entry point: 0×8000f000, size: 0xc5a0

Initializing ATA monitor library…….
loadprog: error – Invalid image for platform
e_machine = 134, cpu_type = 135
boot: cannot load “flash:”  <-doh!!!

Booting did not work, I had better give it more precise directions.  Let’s see what IOS versions we have on the flash:

rommon 2 > dir flash:  <-note: the “:” is important.  rommon will note recognize “flash”
program load complete, entry point: 0×8000f000, size: 0xc5a0

Initializing ATA monitor library…….
Directory of flash:

2      21628224  -rw-     c1841-advipservicesk9-mz.123-8.T6.bin
5283    1536      -rw-     sdmconfig-18xx.cfg
5284    3885056   -rw-     sdm.tar
6233    1463      -rw-     home.html
6234    270848    -rw-     home.tar
6301    93095     -rw-     attack-drop.sdf
6324    1187840   -rw-     ips.tar
6614    34165604  -rw-     c2800nm-spservicesk9-mz.124-13b.bin

Let’s try the newer version:

rommon 7 > boot flash:c2800nm-spservicesk9-mz.124-13b.bin
program load complete, entry point: 0×8000f000, size: 0xc5a0

Initializing ATA monitor library…….

program load complete, entry point: 0×8000f000, size: 0×20951c4
Self decompressing the image : #################################################
################################################################################
######################################################## [OK]

Sweet.  I was in.  I did a write erase and added configured “boot system flash:” and used the handy-dandy autocomplete feature to fill in the name of the version that I wanted to boot.  Unfortunately, I didn’t look closely at what IOS had autocompleted [verb?] and I booted to the wrong code:

boot system flash:c2800nm-spservicesk9-mz.124-13b.bin

No problem, I’ll just configure the router to use the correct code:

bb1(config)#boot system flash:c2800nm-adventerprisek9-mz.124-11.T2.bin

I reloaded and then tried to add some IPv6 statements.  The 2800 told me to take my IPv6 and go get bent.

ARGHH!!!! 

I just assumed (NEVER ASSUME!) that there could only be one “boot system [file]” statement per router.  WRONG!!!

bb1(config)#do sh run | i boot
boot-start-marker
boot system flash:c2800nm-spservicesk9-mz.124-13b.bin
boot system flash:c2800nm-adventerprisek9-mz.124-11.T2.bin

That explains why I wasn’t able to run IPv6 after I “updated” the boot file:

bb1(config)#do sh ver | i IOS
Cisco IOS Software, 2800 Software (C2800NM-SPSERVICESK9-M), Version 12.4(13b), RELEASE SOFTWARE (fc3) <-not Advanced Enterprise IOS

Trimming the config down to the correct boot parameter did the trick…but MORE TIME WASTED!!!

I finally did get the BB routers up and configured.  I only ended up adding one additional device to my stack because I was able to use the existing Frame switch and to remove R7 (2620).  I added the 3750 and the 2851.  I now have a stack of 14 devices (17 rack units) on my desk, BUT my lab is now ready for use with Internetwork Expert practices labs.  Of course I also ended up wasting most of the time that I had allotted for labbing this weekend.  I am still going to chalk this time up to lab time even though very little of it was spent doing actual labs.