CCIE Pursuit Blog

May 21, 2007

A Series Of Unfortunate Router Events: Part II

Filed under: Home Lab,IOS — cciepursuit @ 7:06 pm

In my last post I found out the hard way about the need to verify the memory capabilities of a router before installing IOS code.  Well, now I was stuck in ROMMON mode with an IOS version that would not boot.  I needed to replace that version of IOS with the known good version that I had running on other routers in my lab…AND I had to do it from ROMMON!

Luckily, it’s pretty easy to accomplish this on a 2600.  The steps are detailed in this document:

ROMmon Recovery for the Cisco 2600 Series Router and the VG200

I had to replace the c2600-adventerprisek9-mz.124-10.bin IOS file in flash memory with the old (working) IOS file c2600-is-mz.123-22.bin

rommon 3 > dir flash:
         File size           Checksum   File name
  29631128 bytes (0x1c42298)  0xc7ea    c2600-adventerprisek9-mz.124-10.bin

I was running the old code version on r3.  I first needed to configure that router to act as a tftp server:

Verify the code version and set up router as tftp server:
r3#sh ver
Cisco Internetwork Operating System Software
IOS ™ C2600 Software (C2600-IS-M), Version 12.3(22), RELEASE SOFTWARE (fc2)

System image file is “flash:c2600-is-mz.123-22.bin”

r3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
r3(config)#tftp-server flash:?
flash:c2600-is-mz.123-22.bin

r3(config)#tftp-server flash:
r3(config)#tftp-server flash:c2600-is-mz.123-22.bin
r3(config)#^Z

Now I needed to configure the first ethernet port on the disabled router so that it was on the same subnet as r3 (10.1.1.0/24 in this case).  You can verify the current variables in ROMMON with the “set” command:

rommon 5 > set
PS1=rommon ! >
RET_2_RUTC=1063924721
BOOT=
BSI=0
RET_2_RTS=
RET_2_RCALTS=
?=0

I read the Cisco documentation and set the appropriate variables (see if you can catch my mistake):

rommon 6 > IPADDRESS=10.1.1.2
rommon 7 > IP_SUBNET_MASK=255.255.255.0
rommon 8 > DEFAULT_GATEWAY=10.1.1.1
rommon 9 > TFTP_SERVER=10.1.1.1
rommon 10 > TFTP_FILE=c2600-is-mz.123-22.bin

Okay, let’s use the “set” command to verify that everything is configured correctly:

rommon 11 > set
PS1=rommon ! >
RET_2_RUTC=1063924721
BOOT=
BSI=0
RET_2_RTS=
RET_2_RCALTS=
?=0
IPADDRESS=10.1.1.2
IP_SUBNET_MASK=255.255.255.0
DEFAULT_GATEWAY=10.1.1.1
TFTP_SERVER=10.1.1.1
TFTP_FILE=c2600-is-mz.123-22.bin

Looks good to me.  Let’s download the IOS file from r3 and fix this router:

rommon 12 > tftpdnld

Missing or illegal ip address for variable IP_ADDRESS
Illegal IP address.

usage: tftpdnld [-r]
  Use this command for disaster recovery only to recover an image via TFTP.
  Monitor variables are used to set up parameters for the transfer.
  (Syntax: “VARIABLE_NAME=value” and use “set” to show current variables.)
  “ctrl-c” or “break” stops the transfer before flash erase begins.

  The following variables are REQUIRED to be set for tftpdnld:
            IP_ADDRESS: The IP address for this unit
        IP_SUBNET_MASK: The subnet mask for this unit
       DEFAULT_GATEWAY: The default gateway for this unit
           TFTP_SERVER: The IP address of the server to fetch from
             TFTP_FILE: The filename to fetch

  The following variables are OPTIONAL:
          TFTP_VERBOSE: Print setting. 0=quiet, 1=progress(default), 2=verbose
      TFTP_RETRY_COUNT: Retry count for ARP and TFTP (default=7)
          TFTP_TIMEOUT: Overall timeout of operation in seconds (default=7200)
         TFTP_CHECKSUM: Perform checksum test on image, 0=no, 1=yes (default=1)
         FE_SPEED_MODE: 0=10/hdx, 1=10/fdx, 2=100/hdx, 3=100/fdx, 4=Auto(deflt)

  Command line options:
   -r: do not write flash, load to DRAM only and launch image

WTF???  You have to make sure that you enter the variables EXACTLY as required.  I missed the “underbar” character in “IP_ADDRESS”.  The mistyped variable will show up in the “set” output.  ROMMON does not hold your hand like IOS does.  I typed the variable correctly and verified all of the variables again with the “set” command:

rommon 13 > IP_ADDRESS=10.1.1.2
rommon 14 > set
PS1=rommon ! >
RET_2_RUTC=1063924721
BOOT=
BSI=0
RET_2_RTS=
RET_2_RCALTS=
IP_ADDRESS=10.1.1.2
IP_SUBNET_MASK=255.255.255.0
DEFAULT_GATEWAY=10.1.1.1
TFTP_SERVER=10.1.1.1
TFTP_FILE=c2600-is-mz.123-22.bin
?=1

Okay.  Let’s try the “tftpdnld” command again:

rommon 15 > tftpdnld

          IP_ADDRESS: 10.1.1.2
      IP_SUBNET_MASK: 255.255.255.0
     DEFAULT_GATEWAY: 10.1.1.1
         TFTP_SERVER: 10.1.1.1
           TFTP_FILE: c2600-is-mz.123-22.bin

Invoke this command for disaster recovery only.
WARNING: all existing data in all partitions on flash will be lost!
Do you wish to continue? y/n:  [n]:  y
….
ARP: address resolution for 10.1.1.1 timed out.
ARP failed with failure code 1.  TFTP transfer aborted.

TFTP: Operation terminated prematurely.

WTF Part II!!!  I had the routers each connected to a switch.  I directly connected the routers’ fa0/0 ports with a crossover cable and tried the “tftpdnld” again:

rommon 16 > tftpdnld

          IP_ADDRESS: 10.1.1.2
      IP_SUBNET_MASK: 255.255.255.0
     DEFAULT_GATEWAY: 10.1.1.1
         TFTP_SERVER: 10.1.1.1
           TFTP_FILE: c2600-is-mz.123-22.bin

Invoke this command for disaster recovery only.
WARNING: all existing data in all partitions on flash will be lost!
Do you wish to continue? y/n:  [n]:  y

Receiving c2600-is-mz.123-22.bin from 10.1.1.1 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
File reception completed.
Copying file c2600-is-mz.123-22.bin to flash.
Erasing flash at 0x60a00000 <- This hex value will change.  This process takes awhile (15 minutes in the case of this 2620)

Eventually you’ll get a ROMMON prompt.  It’s time to reload the router.  How do you reload in ROMMON?  Use “i” for “initialize”:

rommon 17 > i

System Bootstrap, Version 12.2(10r)1, RELEASE SOFTWARE (fc1)
TAC Support: http://www.cisco.com/tac
Copyright (c) 2002 by cisco Systems, Inc.
C2600 platform with 65536 Kbytes of main memory

program load complete, entry point: 0x80008000, size: 0x1134810
Self decompressing the image : #################################################

r2#sh ver
Cisco Internetwork Operating System Software
IOS ™ C2600 Software (C2600-IS-M), Version 12.3(22), RELEASE SOFTWARE (fc2)

ROM: System Bootstrap, Version 12.2(10r)1, RELEASE SOFTWARE (fc1)
ROM: C2600 Software (C2600-IS-M), Version 12.3(22), RELEASE SOFTWARE (fc2)

r2 uptime is 1 minute
System returned to ROM by reload
System image file is “flash:c2600-is-mz.123-22.bin”

What a beautiful sight!!!  My router has been “unmelted”!

Documentation:

ROMmon Recovery for the Cisco 2600 Series Router and the VG200

ROMMON Commands

Advertisements

7 Comments »

  1. […] So I’ve discoved how I managed to screw up my router, now how do I fix it?  That is the topic of my next post. […]

    Pingback by A Series Of Unfortunate Router Events: Part I « CCIE Pursuit — May 21, 2007 @ 7:09 pm | Reply

  2. I’m sorry, I can’t help chuckling at your two WTF’s 🙂

    But it was a very instructional experience, for me also. Thanks for sharing 🙂

    Comment by Pandu E Poluan — February 11, 2008 @ 2:12 am | Reply

  3. Well, i found your blog searching to resolve (in the exactly same situation) this “ARP timeout” problem. Did you try to find a solution with this config, ie without connecting the router and the tftp server together ? I ‘m gonna try to resolve it and go back with the solution.

    See you soon – Aymeric

    Comment by Aymeric — March 14, 2008 @ 3:47 am | Reply

  4. What a life saver these posts were 🙂
    I ran into the same memory errors when I upgraded to the IOS Advanced Security Pack.
    Thanks for the step-by-step to get my router working again.

    Comment by HillBilly — May 13, 2008 @ 5:49 am | Reply

  5. THANK YOU!!!! Lol I’m an intern and I accidentally erased the flash off the router and turned it off BEFORE I put a new IOS on the machine. Yeah you can picture my face after that, but thanks to you, the router works fine and best of all, my job is a-okay. Plus they think I’m a frickin genius lol. Thanks again.

    Comment by Intern — July 23, 2008 @ 9:21 am | Reply

  6. hi
    i am copy running-startup to nvram so
    my router start rommon and error load flash.nvram
    pls help me
    tanks

    Comment by javad — December 13, 2008 @ 11:34 am | Reply

  7. hi.,

    Really thanks for this blog and details…..such an helpful stuffs…..

    thanks great…

    Comment by clement — May 26, 2009 @ 7:18 am | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: