CCIE Pursuit Blog

August 7, 2008

“Reload in x” is your friend…except when it isn’t

Filed under: Cisco,IOS,Personal — cciepursuit @ 11:18 am
Tags: ,

I am going to blame my brainlessness today on exercise.  I did squats yesterday (my least favorite exercise) and followed that up with a run on a deceptively hot day.  The end result of that genius workout regime is that my legs are burning today.  This has also afforded my co-workers the opportunity to punch me in the legs knowing that it will not only hurt like hell, but that I can’t hobble fast enough to catch them.

Anyhoo…we have a small overseas site that was having a ton of bounces today.  The WAN circuit was taking errors and eventually dropping the BGP peering.  At that point the backup WAN router would bring up the DSL/Cable/ISDN (I don’t remember what they are running) and traffic would reroute.  Then the peering on the primary would reestablish and traffic would route that way.  This kept repeating.

We don’t have a lot of these types of sites.  Generally we have connections to two different carriers and load balance.  This one was a problem because we don’t have a dial-in modem for the primary (another deviation from norm) so we only have connectivity via the WAN.  We needed to get into the primary and shut down the BGP peering so that the carrier could test the circuit and traffic would flow over the secondary.

The problem was that the bouncing peering meant that if you connected via the primary you ran the risk of getting kicked out of your session in the middle of your configuration.  If you waited until the secondary came up you could get in, but your connection might drop or your TACACS+ authentication would fail if the primary came back up.

I typed up all of the commands in notepad and jumped into the router and quickly pasted the commands to shutdown the BGP peering.  Success!  I walked around the corner to discuss the game plan for this circuit and to show one of the new guys what I did and why.  It was during that explanation that I realized that I had included “reload in 5” in my configuration (in case I got locked out in the middle of my configuration) but I had not issued a “reload cancel” afterwards.  I hobbled as quickly as I could back to my desk in time to see the reload start.  ARGGGGGGHHHH!!!!!!!

The router reloaded and I got to repeat the process plus explain why the router rebooted.  🙂

Advertisements

6 Comments »

  1. Hi
    I must have missed something in the article but was it not possible to issue a neighbor shutdown from the PE rather than jump onto the CE?

    Comment by Roy Waterman — August 7, 2008 @ 1:10 pm | Reply

  2. @Roy Waterman – The PE is owned by the carrier, we don’t have access to it (I wish we did). I could have asked AT&T to shut the peer, but it would have been days before they got the right team to do that and weeks for them to back it out. 🙂

    Shutting the BGP neighbor on the CER is the best way to go in this case as the carrier (and eventually the LEC) can test to their hearts’ content as the circuit stays up but none of our traffic is routed over that circuit.

    Comment by cciepursuit — August 7, 2008 @ 2:07 pm | Reply

  3. I’ll just ask this here, although this blog-marking is not directly connected to CCIE exam.

    I believe I can ask this from proctor during the real lab, but I hope someone knows the answer.

    Are we allowed to make directories and create files on the actual lab devices? I’m currently running the IE labs and found out that it is really nice to save your running config to flash with name like R2_lab4_step_5_3.cfg to flash. Of course I also do wr mem, but with separate “backup” on every step this has been quite handy. So, is this legitimate?

    Also I have found out that I do suck on IGP redisribution…

    Comment by Goingtobrussels — August 8, 2008 @ 4:23 am | Reply

  4. HAHA Lmao I think we all have had a similar exp. No big deal it happens. I still would have pucnched you in your leg!!!! JK You getting ready for your lab?

    Tony!!!

    Comment by Tony — August 8, 2008 @ 8:08 am | Reply

  5. @Goingtobrussels – It is my understanding that you are not allowed to save anything to the devices in the lab.

    Don’t worry, EVERYBODY sucks at IGP redistribution. 🙂

    Comment by cciepursuit — August 8, 2008 @ 4:07 pm | Reply

  6. @Tony – I know YOU would have punched me in the legs. 🙂

    I’m still inching towards my lab. I pretty much need to pass on the first go because my wife keeps adding more costs to our San Jose “vacation”. 🙂

    Comment by cciepursuit — August 8, 2008 @ 4:08 pm | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: