I am going to blame my brainlessness today on exercise. I did squats yesterday (my least favorite exercise) and followed that up with a run on a deceptively hot day. The end result of that genius workout regime is that my legs are burning today. This has also afforded my co-workers the opportunity to punch me in the legs knowing that it will not only hurt like hell, but that I can’t hobble fast enough to catch them.
Anyhoo…we have a small overseas site that was having a ton of bounces today. The WAN circuit was taking errors and eventually dropping the BGP peering. At that point the backup WAN router would bring up the DSL/Cable/ISDN (I don’t remember what they are running) and traffic would reroute. Then the peering on the primary would reestablish and traffic would route that way. This kept repeating.
We don’t have a lot of these types of sites. Generally we have connections to two different carriers and load balance. This one was a problem because we don’t have a dial-in modem for the primary (another deviation from norm) so we only have connectivity via the WAN. We needed to get into the primary and shut down the BGP peering so that the carrier could test the circuit and traffic would flow over the secondary.
The problem was that the bouncing peering meant that if you connected via the primary you ran the risk of getting kicked out of your session in the middle of your configuration. If you waited until the secondary came up you could get in, but your connection might drop or your TACACS+ authentication would fail if the primary came back up.
I typed up all of the commands in notepad and jumped into the router and quickly pasted the commands to shutdown the BGP peering. Success! I walked around the corner to discuss the game plan for this circuit and to show one of the new guys what I did and why. It was during that explanation that I realized that I had included “reload in 5” in my configuration (in case I got locked out in the middle of my configuration) but I had not issued a “reload cancel” afterwards. I hobbled as quickly as I could back to my desk in time to see the reload start. ARGGGGGGHHHH!!!!!!!
The router reloaded and I got to repeat the process plus explain why the router rebooted. 🙂