In Jeff Doyle’s blog (yes, THE Jeff Doyle of Routing TCP/IP fame) he posts about “The Human Touch” in which he relates a couple of situations where he inadvertantly screwed up a router or two:
Back in the early-1990’s I was in Albuquerque, NM, making some configuration changes on a router in an unmanned office in Santa Fe. Specifically, I was making changes to the single WAN interface, the interface my telnet session was crossing, and had IOS in interface config mode.
Yeah, you already know where this is going.
I began to type “show interface,” got as far as typing “sh,” realized I needed to get out of config mode before getting the display I wanted, and hit the Enter key.
There was that amazing, brilliant flash of realization, as my finger went down in the button, of what I had done. Identical to that instant when you are slamming the locked car door and realize your keys are in the ignition. IOS interpreted “sh” as “shutdown,” and like any computer did exactly what it was told. I had locked the door to the router with the keys in the ignition. Because it was an unmanned office, I had no recourse but to get in the car and make the one-hour drive to Santa Fe to turn the interface back up.
And maybe that wasn’t my dumbest mistake. There was the time I configured two OSPF routers with the same Router ID, but at least that experience gave me a nifty troubleshooting exercise for one of my books. And then there was the time…
Aside from taking solace in the fact that even the mightiest of the Cisco gods have made some of the same mistakes that I have, this got me thinking about some of my own past mistakes. I don’t have a single, defining screw up, but I have fallen prey to a long list of smaller, more common ones. I’ve done all of the following (some of them more than once):
- Changed the IP network of the interface that I was telnetted on.
- Forgotten to override the implicit deny in an access-list and locked out a router.
- Shipped routers to a site with the interfaces still in shutdown (of course this was a small site in the middle of nowhere and I did not ship a Cisco console cable).
- Wasted an entire day troubleshooting an issue only to find that I had installed the wrong SFP in the switch.
- Shutdown the only WAN interface on a router.
- Pasted the wrong configuration into a device and then had the joy of trying to remotely fix two devices with the same IP address.
- Authorized a carrier to test the wrong circuit and then watched in horror as most of our Canadian sites went down.
- Made TACACS+ the only authentication method without establishing a connection to the TACACS+ server.
I screw up on a daily basis. Usually, my problems are of the “fat fingering” type that are usually pretty easy to spot and fix. My favorite story happened to a co-worker of mine. He was removing some networks from one of our core routers. He had cut and pasted the EIGRP configuration into notepad and then typed “no” in front of the networks to be removed. Somehow he also typed “no” in front of the EIGRP statement as well: “no router eigrp 100″. In an instant the entire EIGRP process was gone.
We all make mistakes regardless of our years of experience and certifications. Hopefully we learn from our mistakes and we avoid really stupid “Career Limiting Decisions” along the way.
Here are some other (non IT) tales of workplace disasters.
