CCIE Pursuit Blog

July 19, 2008

Weekend Fun: CCIE Keeps City Locked Out FiberWAN Devices

Filed under: Cisco,OT: Humor — cciepursuit @ 1:22 pm
Tags: , ,

There’s a very interesting article about the ongoing ruckus over the San Francisco Network Engineer who – although under arrest and facing the possibility of years in jail – has told his (former) employers to get bent.  He set himself up with sole access to the city’s FiberWAN network and is not giving up his login info.  Give it a read during your study downtime.  I’ve included some of the more interesting bits below:

It seems that Terry Childs is a very intelligent man. According to my source, Childs holds a Cisco Certified Internetwork Expert certification, the highest level of certification offered by Cisco.

The routing configuration of the FiberWAN is extremely complex. Probably more so than it ought to be; I sometimes got the feeling that, in order to maintain more centralized control over the routing structure, [Childs] bent some of the rules of MPLS networks and caused problems for himself in terms of maintaining the routing.

Because the system was so complex (and also because he didn’t involve any of the other network engineers in his unit), Terry was the only person who fully understood the FiberWAN configuration. Therefore, to prevent inadvertent disruption of this admittedly critical network, he locked everyone else out. I know most of the networking equipment … does use centralized AAA, but I get the impression he may have configured the FiberWAN equipment for local authentication only.

This is where it gets tricky for the prosecution, IMO, because the localized authentication, with Terry as sole administrator, has been in place for months, if not years. His coworkers knew it (my coworkers and I were told many times by Terry’s coworkers, “If your request has anything to do with the FiberWAN, it’ll have to wait for Terry. He’s the only one with access to those routers”). His managers knew it.

Terry also, obviously, had a terrible relationship with his superiors. I should point out that he’s not just a network engineer — he was the lead network engineer for the entire City. His bosses were all managerial rather than technical, and while the other engineers did not actually report to Terry, they did defer to him in any technical matters. Even the network architect left it to Terry to actually figure out implementation. Terry felt that his direct superior was intrusive, incompetent, and obstructive, and that the managers above him had no real idea of what was going on, and were more interested in office politics than in getting anything done.

Later in the e-mail, my source offered some insight into what may be at the core of the issue: Childs was so paranoid about the security of the network that he even refused to write router and switch configs to flash, which would mean that if the device was powered off, all configurations would be lost.

At one point he was concerned about the security of the FiberWAN routers in remote offices, so he had them set up without saving the config to flash. “If they go down, I’ll get alerted, and connect up to them and reload the config.” Great, except we have power outages all the time in this city, some of those devices aren’t on UPSs, and what happens if you’re on vacation? And what about the 15 to 60 minutes it might take you to connect up and reload? He eventually conceded and (ahem) decided that disabling password recovery was sufficient security.

—Read The Rest Here—



  1. […] 23, 2008 If you haven’t been following the story of the San Francisco network engineer, here’s the backstory: a CCIE working for the city of San Francisco set up the FiberWAN […]

    Pingback by WTF Ending(?) To San Francisco Network Engineer Scandal « CCIE Pursuit Blog — July 23, 2008 @ 4:36 pm | Reply

  2. […] gone wild… I originally read about this, on CCIE Pursuit back in July.  Here’s an interesting summary/update of the CCIE gone […]

    Pingback by CCIEs gone wild… « ::cisco black belt:: — September 13, 2008 @ 3:20 pm | Reply

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Blog at

%d bloggers like this: