CCIE Pursuit Blog

March 28, 2008

ATTACK BY PROCTOR!!!

Every few months on GroupStudy there will be a story about a CCIE candidate going to lunch and coming back to find out that the proctor has messed with their configuration.  I always figured that this was just an urban legend, but then I experienced an ATTACK BY PROCTOR firsthand.

On Wednesday I took one of Internetwork Expert’s Graded Mock Labs.  I had successfully completed IGP redistribution  and had verified that I had end-to-end connectivity via TCL scripts.  I hurried up and applied the basic BGP peerings.  Then I consoled into each device and wrote the configuration.  Just before I went to lunch (actually just a 20 minute break to pick up my son) I reloaded all of the devices. 

When I got back I ran my TCL script again and noticed that my connectivity was broken.  Between IGP redistribution and basic BGP peering my network somehow “broke”.  I figured that I must have really messed something up with my BGP configuration since everything was fine after IGP redistribution.  I went back and verified my BGP configuration and peering on each device.  Maybe I lost some configuration when I reloaded – even though I was careful to make sure that I wrote each device before reloading.  All of my BGP configuration was there so I obviously wrote each device as that was the last configuration I had completed.  All of the BGP peerings were up and I was seeing routes coming in on the peerings (the ones that should be sending routes at least).  WTF?

I didn’t want to waste a ton of time troubleshooting this mess, so I soldiered on with the exam and made a note to run the TCL scripts again later.  When I did eventually run the scripts again, my network was still broken.

I finally started to look at the individual routes that were missing.  My issue seemed to be between the hub and spokes on one of the Frame Relay networks.  I still could not find an issue with BGP.  I took a look at my OSPF neighbors and I found that my OSPF adjacencies from my hub to my spokes were gone.  I still had some routes coming in via a virtual link between my hub and one of my spokes.  How the hell did that happen?

I looked at my OSPF configuration on my hub and found that my neighbor statements to the spokes were missing.  I added them back into the configuration and then ran my TCL scripts again.  Hallelujah!!!  I had full connectivity once again.  Although I had written my configurations after each task I must have somehow not written r5’s configuration after I finished my first OSPF task?  I continued on with the lab as I only had about 20 minutes left at that point.  I did not reload the routers again before finishing the test.  I did make sure that I wrote my configurations on each device and saved out my configurations to textfiles on my desktop.  I probably verified that those neighbor statements were still on r5 about 10 times before the lab ended.  🙂

Thinking back about this issue later I realized that since my BGP configuration was still on r5 I had definitely written the configuration on that device.  The neighbor statements must have been removed when I reloaded that router.  Somehow those devious bastards at IE must have removed my configuration.  They probably have a script that somehow removes those two lines of configuration whenever r5 is reloaded.  I had experienced the dread ATTACK BY PROCTOR!!!

Or not…. 🙂

I went to ccie-in-3-months to see if Tassos had experienced any similar issues with his mock lab and found this:

…I decided to reload all routers! What was even more disastrous, was the fact that i reloaded them all at the same time and i didn’t look at their logs while booting.

The result? After the reload, something wasn’t working as expected. After a quick search I found one router which seemed not to be running OSPF. I checked its configuration (thank god I had saved all my session locally) and I found that there were two “neighbor” commands missing! I added them and reloaded again. This time I watched the logs and there was an error message saying that this particular command is not supported on this kind of topologies (a bug? command is accepted while configuring, but it’s rejected after reloading). So i saved my configuration and warned (through email) the proctor about this behavior.

Here’s his post on the Internetwork Expert Mock Lab 1 forum:

Task 4.1

After reload:

Cisco 3640 (R4700) processor (revision 0x00) with 111616K/19456K bytes of memory.
Processor board ID 13831044
R4700 CPU at 100MHz, Implementation 33, Rev 1.0
2 Ethernet interfaces
2 Serial interfaces
DRAM configuration is 64 bits wide with parity disabled.
125K bytes of NVRAM.
32768K bytes of processor board System flash (Read/Write)

OSPF: Neighbor command is allowed only on NBMA and point-to-multipoint networks
OSPF: Neighbor command is allowed only on NBMA and point-to-multipoint networks

Press RETURN to get started!

Like I said, I didn’t reload the router a second time.  I also would not have thought to watch the reload output.  So the good news is that it was not ATTACK BY PROCTOR but I was the victim of a bug.  [My network type on the hub and spoke was NBMA per the task].

This was actually a good experience as I got to do some troubleshooting under pressure.  I did fuck up my troubleshooting as I assumed that the issue was with BGP and did not troubleshoot outside of BGP until much later.  This was also a potentially disastrous experience as I would have lost an untold number of points due to not having end-to-end connectivity.  I am a bit disappointed that IE had not addressed this bug.

Oh, and the ATTACK BY PROCTOR rumor?  It’s exactly that – a rumor:

Hi Maruilio,

Thanks for the reply!!!

Could you please give a clarity on proctor’s behaviour during the Lab exam? I’ve heard that proctor’s occasionally changes the configuration while the exam is on eg- erase passwords, erase configurations, shutting down the interface, changing the IP addreses, etc. I am not sure if that’s all true? If it is, is that justified? I’ve also heard that you should not be too fast even if you know are pretty confident of your configurations because proctor’s probability of changing the configurations increases. Is that also true?

If the above is true, I also want to know to what extent are proctor’s authorised to play with the configurations during the exam?

My second query is – when the lab starts do we get all the devices with zero configurations or we have to first erase all the pre-configured inappropriate configuration on the devices?

My third query is out of curiosity – what method does proctor’s use for checking the lab created by the student?
Regards,
Saurabh Garg

Hi Saurabh,

These are all rumors and do not reflect the CCIE Lab environment.

Proctors do not touch any of the candidate’s devices during the exam.The only exception will be if a candidate thinks that something is not working because a possible failure on your rack the Proctor will ver[if]y it, but the candidate will be aware of it. Proctors do not touch or play with candidate’s configuration during or after the exam.

When you start the exam your routers and switches will have an initial configuration such IP addresses, hostnames, passwords. Depending of the exam you may have more pre-configuration. The ‘General Guidelines’ of the exam will state what you can change and what not can be changed.

We do have a process to development each question of the exam and it is based on results. By the end of the exam Proctors use an automatic tool to save the candidate’s configuration into our database and to verify some questions and do some connectivity tests like pings, verify routing tables, and so on. Then Proctors will manually verify the results and all remaining questions to come up with the final score.

Regards,
Maurilio

Advertisements

Create a free website or blog at WordPress.com.