CCIE Pursuit Blog

March 28, 2008

ATTACK BY PROCTOR!!!

Every few months on GroupStudy there will be a story about a CCIE candidate going to lunch and coming back to find out that the proctor has messed with their configuration.  I always figured that this was just an urban legend, but then I experienced an ATTACK BY PROCTOR firsthand.

On Wednesday I took one of Internetwork Expert’s Graded Mock Labs.  I had successfully completed IGP redistribution  and had verified that I had end-to-end connectivity via TCL scripts.  I hurried up and applied the basic BGP peerings.  Then I consoled into each device and wrote the configuration.  Just before I went to lunch (actually just a 20 minute break to pick up my son) I reloaded all of the devices. 

When I got back I ran my TCL script again and noticed that my connectivity was broken.  Between IGP redistribution and basic BGP peering my network somehow “broke”.  I figured that I must have really messed something up with my BGP configuration since everything was fine after IGP redistribution.  I went back and verified my BGP configuration and peering on each device.  Maybe I lost some configuration when I reloaded – even though I was careful to make sure that I wrote each device before reloading.  All of my BGP configuration was there so I obviously wrote each device as that was the last configuration I had completed.  All of the BGP peerings were up and I was seeing routes coming in on the peerings (the ones that should be sending routes at least).  WTF?

I didn’t want to waste a ton of time troubleshooting this mess, so I soldiered on with the exam and made a note to run the TCL scripts again later.  When I did eventually run the scripts again, my network was still broken.

I finally started to look at the individual routes that were missing.  My issue seemed to be between the hub and spokes on one of the Frame Relay networks.  I still could not find an issue with BGP.  I took a look at my OSPF neighbors and I found that my OSPF adjacencies from my hub to my spokes were gone.  I still had some routes coming in via a virtual link between my hub and one of my spokes.  How the hell did that happen?

I looked at my OSPF configuration on my hub and found that my neighbor statements to the spokes were missing.  I added them back into the configuration and then ran my TCL scripts again.  Hallelujah!!!  I had full connectivity once again.  Although I had written my configurations after each task I must have somehow not written r5’s configuration after I finished my first OSPF task?  I continued on with the lab as I only had about 20 minutes left at that point.  I did not reload the routers again before finishing the test.  I did make sure that I wrote my configurations on each device and saved out my configurations to textfiles on my desktop.  I probably verified that those neighbor statements were still on r5 about 10 times before the lab ended.  🙂

Thinking back about this issue later I realized that since my BGP configuration was still on r5 I had definitely written the configuration on that device.  The neighbor statements must have been removed when I reloaded that router.  Somehow those devious bastards at IE must have removed my configuration.  They probably have a script that somehow removes those two lines of configuration whenever r5 is reloaded.  I had experienced the dread ATTACK BY PROCTOR!!!

Or not…. 🙂

I went to ccie-in-3-months to see if Tassos had experienced any similar issues with his mock lab and found this:

…I decided to reload all routers! What was even more disastrous, was the fact that i reloaded them all at the same time and i didn’t look at their logs while booting.

The result? After the reload, something wasn’t working as expected. After a quick search I found one router which seemed not to be running OSPF. I checked its configuration (thank god I had saved all my session locally) and I found that there were two “neighbor” commands missing! I added them and reloaded again. This time I watched the logs and there was an error message saying that this particular command is not supported on this kind of topologies (a bug? command is accepted while configuring, but it’s rejected after reloading). So i saved my configuration and warned (through email) the proctor about this behavior.

Here’s his post on the Internetwork Expert Mock Lab 1 forum:

Task 4.1

After reload:

Cisco 3640 (R4700) processor (revision 0x00) with 111616K/19456K bytes of memory.
Processor board ID 13831044
R4700 CPU at 100MHz, Implementation 33, Rev 1.0
2 Ethernet interfaces
2 Serial interfaces
DRAM configuration is 64 bits wide with parity disabled.
125K bytes of NVRAM.
32768K bytes of processor board System flash (Read/Write)

OSPF: Neighbor command is allowed only on NBMA and point-to-multipoint networks
OSPF: Neighbor command is allowed only on NBMA and point-to-multipoint networks

Press RETURN to get started!

Like I said, I didn’t reload the router a second time.  I also would not have thought to watch the reload output.  So the good news is that it was not ATTACK BY PROCTOR but I was the victim of a bug.  [My network type on the hub and spoke was NBMA per the task].

This was actually a good experience as I got to do some troubleshooting under pressure.  I did fuck up my troubleshooting as I assumed that the issue was with BGP and did not troubleshoot outside of BGP until much later.  This was also a potentially disastrous experience as I would have lost an untold number of points due to not having end-to-end connectivity.  I am a bit disappointed that IE had not addressed this bug.

Oh, and the ATTACK BY PROCTOR rumor?  It’s exactly that – a rumor:

Hi Maruilio,

Thanks for the reply!!!

Could you please give a clarity on proctor’s behaviour during the Lab exam? I’ve heard that proctor’s occasionally changes the configuration while the exam is on eg- erase passwords, erase configurations, shutting down the interface, changing the IP addreses, etc. I am not sure if that’s all true? If it is, is that justified? I’ve also heard that you should not be too fast even if you know are pretty confident of your configurations because proctor’s probability of changing the configurations increases. Is that also true?

If the above is true, I also want to know to what extent are proctor’s authorised to play with the configurations during the exam?

My second query is – when the lab starts do we get all the devices with zero configurations or we have to first erase all the pre-configured inappropriate configuration on the devices?

My third query is out of curiosity – what method does proctor’s use for checking the lab created by the student?
Regards,
Saurabh Garg

Hi Saurabh,

These are all rumors and do not reflect the CCIE Lab environment.

Proctors do not touch any of the candidate’s devices during the exam.The only exception will be if a candidate thinks that something is not working because a possible failure on your rack the Proctor will ver[if]y it, but the candidate will be aware of it. Proctors do not touch or play with candidate’s configuration during or after the exam.

When you start the exam your routers and switches will have an initial configuration such IP addresses, hostnames, passwords. Depending of the exam you may have more pre-configuration. The ‘General Guidelines’ of the exam will state what you can change and what not can be changed.

We do have a process to development each question of the exam and it is based on results. By the end of the exam Proctors use an automatic tool to save the candidate’s configuration into our database and to verify some questions and do some connectivity tests like pings, verify routing tables, and so on. Then Proctors will manually verify the results and all remaining questions to come up with the final score.

Regards,
Maurilio

Advertisements

5 Comments »

  1. […] with about 30 minutes left.  I ended up not attempting 3 tasks. I spent quite a bit of time troubleshooting an issue that was probably due to an IOS bug, but I had plenty of time to complete the lab.  I did start getting mentally fatigued about 6 […]

    Pingback by Internetwork Expert: Mock Lab I Review « CCIE Pursuit — March 31, 2008 @ 5:00 pm | Reply

  2. OSPF: Neighbor command is allowed only on NBMA and point-to-multipoint networks
    OSPF: Neighbor command is allowed only on NBMA and point-to-multipoint networks

    I actually received the above while doing a config using IPexpert’s workbook and Dynamips. Didn’t have to reboot to get it, might be the version of code has changed things.
    ADVIPSERVICESK9-M, Version 12.4(18).

    Did you enjoy snow?

    Comment by Rick — April 1, 2008 @ 10:58 am | Reply

  3. @Rick – Thanks for the tip. If I have time I’ll run that code on Dynamips and see if I get the same error. I didn’t verify the code version while doing the mock lab. In the mock lab, the router took the commands and they stayed in the running and startup configuration (I did a ‘show start | sec router ospf’ after writing the running config) but seemed to be stripped only after a reload (I only reloaded once, so I’m going off of Tassos’ findings). Still, it’s good to know that there are possible bugs out there that could affect your configuration.

    And NO, I did not enjoy the snow. 🙂

    Comment by cciepursuit — April 2, 2008 @ 9:53 am | Reply

  4. You’re right…that is a bug. In fact, after you enter ip ospf priority 0 on an interface then do a show run, the neighbor commands are not there, even though it is working. I think it is b/c this isn’t the way it was intended to work “in the real world”. But since the test is not a “real world” scenario, I’m sure there are several things like this!

    Comment by Jody — October 15, 2008 @ 8:04 pm | Reply

  5. This is a bug, proctors do not mess with your configs. Did you had a virtual linke? here is the bug ID as I experienced it too in a IEWBII.
    CSCse72792

    Comment by Alex — March 26, 2009 @ 6:34 am | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: