CCIE Pursuit Blog

March 28, 2008


Every few months on GroupStudy there will be a story about a CCIE candidate going to lunch and coming back to find out that the proctor has messed with their configuration.  I always figured that this was just an urban legend, but then I experienced an ATTACK BY PROCTOR firsthand.

On Wednesday I took one of Internetwork Expert’s Graded Mock Labs.  I had successfully completed IGP redistribution  and had verified that I had end-to-end connectivity via TCL scripts.  I hurried up and applied the basic BGP peerings.  Then I consoled into each device and wrote the configuration.  Just before I went to lunch (actually just a 20 minute break to pick up my son) I reloaded all of the devices. 

When I got back I ran my TCL script again and noticed that my connectivity was broken.  Between IGP redistribution and basic BGP peering my network somehow “broke”.  I figured that I must have really messed something up with my BGP configuration since everything was fine after IGP redistribution.  I went back and verified my BGP configuration and peering on each device.  Maybe I lost some configuration when I reloaded – even though I was careful to make sure that I wrote each device before reloading.  All of my BGP configuration was there so I obviously wrote each device as that was the last configuration I had completed.  All of the BGP peerings were up and I was seeing routes coming in on the peerings (the ones that should be sending routes at least).  WTF?

I didn’t want to waste a ton of time troubleshooting this mess, so I soldiered on with the exam and made a note to run the TCL scripts again later.  When I did eventually run the scripts again, my network was still broken.

I finally started to look at the individual routes that were missing.  My issue seemed to be between the hub and spokes on one of the Frame Relay networks.  I still could not find an issue with BGP.  I took a look at my OSPF neighbors and I found that my OSPF adjacencies from my hub to my spokes were gone.  I still had some routes coming in via a virtual link between my hub and one of my spokes.  How the hell did that happen?

I looked at my OSPF configuration on my hub and found that my neighbor statements to the spokes were missing.  I added them back into the configuration and then ran my TCL scripts again.  Hallelujah!!!  I had full connectivity once again.  Although I had written my configurations after each task I must have somehow not written r5’s configuration after I finished my first OSPF task?  I continued on with the lab as I only had about 20 minutes left at that point.  I did not reload the routers again before finishing the test.  I did make sure that I wrote my configurations on each device and saved out my configurations to textfiles on my desktop.  I probably verified that those neighbor statements were still on r5 about 10 times before the lab ended.  🙂

Thinking back about this issue later I realized that since my BGP configuration was still on r5 I had definitely written the configuration on that device.  The neighbor statements must have been removed when I reloaded that router.  Somehow those devious bastards at IE must have removed my configuration.  They probably have a script that somehow removes those two lines of configuration whenever r5 is reloaded.  I had experienced the dread ATTACK BY PROCTOR!!!

Or not…. 🙂

I went to ccie-in-3-months to see if Tassos had experienced any similar issues with his mock lab and found this:

…I decided to reload all routers! What was even more disastrous, was the fact that i reloaded them all at the same time and i didn’t look at their logs while booting.

The result? After the reload, something wasn’t working as expected. After a quick search I found one router which seemed not to be running OSPF. I checked its configuration (thank god I had saved all my session locally) and I found that there were two “neighbor” commands missing! I added them and reloaded again. This time I watched the logs and there was an error message saying that this particular command is not supported on this kind of topologies (a bug? command is accepted while configuring, but it’s rejected after reloading). So i saved my configuration and warned (through email) the proctor about this behavior.

Here’s his post on the Internetwork Expert Mock Lab 1 forum:

Task 4.1

After reload:

Cisco 3640 (R4700) processor (revision 0x00) with 111616K/19456K bytes of memory.
Processor board ID 13831044
R4700 CPU at 100MHz, Implementation 33, Rev 1.0
2 Ethernet interfaces
2 Serial interfaces
DRAM configuration is 64 bits wide with parity disabled.
125K bytes of NVRAM.
32768K bytes of processor board System flash (Read/Write)

OSPF: Neighbor command is allowed only on NBMA and point-to-multipoint networks
OSPF: Neighbor command is allowed only on NBMA and point-to-multipoint networks

Press RETURN to get started!

Like I said, I didn’t reload the router a second time.  I also would not have thought to watch the reload output.  So the good news is that it was not ATTACK BY PROCTOR but I was the victim of a bug.  [My network type on the hub and spoke was NBMA per the task].

This was actually a good experience as I got to do some troubleshooting under pressure.  I did fuck up my troubleshooting as I assumed that the issue was with BGP and did not troubleshoot outside of BGP until much later.  This was also a potentially disastrous experience as I would have lost an untold number of points due to not having end-to-end connectivity.  I am a bit disappointed that IE had not addressed this bug.

Oh, and the ATTACK BY PROCTOR rumor?  It’s exactly that – a rumor:

Hi Maruilio,

Thanks for the reply!!!

Could you please give a clarity on proctor’s behaviour during the Lab exam? I’ve heard that proctor’s occasionally changes the configuration while the exam is on eg- erase passwords, erase configurations, shutting down the interface, changing the IP addreses, etc. I am not sure if that’s all true? If it is, is that justified? I’ve also heard that you should not be too fast even if you know are pretty confident of your configurations because proctor’s probability of changing the configurations increases. Is that also true?

If the above is true, I also want to know to what extent are proctor’s authorised to play with the configurations during the exam?

My second query is – when the lab starts do we get all the devices with zero configurations or we have to first erase all the pre-configured inappropriate configuration on the devices?

My third query is out of curiosity – what method does proctor’s use for checking the lab created by the student?
Saurabh Garg

Hi Saurabh,

These are all rumors and do not reflect the CCIE Lab environment.

Proctors do not touch any of the candidate’s devices during the exam.The only exception will be if a candidate thinks that something is not working because a possible failure on your rack the Proctor will ver[if]y it, but the candidate will be aware of it. Proctors do not touch or play with candidate’s configuration during or after the exam.

When you start the exam your routers and switches will have an initial configuration such IP addresses, hostnames, passwords. Depending of the exam you may have more pre-configuration. The ‘General Guidelines’ of the exam will state what you can change and what not can be changed.

We do have a process to development each question of the exam and it is based on results. By the end of the exam Proctors use an automatic tool to save the candidate’s configuration into our database and to verify some questions and do some connectivity tests like pings, verify routing tables, and so on. Then Proctors will manually verify the results and all remaining questions to come up with the final score.


March 26, 2008

Internetwork Expert Graded Mock Lab 1: First Impressions

I just finished IE’s Graded Mock Lab 1.  This culminates a week and a half of hardcore labbing.  Now I get to go back to work and spend the next week and a half digging out of the work that’s been backing up over my “vacation”.

Mock Lab 1 is a difficultly level 6 lab.  That means that it’s easier than the actual lab.  After reading through the lab I felt that I had a very good chance of passing it.  There were only a couple of tasks that I was not sure how to do.  I cruised through the core tasks up until route-redistribution.  Even though the redistribution was insanely easy, I still spent a lot of time running scripts and verifying connections.  By the time I took my “lunch break” I had just completed BGP peering.

I left 9 points on the table (3 points each in BGP, Security, and IP Services).  Of those three tasks, there was only one (in IP Services) that was something completely alien to me. 

I really felt that I passed the lab.  I estimate that I scored between 75 and a 91.  There were about 4 or 5 questions that I would have asked of the proctor.

As soon as the lab is over you get an initial scoring report.  The initial score report is graded by a script.  You lab will also be graded by a proctor in the next two business days.  After that you’ll receive your final score and feedback from the proctor.

My initial score report is 70.

You Scored: 70
Mock Lab:R&S Mock Lab 1
Difficulty Level:6
You scored better than 69% of all students who took this lab.

Note: This mock lab was graded by our automated grading system.  A proctor will also grade your configuration and it will appear in your members site within 2 business days.

Hopefully I can get another 10 points from the proctor.  As it stands, these results mirror my experience with most of the Volume II labs.  I do really well on the all of the core IGP and switching tasks.  I then fall down a little on BGP.  Then I nosedive on the last sections.  In this case, I got 56 of the first 64 points.  That means that I only got 14 of the last 36 points.  If I score over 80 (my goal) then I will be happy.  If my score stays at 70, I will be satisfied but still a little disappointed.

I’m too tired and punch drunk to go over the solution guide right now.  I’ll post more about this lab later.

March 24, 2008

Status Update: 17 – 23 March

I’ll make this update pretty quick as I have already pretty much covered last week’s progress in a previous post.

I will be redoing IE Volume II lab 3 on Tuesday and on Wednesday I will be doing IE Mock Lab 1.  I spent some time today going over basic Multicast and a little bit of FRTS.

Kind of off topic: I’m going to start posting some “cool commands” and “question of the day” posts.  I had always planned to share some of my favorite commands but haven’t gotten around to it until now.  I’ll also be posting some review questions (and possibly quick lab scenarios) soon.  I’m probably going to be spending a lot more time labbing and less time posting over the next couple of weeks so hopefully these types of shorter posts will be beneficial as well as interesting.

Here are my goals for the remainder of this week: Redo Volume II labs 3, 8. and 9.  Review the IEATC videos for Multicast and IP Services.

Days Until Lab: 119
Days Until Mock Lab 1: 2
Days Until Mock Lab Workshop: 84
Readiness (1 to 10): 2
Lab Hours This Week 50
Study Hours This Week (estimate): 4

Worst. Vacation. Ever.

The room is filled with papers scrawled with multicolored marks and a waste basket overflowing with crushed Diet Mt. Dew cans.  A closer inspection shows strange drawings filled with circles connected by lines interspersed with numbers and strange combinations of letters.  The desk looks like someone or something has repeatedly banged its head on it in frustration.  There is a wet spot on the desk that looks like human tears.

I’m cleaning up the guest bedroom today.  Last week I made this room my “operations center” while working through six labs in seven days.   I took vacation from work to devote time to doing these labs and labeled the experience, “Labapolooza”.  This name used to recall memories of large ice cream desserts and alternative music festivals.  Now it reminds me of endless hours working on the CLI and cursing the Brians.  🙂

Of course I bit off more than I could chew.  For some reason I believed that I would be able to complete the labs quicker than I actually did.  Although I did manage to complete most of the labs (I skipped Multicast on a couple of them), it took at least 11 hours per lab and sometimes more.  I spent Easter on the CLI shoveling Peeps into my mouth (they’re not the brain food you’d think they would be  🙂 ) while labbing away.  Most of my breaks from labbing were to do mundane tasks like get rid of the 10 inches of wet March snow that fell last week.

I was at the point where I was doing pretty well on difficultly level 6 and 7 labs so I didn’t think that the step up to level 8 labs would be too drastic.  I was wrong.  While there were only a few different technologies in the mix, the tasks were more difficult to understand and seemed to take more steps to implement.  It took me about 4 hours to get through the first two (short) sections of lab 16 before I finally just gave up on that lab.  That was my low point for the week.

All in all, the experience was a pain in the ass but ultimately worth it.  I discovered that I’m probably even less prepared for the lab than I thought that I was.  The ultimate purpose of this was to make strides in completing the 20 IE Volume II labs (I want to have completed them at least one time 60 days before my lab date) and to ramp up for my first IE mock lab this Wednesday.  Like most things CCIE, I feel like I’ve taken two steps forward and one and a half steps backward.

I still need to review the basics on a few technologies.  I’m pretty good at switching, IGP (with the exception of route redistribution), QoS, and WAN technologies.  I’m okay at BGP (but far from the level I need to be at).  I’m not very good at IPv6 and IP Services.  I’m horrible at Multicast and security.

I still have three more days off.  Today I will review some of my week spots.  I had planned to do a simulated mock lab tomorrow, but I’m going to redo lab 6 instead.  I’ll probably be redoing all of the level 8 labs again in the next few weeks.  I will post about them then.

There is a special place in hell for the team that is currently redesigning the Cisco documentation.  A few weeks ago I praised the new look of the site.  The 12.4 documentation is completely fucked right now as well as parts of the 12.3 documentation.  It’s been that way for a couple of weeks now.  I’m at a complete loss as to how a very large company like Cisco allows this to happen.  Most companies would do the site redesign on test servers, then verify that it works, and only then would they move it to production servers.  Cisco seems to think that hundreds of broken links and bad redirections on thier documentation website is okay.  Ethan recently posted an open letter concerning this.  My statement to Cisco is a little less elegant: “Fix your shit!  If you can’t do this, then for fuck’s sake mirror the old working site somewhere while you continue to tear apart your site.”

March 16, 2008


Stupid name.  I know.

I am on vacation this week which means rest and relaxationan aggressive schedule of Volume II labs.  I’m using this week to get a lot of labbing done in a run up to my first Internetwork Expert Graded Mock lab on Wednesday, 26 March.  Here’s my schedule for this week (and part of the next):

Monday, 17 March Lab 8
Tuesday, 18 March Lab 9
Wednesday, 19 March Break/Review
Thursday, 20 March Lab 10
Friday, 21 March Lab 16
Saturday, 22 March Lab 6 (Review)
Sunday, 22 March Lab 3 (Review)
Tuesday, 25 March Lab 18 (Simulated Lab)
Wednesday, 26 March IE Mock Lab 1

I will be completing 7 Volume III (2 will be labs I’ve done before) labs over 10 days.  That’s more Volume II labs than I’ve completed so far (6) in a very compressed time period.

I will probably only be blogging highlights from each of these labs as I will be strapped for time.  I’ll more than likely be doing these labs again, so I will blog them in more detail the next time around.

March 6, 2008

Status Update: 11 February – 02 March

It’s been a long time since I have penned a status update.  I’ve been absolutely overwhelmed at work.  We have an annual “freeze period” that coincides with our company’s busiest time of the year (I have no idea what we’re so busy doing, I just push ones and zeros).  This means that at the end of the calendar year we have a nice 4 – 6 week period in which only emergency changes can be made.  The bad part is that as soon as the freeze is done we get bombarded with work.  I’m just starting to dig out of that mess.

My lab time has dropped off over the last couple of weeks for two reasons: I am exhausted after long hours in the office, and I have decided to make another pass through the IEATC lessons.  I really need to solidify my understanding of the technologies on the lab.  So far I’ve gone over stuff that I’m really good at (Frame Relay and Switching).  Even though these are the technologies that I am strongest in, I still (re)discover bits and pieces in the IEATC videos.  I will try to get through most of OSPF this week.  Next week will be the technologies that I am weakest in (BGP, Multicast, and IPv6).

I’m getting “lab withdrawal” symptoms, so I will be interspersing some of the Volume III labs with the IEATC this weekend.  I hope to complete all of the IEATC lessons by the end of next week.  The week after that (17 – 23 March) I have dubbed “Labapalooza”.  I am taking the week off from work and will be attempting to complete as many of the Volume II and III labs as I can during that week.

My lab date has changed yet again.  I will be attending the Internetwork Expert Mock Lab Workshop in Reno from 16 – 20 June.  As I mentioned in this post, I will schedule my lab date 28 days past the end of the workshop.  My new target date is 21 July.  While I am bummed that this new dates means that I am going to lose a good chunk of my summer to my CCIE pursuit, it does mean that I get more time to study. 

Scheduling the Mock Lab Workshop has left my “CCIE Wish List” nearly empty.  The only thing that I will definitely be purchasing between now and my exam is the Cisco Assessor labs.  Other than that, I may try to pick up a lab book from another vendor (just to get the experience of a different topology/testing style).  I may also pick up a mock lab from another vendor for the same reasons.  Those two things will depend on my funds and perceived necessity at the time.

I’m disappointed that I dropped the ball on posting weekly status updates for the last couple of weeks.  I use these updates to motivate/guide myself.  If I take the time to write them up and post them, then it makes me that more inclined to do what I’ve laid out.

Anyhoo…here are my goals for the remainder of this week: Finish up (at least) the first week OSPF videos.  Redo the first 3 Volume III labs.

Days Until Lab: 137
Days Until Mock Lab 1: 20
Days Until Mock Lab Workshop: 102
Readiness (1 to 10): 2
Lab Hours This Week 0
Study Hours This Week (estimate): 15

February 11, 2008

Status Update: 04 – 10 February

Last week started out pretty well, but I had to write off my second weekend in a row.  This time it was because the temperature here was barely above absolute zero (at least it felt that way).  I spent one day mucking with our cars.  One day after bragging that my car ALWAYS starts, I spent a good chunk of time trying to get it to start.  I didn’t park in the garage and the next morning it was around -40 degrees and my car would not start.  My wife’s car also had a cold-induced flat tire.  So I got to change her tire as well.  On Sunday we had friends over so that day was scratched as well.

I still got 12 hours on the rack during the week, so I still felt pretty good about my progress.

Here are my goals from last week: 

Do Volume II lab 1.  Read OSPF chapter in Routing TCP/IP.  Redo Volume III lab 1.

I finished all of my goals this week.  I did Volume II lab 1 as a simulated lab.

Goals for this week:  Do Volume II lab 12.  Redo Volume II lab3.  Do Volume III lab 5. 

Days Until Lab: 117
Readiness (1 to 10): 2
Lab Hours This Week 12
Study Hours This Week (estimate): 3

February 5, 2008

Status Update: 27 January – 03 February

It had to happen sometime, but this weekend I completely dropped the ball on my studies.  I started the week out strong with some rare mid-week rack time, but ended up only getting about 4 hours on the rack over the weekend.  I really couldn’t get started on Saturday.  Between ennui and getting sucked into issues at work (work was really hectic last week) I only managed 4 hours on the rack.  I vowed to make up the time on Sunday.

I went out Saturday night to get together with a couple of friends that I haven’t seen in years.  My Google calendar had this event running from 6 pm – 8 pm.  Eat, shoot the shit, and get back home.  Enter the $3 margarita.  Needless to say, the night stretched way into Sunday morning.  Sunday my head was absolute mush.  I attempted to do some labbing, but could never get started.  I gave up and watched the Super Bowl.

Here are my goals from last week: 

Finish Volume II lab 6.  Redo Volume III lab 1.  Do the Volume I BGP labs.

I finished Volume II lab 6.  That’s all I finished.  This was a very good lab.  It was the first difficulty level 7 lab (equivalent to the difficulty of the actual lab) I have attempted.  It was a tough lab, but very fair.  I am still messing up enough that I have no chance of passing, but I’m not hitting much that I don’t know how to configure.

This Wednesday I have a day off from work.  I have rented some rack time and will attempt Volume I lab 1 as a mock lab.  I will simulate the lab enviroment by doing the lab from beginning to end without any resources except the DOC in an 8.5 (30 minutes for lunch) block.  Although this is the easiest of the Volume II labs (difficulty rating of 5) it will at least give me some idea of where I’m at time-management wise.  It will also be the first lab that I have attempted where I will not be verifying my solutions at the end of each section.  So an early mistake will probably go unnoticed.  I will not be blogging the lab with the detail that I have for the other labs.  I will eventually redo this lab and blog about it at that time.

I am also going to commit more time to reading.  I started rereading the OSPF chapter(s) in Routing TCP/IP again.  This is probably my third time through this material, but I am finding that I still pick up new bits each time as well as fortifying the stuff that I already know.

Goals for this week:  Do Volume II lab 1.  Read OSPF chapter in Routing TCP/IP.  Redo Volume III lab 1.

Days Until Lab: 117
Readiness (1 to 10): 2
Lab Hours This Week 10
Study Hours This Week (estimate): 3

January 30, 2008

Status Update: 20 – 26 January

I’m still having ups and downs with my studies.  I am at the point where I am able to recognize what technology solves each core task, but I’m still missing bits of configuration and making dumb mistakes.  I am also still pretty weak in BGP.  My time is horrible as well.  Some of that is due to me typing notes as I lab as well as searching for everything in the DOC.  Next Wednesday will be my first simulated lab (I’m doing Volume II lab 1).  This will give me a better idea of how slow I really am.

I don’t know if it’s the weather (it’s been under zero here for over a week – that translates to “insanely cold” in Celsius), but I’ve been a little disappointed with my progress lately.  I’m probably going take a week off pretty soon.  By “a week off”, I mean that I will not be labbing but I will still be reading and watching some IEATC videos.

Here are my goals from last week: 

Do Volume II lab 6.  Do Volume III lab 4.  Do 4 random labs from each of Volume I OSPF, EIGRP, RIP (12 total).

I finished Volume III lab 4 and I did my 12 random labs.  I only finished the first three sections (through IGP) redistribution of Volume II lab 6.

Goals for this week:  Finish Volume II lab 6.  Redo Volume III lab 1.  Do the Volume I BGP labs.

Days Until Lab: 123
Readiness (1 to 10): 2
Lab Hours This Week 12
Study Hours This Week (estimate): 3

January 22, 2008

Status Update: 14 – 20 January

Last week was a pretty good week.  I redid a couple of labs and watched a couple of the IEATC videos.  I don’t have a lot to report.  I think that redoing labs is a good idea, but I was surprised (and disappointed) that I fell for a couple of the same pitfalls the second time around.

I got a new laptop.  It has 2 Gigs of RAM and a duo-core CPU.  I loaded up the entire IE Dynamips lab.  The processor usage fluctuated between 60 – 90%.  I will try out a full-scale Volume III lab (minus some Layer 2 bits) this week to see if this laptop can handle the load.  I’m probably going to buy a CPU/Mobo combo for one of my desktops and drop Linux on it if this Windows laptop can’t handle the load.  I’ll probably end up buying the Dynamips version of the IE Volume II labs (only $99 because I already own the workbook – not sure if that’s $99 for each 10 lab installment or $99 for all 20 labs).  I really need to get cracking on more full-scale labs.  I am thinking about redoing a lab during the week (spread out over a couple of nights) and using the weekend for new labs.  I am probably going to do my first “mock lab” the weekend after this one.

Here are my goals from last week: 

Redo Volume II lab 3.  Redo Volume III lab 2.  Do 4 random labs from each of Volume I OSPF, EIGRP, RIP (12 total).

I managed to complete all of my goals this week.

Goals for this week:  Do Volume II lab 6.  Do Volume III lab 4.  Do 4 random labs from each of Volume I OSPF, EIGRP, RIP (12 total).

Days Until Lab: 131
Readiness (1 to 10): 2
Lab Hours This Week 20
Study Hours This Week (estimate): 2
« Previous PageNext Page »

Blog at