CCIE Pursuit Blog

August 10, 2009

Internetwork Expert Volume IV (Troubleshooting) Workbook Review: Part 2

Access to the Volume IV workbook is the same as for all INE workbooks.  Once you’re logged in you’ll find (at present) three troubleshooting labs available.  The workbook will eventually contain 10 troubleshooting scenarios.  Each lab contains 10 trouble tickets which vary in value from 2 – 4 points.  You’ll also get the lab topology, along with initial configurations and the solution guide.

INE Workbook IV

INE Workbook IV

I rented some rack time at INE’s rack rental company, Graded Labs.  I booked a single, 5.5 hour session and planned to attempt the first two labs.  You’re allotted 2 hours for each lab, so I figured I should be able to complete both labs.  I was very wrong.

Graded Labs has a number of configurations for the INE product line which you can choose to have automatically loaded to your rack.  Unfortunately Graded Labs does not have the configs for the new Volume IV workbook yet.  This means that you will initially need to connect to each device and load in the appropriate config.  I had rented rack 16 and was worried that I would need to parse the initial configurations replacing IP addresses to fit the rack number, but INE has the initial configs for 31 different racks (1-30 and the mysterious rack 42) so you don’t need to go through the configs and change the IP addresses to match the rack.  It was just a matter of cutting and pasting in the appropriate configurations.

Initial Configurations For 31 Racks Included

Initial Configurations For 31 Racks Included

After loading the configurations, it was time to take a look at the lab.  I would very strongly recommend that you take the time to read the Workbook Introduction.  This six page PDF explains INE’s methodological approach to troubleshooting.  You will want to be familiar with it before attempting the labs as it will help you with troubleshooting as well as prepare you for the explanations in the solution guide.

After digesting the Workbook Introduction, it was time to look at the lab.  This is where the first surprises appear.

Lab Do’s and Don’ts:
•  Do not access the routers that are marked as restricted for your access.
•  Do not use the show running-config or show startup-config commands or their equivalents when performing troubleshooting.
•  Do not change or add any IP addresses from the initial configuration unless required for troubleshooting.
•  Do not change any interface encapsulations unless required for troubleshooting.
•  Do not change the console, AUX, and VTY passwords or access methods unless otherwise specified.
•  Do not use any static routes, default routes, default networks, or policy routing unless otherwise specified.
• Save your configurations often.

Those first two restrictions are killers.  While you will use the tried and true INE lab topology for these labs, each lab will include a number of restricted devices – outside of the normally restricted backbone devices.

The topology used for every scenario is the same that we use for all our RS products, including VOL1 (technology- focused labs), VOL2 (configuration mock lab scenarios) and VOL3 (core technologies scenarios).

However, unlike our previous workbooks, we restrict access to some of the devices in the lab topology. For every scenario this “restricted” set may be different and it is clearly outlined in the scenario’s baseline. Using this technique we increase the scenario complexity by allowing candidates to see only “one” side of the problem. When looking at the lab diagram, you will clearly see routers not under your control as being displayed in orange color. Also, when you log onto the “restricted” device, it will warn you using a banner message.

Lab 1 Topology - Note Restricted Devices in Orange

Lab 1 Topology - Note Restricted Devices in Orange

In lab 1 for instance, you are not allowed to access BB1, BB2, BB3, R2, R3, SW3, and SW4.  You’re pretty much on the honor system for the internal network devices as the you’ll be warned by a banner message, but only the first time you connnect (or if you telnet to the device(s)).

For instance, r3 is a restricted device:

Trying … Open

User Access Verification

Username: cisco

*                                                                *
*           Per the requirements of this scenario                *
*         You are not allowed to access this router              *
*                                                                *


You will not see the exec banner as long as the session to the console line is open from from the access server.  Regardless, it is important that you do not access the verboten devices during the lab.

While the restricted devices threw me an unexpected curve, it was the second requirement that really floored me.

In addition to the above restriction, we highly encourage you not using the show running-configuration, show startup-configuration commands or any other command that shows you the textual representation of the router’s configuration. This requirement makes you focus on using the show and debugging commands, which is invaluable when troubleshooting the real-world scenarios.

Our ultimate goal is not only prepare you for passing the Troubleshooting section of the CCIE R&S lab exam, but also to teach you a structured troubleshooting approach. As opposed to simple guessing and peeking at the routers running configurations you should learn using the debugging commands and interpreting various show commands output.

INE is serious about this too.  The solution guides will only use show and debugging commands to determine the root cause of each issue.  I don’t know if this will be a requirement in the actual lab; I certainly hope not!  In “real life” though, the use of debug commands is pretty much forbidden in the environments that I’ve worked in.  There’s a saying at my job “If you turn on a debugging command you had better have created a network change ticket, otherwise you’ve just created a job change ticket.”  Still, using the debug commands is a very good way to understand the underlying technology and there are some instances where you will get important troubleshooting information for a debug command that you cannot get in a show command.

For every ticket, we are going to follow the same structured procedure to resolve the issue.

Here is an outline of this procedure:

1. Build and Analyze the Baseline
2. Analyze the Symptoms (propose hypothesis)
3. Isolate the issue (gather more symptoms)
4. Fix the Issue (by comparing to the Baseline)

We recommend making your own diagrams, including the following
• IP addressing + IGPs.
• Layer 2 topology.
• BGP diagram.
• IPv6 topology.
• Multicast and Redistribution diagram.

Overall, don’t spend too much time building the baseline – the goal is to spend around 20 minutes. By the end of the baseline analysis phase, you should have clear understanding of the protocols and applications deployed in your network.

This is a brief review of the systematic troubleshooting procedure that you’ll be using for the labs.  You’ll be familiar with the topology drawing bit from your practices labs.  Although, I did pick up some good tips from the solution guide about making topology maps.

Okay, enough jibba jabba about the labs.  Let’s actually dive into one.  That will be the focus of the next part of the review.

Create a free website or blog at