Uber-Lab

Okay, this is insane:

The $2.3 million home lab of Scott Morris, Quadruple CCIE

The $2.3 million home lab of Quadruple CCIE, Scott Morris, started out as a single standard 7-foot 19″ rack.

Bloomed to two standard racks…

Then it grew into three Ortronics Mighty-Mo 19″ rack systems.

Finally it blossomed into four separate rack cabinets.

Read the rest at Brad Reese’s blog

“Once we had a discussion on GroupStudy about the most expensive thing anyone had to purchase because they had their lab, and many people talked about racks, or cabling or the electric bill.”

“I mentioned that I needed to get a completely new air conditioner and place the equipment in a room by itself in order to combat the sheer amount of heat generated.”

“It’s all part of the entertainment in being an Uber-Geek!”

Be sure to check out the equipment list(s).

Where Have You Been All My Life?

I just recently discovered that you can sort the output of “show processes” commands.  The fact that I just recently found out about the sort option means that it’s probably been there for years. 🙂

r1#sh proc cpu ?
  history  Show CPU history in graph format
  sorted   Show sorted output based on percentage of utilization
  |        Output modifiers
  <cr>

r1#sh proc cpu sorted ?
  1min  Sort based on 1 minute utilization
  5min  Sort based on 5 minutes utilization
  5sec  Sort based on 5 seconds utilization
  |     Output modifiers
  <cr>

r1#sh proc cpu sorted
CPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes: 0%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  36      195092   5027891         38  0.07%  0.08%  0.07%   0 Per-Second Jobs
 149       16096  50258709          0  0.07%  0.02%  0.00%   0 CCPROXY_CT
   2         244   1005901          0  0.00%  0.00%  0.00%   0 Load Meter
   4           0         1          0  0.00%  0.00%  0.00%   0 EDDRI_MAIN
   3      109264   1968991         55  0.00%  0.00%  0.00%   0 OSPF Hello 100
   1           0         4          0  0.00%  0.00%  0.00%   0 Chunk Manager
…
 124          16        52        307  0.00%  0.02%  0.00%   2 Virtual Exec
…
 176     5404540    335473      16110  0.00%  0.06%  0.05%   0 BGP Scanner

Looks like the default is 5sec (otherwise we would not see entries further down the list for 1min and 5min).  I think that 5min is a better default.

r1#sh proc cpu sorted 5min
CPU utilization for five seconds: 1%/0%; one minute: 0%; five minutes: 0%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  36      195096   5028041         38  0.00%  0.07%  0.07%   0 Per-Second Jobs
 176     5404720    335483      16110  0.00%  0.10%  0.06%   0 BGP Scanner
   5     3289732    510713       6441  0.63%  0.09%  0.06%   0 Check heaps
  54      282196   2531018        111  0.07%  0.00%  0.00%   0 IP Input
  89        9648  50260186          0  0.07%  0.01%  0.00%   0 RBSCP Background

Because a process could hammer your cpu for 4minutes and 54 seconds and could be buried in the default sort (5sec), you may still want to use the unwieldy “sh proc cpu | e 0.00%  0.00%  0.00%” command as it will show processes that have used the cpu in any of the three time intervals:

r1#sh proc cpu | e 0.00%  0.00%  0.00%
CPU utilization for five seconds: 3%/1%; one minute: 1%; five minutes: 0%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   5     3289796    510727       6441  0.00%  0.04%  0.05%   0 Check heaps
  36      195104   5028200         38  0.07%  0.08%  0.07%   0 Per-Second Jobs
  54      282208   2531191        111  0.07%  0.01%  0.00%   0 IP Input
  89        9648  50261772          0  0.00%  0.01%  0.00%   0 RBSCP Background
 124          88      4060         21  0.15%  0.03%  0.00%   2 Virtual Exec
 145     1093612     85603      12775  0.23%  0.02%  0.00%   0 Per-minute Jobs
 149       16096  50261800          0  0.00%  0.02%  0.00%   0 CCPROXY_CT
 176     5404900    335494      16110  1.27%  0.15%  0.07%   0 BGP Scanner

Remember that you also have the “graphic” output of the “show processes cpu history”:

r1#show proc cpu hist

r1   01:44:11 PM Saturday Jun 30 2007 CDT

    11111                         11111               22222
100
 90
 80
 70
 60
 50
 40
 30
 20
 10
   0….5….1….1….2….2….3….3….4….4….5….5….
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)
    2322222222222222222232232222222222222222232222232333222222
100
 90
 80
 70
 60
 50
 40
 30
 20
 10
   0….5….1….1….2….2….3….3….4….4….5….5….
             0    5    0    5    0    5    0    5    0    5
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

                1     1122332332231            122222322221            122
    4846454332452553460424118209921765555565579423697077635535457533458356
100
 90
 80
 70
 60
 50
 40
 30                       ********                *******               ****
 20                     *########*              *########**             *###
 10  * * *     ****  **############************###########** * ***   **#####
   0….5….1….1….2….2….3….3….4….4….5….5….6….6….7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

Wow!!! That’s purty.  🙂

Cisco Documentation:

show processes cpu

Early Morning Hieroglyphics and 3750s

I stumbled into work this morning and started checking my email before I began to ingest my daily dose of caffeine.  I received the following email from a co-worker (switch names have been changed to protect the innocent – and to make sure I don’t get fired):

Subject: Nac change for 6 am
need a reload and the config change for NAC IOS code is there need reload
productionswitch01
productionswitch02
…

need to be reloaded
productionswitch03
productionswitch04

The guy who sent me the email is a great guy and English IS his native tongue, but typing is a chore for him.  I couldn’t even begin to parse that first sentence, so I decided to do the easy stuff first and just reload the last two switches.

Before I reload a switch I do a check to make sure that the configuration is written, all of the uplinks are up/up, that I have out-of-band (dial in) access (if available), and that the new IOS version is loaded and set to be booted.  The last bit was throwing me for a loop when I looked at the 3750s this morning:

productionswitch03#sh ver
Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(25)SEE1,
 RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2006 by Cisco Systems, Inc.
Compiled Mon 22-May-06 08:51 by yenanh
Image text-base: 0x00003000, data-base: 0x01210B18

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.1(14r)EA1a, RELEASE SOFTWA
RE (fc1)

productionswitch03 uptime is 23 weeks, 3 days, 8 hours, 52 minutes
System returned to ROM by power-on
System restarted at 20:22:40 cst Tue Jan 16 2007
System image file is “flash:c3750-ipservicesk9-mz.122-25.SEE1.bin”
…

Weird.  Generally I expect to see the new IOS version int the “System image file is” section.  Let’s make sure that there is a newer IOS version in flash:

The newer version is in flash (12.2(35)SE2)
productionswitch03#sh flash
Directory of flash:/
    2  drwx         128  Jan 16 2007 14:19:41 -06:00  c3750-ipservices-mz.122-25.SEB2
  360  -rwx         109  Feb 28 1993 19:49:49 -06:00  info
  361  -rwx        1216  Feb 28 1993 18:01:08 -06:00  vlan.dat
    3  -rwx       14093  Feb 28 1993 18:01:37 -06:00  config.text
    5  -rwx        4917  Feb 28 1993 18:01:38 -06:00  private-config.text
  362  drwx         192  Jun 29 2007 21:02:20 -05:00  c3750-ipservicesk9-mz.122-35.SE2
    6  -rwx        3096  Feb 28 1993 18:01:38 -06:00  multiple-fs

Is there something in the configuration pointing to the newer IOS?

productionswitch03#sh run | i boot
productionswitch03#

Nothing.  Hmmm….let’s check the “bootvar”…well, it’s just “boot” on the 3750:

productionswitch03#sh boot
BOOT path-list      : flash:c3750-ipservicesk9-mz.122-35.SE2/c3750-ipservicesk9-
mz.122-35.SE2.bin
Config file         : flash:/config.text
Private Config file : flash:/private-config.text
Enable Break        : no
Manual Boot         : no
HELPER path-list    :
Auto upgrade        : yes

Sweet.  So the newer IOS image is in flash and the switch will boot to that image.  But we did not configure it to choose that image, so why would it boot to the newer image versus the older one?  Here’s what the 3750 Configuration Guide has to say about that:

“By default, the switch attempts to automatically boot the system using information in the BOOT environment variable. If this variable is not set, the switch attempts to load and execute the first executable image it can by performing a recursive, depth-first search throughout the flash file system. In a depth-first search of a directory, each encountered subdirectory is completely searched before continuing the search in the original directory. However, you can specify a specific image to boot.”

The 3750 IOS Command Guide says the following about the “BOOT path-list”:

“Displays a semicolon separated list of executable files to try to load and execute when automatically booting.”

That explains why both IOS versions are listed after that variable.  I am still interested in how the switch chose to boot to the newer image first.  Does the switch choose the most recently added image if “boot system” is not configured?  Or did we just get lucky in that the switch found the newer IOS first in its “recursive, depth-first search throughout the flash file system”?

I should have mentioned this before, but the switch that I am looking at is a single switch (not a member of a stack).

Maybe the answer lies in the “archive download-sw” command used to add the new IOS version to flash.  Here are the configuration options for that command:

archive download-sw {/force-reload | /imageonly | /leave-old-sw | /no-set-boot | /no-version-check | /destination-system stack-member-number | /only-system-type system-type | /overwrite | /reload | /safe} source-url

I’m not sure what variables my co-worker used when downloading the IOS from the TFTP server.  I generally use “/overwrite”.  It doesn’t look like he used that option as both images were still in flash.

Ah!  Here’s the answer:

Defaults
The current software image is not overwritten with the downloaded image.
Both the software image and HTML files are downloaded.
The new image is downloaded to the flash: file system.
The BOOT environment variable is changed to point to the new software image on the flash: file system.
Image names are case sensitive; the image file is provided in tar format.
Compatibility of the stack protocol version on the image to be downloaded is checked with the version on the switch stack.

That explains why the switch booted to the new image.  It doesn’t seem that the IOS explicitly configures the “boot system” variable (in the running/startup config), but it does update the BOOT path-list to choose the IOS that was most recently added to flash as the first IOS version to boot.

Let’s reload this sucker and see what IOS version it boots:

After reload:
Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(35)SE2,
RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Tue 06-Feb-07 05:07 by antonino
Image text-base: 0x00003000, data-base: 0x0138517C

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.1(14r)EA1a, RELEASE SOFTWARE (fc1)

productionswitch03 uptime is 3 minutes
System returned to ROM by power-on
System restarted at 06:31:22 CDT Sat Jun 30 2007
System image file is “flash:c3750-ipservicesk9-mz.122-35.SE2/c3750-ipservicesk9-mz.122-35.SE2.bin”

productionswitch03#sh boot
BOOT path-list      : flash:c3750-ipservicesk9-mz.122-35.SE2/c3750-ipservicesk9-mz.122-35.SE2.bin
Config file         : flash:/config.text
Private Config file : flash:/private-config.text
Enable Break        : no
Manual Boot         : no
HELPER path-list    :
Auto upgrade        : yes
Auto upgrade path   :
productionswitch03#sh flash

Directory of flash:/

    2  drwx         128  Jan 16 2007 14:19:41 -06:00  c3750-ipservices-mz.122-25.SEB2
  360  -rwx         109  Feb 28 1993 19:49:49 -06:00  info
  361  -rwx        1216  Feb 28 1993 18:01:08 -06:00  vlan.dat
    3  -rwx       14093  Feb 28 1993 18:01:37 -06:00  config.text
    5  -rwx        4917  Feb 28 1993 18:01:38 -06:00  private-config.text
  362  drwx         192  Jun 29 2007 21:02:20 -05:00  c3750-ipservicesk9-mz.122-35.SE2
    6  -rwx        3096  Feb 28 1993 18:01:38 -06:00  multiple-fs

15998976 bytes total (4133376 bytes free)

Mystery solved!  Not really a CCIE issue, but good to know anyways.  Now to figure out what the fuck “need a reload and the config change for NAC IOS code is there need reload” means.  🙂

Cisco Documentation:

Catalyst 3750 Switch Command Reference
Cisco IOS Release 12.2(25)SEE

Catalyst 3750 Switch SoftwareConfiguration Guide
Cisco IOS Release 12.2(25)SEE

VTP MD5 Hash Utilizes VTP Domain Name

In a previous post, I hypothesized that VTP  takes the VTP domain name into account when calculating the VTP MD5 hash.  I decided to test this by configuring two switches with the same VTP configurations (and not adding any vlans) except for the VTP domain name.  If I am correct, then the VTP MD5 hash of the two switches should be different.

Here’s our first switch that has been defaulted:
sw1#sh vtp status
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Transparent
VTP Domain Name                 :
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x57 0xCD 0x40 0x65 0x63 0x59 0x47 0xBD
Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00
sw1#sh vtp pass
The VTP password is not configured.

Let’s configure this sucker:
sw1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
sw1(config)#vtp domain DOMAIN1
Changing VTP domain name from NULL to DOMAIN1
sw1(config)#vtp mode server
Setting device to VTP SERVER mode
sw1(config)#vtp password PASSWORD
Setting device VLAN database password to PASSWORD
sw1(config)#^Z
sw1#sh vtp
*Mar  1 00:29:22: %SYS-5-CONFIG_I: Configured from console by console
sw1#sh vtp statu
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Server
VTP Domain Name                 : DOMAIN1
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0xB9 0x5B 0x76 0xF1 0x64 0x83 0x17 0xF3
Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00
Local updater ID is 0.0.0.0 (no valid interface found)

Okay.  Here’s the second switch:
sw2#sh vtp status
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Transparent
VTP Domain Name                 :
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x57 0xCD 0x40 0x65 0x63 0x59 0x47 0xBD
Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00
sw2#sh vtp pass
The VTP password is not configured.

Let’s give it the same configuration as sw1 except for the VTP domain name:
sw2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
sw2(config)#vtp domain DOMAIN2
Changing VTP domain name from NULL to DOMAIN2
sw2(config)#vtp mode server
Setting device to VTP SERVER mode
sw2(config)#vtp pass PASSWORD
Setting device VLAN database password to PASSWORD
sw2(config)#^Z
sw2#sh vtp
*Mar  1 00:30:36: %SYS-5-CONFIG_I: Configured from console by console
sw2#sh vtp statu
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Server
VTP Domain Name                 : DOMAIN2
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x2E 0xC6 0x1E 0x6E 0xA6 0xC3 0xA7 0x86
Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00
Local updater ID is 0.0.0.0 (no valid interface found)

Let’s compare the two MD5 hashes (different VTP domain names):
sw1: MD5 digest: 0xB9 0x5B 0x76 0xF1 0x64 0x83 0x17 0xF3
sw2: MD5 digest: 0x2E 0xC6 0x1E 0x6E 0xA6 0xC3 0xA7 0x86

They are different.  This proves my hypothesis.  Now let’s Change sw2’s domain to DOMAIN1 (to match sw1) and see if the MD5 hashes match:

sw2(config)#vtp domain DOMAIN1
Changing VTP domain name from DOMAIN2 to DOMAIN1
sw2(config)#^Z
sw2#sh
*Mar  1 00:32:30: %SYS-5-CONFIG_I: Configured from console by console
sw2#sh vtp statu
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Server
VTP Domain Name                 : DOMAIN1
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0xB9 0x5B 0x76 0xF1 0x64 0x83 0x17 0xF3
Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00
Local updater ID is 0.0.0.0 (no valid interface found)

sw1: MD5 digest: 0xB9 0x5B 0x76 0xF1 0x64 0x83 0x17 0xF3
sw2: MD5 digest: 0xB9 0x5B 0x76 0xF1 0x64 0x83 0x17 0xF3

---

Booyah!!!!  This proves that the VTP MD5 hash takes into account the VTP domain name as well as the password.

Completely Clearing a Cisco Switch…Kinda

I found this post on the excellent Cisco Blog:

Completely Clearing a Cisco Switch…The Easy Way!

Clearing out a Cisco switch configuration is always a pain because VLANs are kept in a seperate file from the startup-config (NVRAM). There’s two ways to clear a switch back to the factory defaults – the easy way and the REALLY easy way:

The easy way –

Switch# write erase
Switch# delete flash:vlan.dat
Switch# reload

The REALLY easy way –

Hold the “mode” button on the front of the switch for 10 seconds. The lights will blink then go solid – the switch completely wipes all configuration and then reboots. Obviously, this method only works on stackable switches as the chassis based switches do not have mode buttons.

Comments

Umm, is it me or is this not a terribly great feature. I know physical security is part of maintaining a secure environment, but this kind of kicks that higher up the requirements chain.

Any know of a way of disabling this ‘feature’?

Posted by: Scared at May 20, 2007 7:21 PM

In my experience this does not wipe the switch at all, it just renames the configfiles and reboots.

“no setup express” disables the button.

Posted by: ior at May 25, 2007 3:08 PM 

Let’s attempt to replicate this on a 3560 and see what happens.

Let’s go ahead and add some vlans, vtp config, and some descriptions on the first 5 interfaces:
Switch(config)#vtp domain CISCO
Changing VTP domain name from NULL to CISCO
Switch(config)#vtp mode server
Device mode already VTP SERVER.
Switch(config)#vtp password PASSWORD
Setting device VLAN database password to PASSWORD
Switch(config)#int range fa0/1 – 5
Switch(config-if-range)#desc -> I like beer!!!
Switch(config-if-range)#vlan 2-10,13,69
Switch(config-vlan)#^Z
Switch#sh vtp
00:07:19: %SYS-5-CONFIG_I: Configured from console by consolestatus
VTP Version                     : 2
Configuration Revision          : 1
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 16
VTP Operating Mode              : Server
VTP Domain Name                 : CISCO
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x9C 0x62 0xCB 0xFE 0xB7 0x89 0x4A 0xB8
Configuration last modified by 0.0.0.0 at 3-1-93 00:07:19
Local updater ID is 0.0.0.0 (no valid interface found)
Switch#sh vlan sum
Number of existing VLANs           : 16
 Number of existing VTP VLANs      : 16
 Number of existing extended VLANs : 0

Switch#sh int desc
Interface                      Status         Protocol Description
Vl1                            admin down     down
Fa0/1                          down           down     -> I like beer!!!
Fa0/2                          down           down     -> I like beer!!!
Fa0/3                          down           down     -> I like beer!!!
Fa0/4                          down           down     -> I like beer!!!
Fa0/5                          down           down     -> I like beer!!!

and the hostname 🙂

Switch(config)#hostname sw1
sw1(config)#^Z
sw1#
00:08:10: %SYS-5-CONFIG_I: Configured from console by console
sw1#write
Building configuration…
[OK]

Now let’s take a look at the flash:
sw1#sh flash

Directory of flash:/

    2  -rwx        1216   Mar 01 1993 00:07:19  vlan.dat
    3  -rwx           5   Mar 01 1993 00:08:26  private-config.text
    5  drwx         192   Mar 01 1993 00:05:28  c3560-i9-mz.121-19.EA1d
   85  -rwx        3485   Mar 01 1993 00:08:26  config.text

15998976 bytes total (9540608 bytes free)

Let’s go ahead and hold the mode button for 10 seconds:

As I hold the mode button – STAT, DUPLX, SPEED, and PoE lights blink together and then go solid.  I stop pushing the mode button at that point.

Notice this message after I release the mode button:
sw1#
00:09:18: %SYS-7-NV_BLOCK_INIT: Initalized the geometry of nvram
00:09:18: %EXPRESS_SETUP-6-CONFIG_IS_RESET: The configuration is reset and the system will now reboot
00:09:19: %SYS-5-RELOAD: Reload requested

The switch comes back up:
Switch uptime is 2 minutes
System returned to ROM by power-on

Switch#sh start
startup-config is not present
Switch#sh flash

Directory of flash:/

    2  -rwx        1216   Mar 01 1993 00:07:19  vlan.dat
    3  -rwx           5   Mar 01 1993 00:08:26  private-config.text.renamed
    5  drwx         192   Mar 01 1993 00:05:28  c3560-i9-mz.121-19.EA1d
   85  -rwx        3485   Mar 01 1993 00:08:26  config.text.renamed

15998976 bytes total (9540608 bytes free)

The VTP configuration is still intact (due to vlan.dat not being deleted):
Switch#sh vtp statu
VTP Version                     : 2
Configuration Revision          : 1
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 16
VTP Operating Mode              : Server
VTP Domain Name                 : CISCO
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x9C 0x62 0xCB 0xFE 0xB7 0x89 0x4A 0xB8
Configuration last modified by 0.0.0.0 at 3-1-93 00:07:19
Local updater ID is 0.0.0.0 (no valid interface found)
Switch#sh vlan sum
Number of existing VLANs           : 16
 Number of existing VTP VLANs      : 16
 Number of existing extended VLANs : 0

Interesting.  The startup-configuration is blown away but still appears in flash, albeit with “renamed” tacked on the end.  The vlan.dat file is still present, so we retain our VTP settings.  The private-config.text* file has also been retained, but with “renamed” appended to it as well. 

* According to this page, the private-config.text file is used by Cisco devices to store things like crypto private keys.

Let’s remove the “renamed” portion of the config.text file and restore our startup-configuration:

Switch#rename flash:config.text.renamed flash:config.text
Destination filename [config.text]?

Switch#sh start
Using 3485 out of 524288 bytes
!
version 12.1
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname sw1
!
!
ip subnet-zero
!
!
spanning-tree mode pvst
no spanning-tree optimize bpdu transmission
spanning-tree extend system-id
!
!
interface FastEthernet0/1
 description -> I like beer!!!
 no ip address
 no mdix auto
!
interface FastEthernet0/2
 description -> I like beer!!!
 no ip address
 no mdix auto
…..

So the config.text.renamed is the startup config.  It’s good to know that this is still around.  We can just rename it and then do “copy start run”

Switch#copy start run
%% Non-volatile configuration memory invalid or not present

Zoiks!!!  Maybe not! 🙂

Let’s rename the private-config file and see if that helps.

Switch#$h:private-config.text.renamed flash:private-config.text
Destination filename [private-config.text]?
Switch#sh flash

Directory of flash:/

    2  -rwx        1216   Mar 01 1993 00:07:19  vlan.dat
    3  -rwx           5   Mar 01 1993 00:08:26  private-config.text
    5  drwx         192   Mar 01 1993 00:05:28  c3560-i9-mz.121-19.EA1d
   85  -rwx        3485   Mar 01 1993 00:08:26  config.text

15998976 bytes total (9540608 bytes free)

Switch#copy start run
%% Non-volatile configuration memory invalid or not present

Nope.  Interesting.  Well a reload ought to do the trick

Switch#reload
Proceed with reload? [confirm]

00:27:04: %SYS-5-RELOAD: Reload requested
And it works:

sw1>en
sw1#sh run
Building configuration…

Current configuration : 3512 bytes
!
version 12.1
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname sw1
!
!
ip subnet-zero
!
!
spanning-tree mode pvst
no spanning-tree optimize bpdu transmission
spanning-tree extend system-id
!
!
interface FastEthernet0/1
 description -> I like beer!!!
 no ip address
……

Okay.  Let’s see if we can turn this behavior off by configuring “no express setup”:

sw1(config)#no setup ?
  express  Configure whether express setup mode is enabled

sw1(config)#no setup express

Let’s see what’s in flash and then write the config:
sw1#sh flash

Directory of flash:/

    2  -rwx        3378   Mar 01 1993 00:06:38  config.text
    3  -rwx           5   Mar 01 1993 00:06:38  private-config.text
    4  -rwx         976   Mar 01 1993 00:10:31  vlan.dat
    5  drwx         192   Mar 01 1993 00:05:28  c3560-i9-mz.121-19.EA1d

15998976 bytes total (9541120 bytes free)
sw1#write
Building configuration…
[OK]
sw1#

Now let’s hold the mode button and see what happens.

Not a damned thing!!! 🙂  I held the mode button down and all it ever did was move the light from STAT to DUPLX.

I would say that “no express setup” is a great command to throw into your base configuration for all of your switches.  While holding down the mode button until the lights are solid will default your configuration, that configuration will not be lost and your vlan.dat file will not be deleted (your vtp configuration will persist).  At least on the 3560, this is not a method that will acheive the same results as write erase, delete flash:vlan.dat, reload.

For comparison, here’s what happens with when we do these steps:

sw1#sh flash

Directory of flash:/

    2  -rwx           5   Mar 01 1993 00:11:59  private-config.text
    4  -rwx         976   Mar 01 1993 00:10:31  vlan.dat
    5  drwx         192   Mar 01 1993 00:05:28  c3560-i9-mz.121-19.EA1d
   85  -rwx        3555   Mar 01 1993 00:11:59  config.text

15998976 bytes total (9541120 bytes free)
sw1#write erase
Erasing the nvram filesystem will remove all configuration files! Continue? [confirm]
[OK]
Erase of nvram: complete
sw1#sh fla
00:16:59: %SYS-7-NV_BLOCK_INIT: Initalized the geometry of nvram

Write erase deletes “private-config.text” and “config.txt” from flash:
sw1#sh flash

Directory of flash:/

    4  -rwx         976   Mar 01 1993 00:10:31  vlan.dat
    5  drwx         192   Mar 01 1993 00:05:28  c3560-i9-mz.121-19.EA1d

15998976 bytes total (9545216 bytes free)
sw1#delete flash:vlan.dat
Delete filename [vlan.dat]?
Delete flash:vlan.dat? [confirm]
sw1#sh flash

Directory of flash:/

    5  drwx         192   Mar 01 1993 00:05:28  c3560-i9-mz.121-19.EA1d

15998976 bytes total (9546240 bytes free)
sw1#sh start
startup-config is not present
sw1#reload

System configuration has been modified. Save? [yes/no]: no
Proceed with reload? [confirm]

00:18:44: %SYS-5-RELOAD: Reload requested
….

After the reload:
Switch>en
Switch#sh flash

Directory of flash:/

    5  drwx         192   Mar 01 1993 00:05:28  c3560-i9-mz.121-19.EA1d

15998976 bytes total (9546240 bytes free)

---

To summarize: Resetting the switch will not acheive the same result as deleting the vlan.dat, doing a “write erase”, and then reloading the switch.  It will blow away your startup-configuration, but the good news is that you can easily recover that file (along with the private-config.text file) and restore your configuration by just renaming a couple of files and reloading.  I would strongly recommend disabling this “feature” by using the “no express setup” command in your switch’s configuration.  There is no reason to allow someone to blow away your switch’s configuration by simply pressing a button. 🙂

Catalyst 6500 On The CCIE Lab???

Rumor?  Follow the debate on the GroupStudy thread:

Rumor mill time – Cat 6500 on R&S lab exam?

Some of the responses:

“Not sure of the point of that in the R&S lab…. Really don’t see them starting to test on the service modules on the 6500 in the R&S.  And, in many cases the 3560 has greater feature velocity.  So, it would cost the CCIE program more money, a more rack space, but doesn’t really give them many more test topics.” – Mike Kraus

“Actually i hope they do that, because that would be the closest thing to the real world. No matter where you go these days, you see at least one 6500 series up and running. I think it would benefit the program. You may see a 3560 here and there, but you will definitely see a lot of firms having 6500 series. To be honest i typically see a mixture of 4000/4500 and a 6500 in a collapsed core design.” – Narbik Kocharians

“So you think it helps the value of the CCIE that it can produce a CCIE that knows NOTHING about a 6500? What is cisco’s top product? What are you considered if you are a CCIE. Doesn’t mesh in my mind. Totally disagree.” – Jason Plank

My favorite:

“Nope nope! Cisco has recently faced some severe budget limitations and brought back the 2500s and L2 1900 switches!! :p” – Digital Yemeni

It’s a Psychology Test

The GroupStudy email list includes quite a few folks who have passed the CCIE lab sharing their experiences and tips.  Here’s one of the more recent ones:

It’s a Psychology Test

I just want to start by saying thank you to everyone on Groupstudy,
the guys over at InternetworkExpert, and all the 50 pound brains at my
old shop.  I passed my R/S lab yesterday and now I don’t know what to
do with myself.  I think I’m in a bit of shock.

It really is all about psychology.  I know everyone always says this,
but I’m a firm believer now.  I went in for my first attempt yesterday
morning, but I’m convinced that the test really started four days ago
when I stopped studying.  I made it a point NOT to study the few days
prior to my test.  It was really hard not to go digging around in the
doc cd or try and do one or two more labs, but that may have been the
best decision I made over the course of my studies.  It REALLY helped
me keep the stress down and get a good nights sleep the night before.
Of course, the wonderful bed at the Residence Inn helped a little on
that part, too.  When I got to the testing facility in the morning at
RTP, I was refreshed and ready to go.

Now for the study stuff.  Dynamips is a godsend.  It can be really
annoying and really buggy.  But for guys on a budget like me, it’s
perfect.  Once you get the kinks worked out, you can lab anything up
in minutes.  I’ll probably keep using it from here on out for quick
testing at work.  The workbooks from InternetworkExpert are wonderful.
Their advanced technology series are a great place to start your
labbing.  The core workbooks are excellent for building your speed.
And their main workbooks are great for building technique and
developing good lab based critical-thinking.  Reading the Doc CD can’t
be stressed enough, either.  Use it as your primary source of
information.  It’s your only asset in the lab, so why not base your
studies off of it?  It helped me a lot when I was unsure of something.
It should only take 15-20 seconds to look something up for
clarification.  Lastly, don’t get overwhelmed.  It’s a lot of
material.  But, it’s not so bad if you break it up.

I guess it’s time to get back to the real world.

Thanks again for all your help.
Blaine Williams, CCIE #18316
Network Architecture Engineer
University of South Carolina

I like the advice about not studying for the last four days.  I doubt that I would have the willpower to avoid studying right before the test though.   It also looks like more people are using Dynamips for lab preparation.  I’ve loaded Dynamips on my laptop at home and on my PC at work so I can do labs (all small-scale labs at this point) without renting a rack or firing up the stack of hardware on my desk.  I’m on board with his recommedation of the Internetwork Expert COD (I just started viewing them and am very impressed).  Blaine’s final point about finding information in the Doc CD is very sound.  I am still fumbling about when using this resource.  I usually use it when trying to find information if I am labbing, but I find myself still using Google most of the time when troubleshooting at work.

LFU2 – Know Your Acronyms

This one is from my first lab rental.  I was just getting my feet wet with lab rentals (especially use reverse telnet from the access router) and decided to do a couple of old CCNP labs.  I decided to warm up with some simple VTP labs. 

For whatever reason I simply could not get the two switches to exchange VTP information.  I’ll spare you the ugly details, but an hour and a half into my “easy” lab I had exhausted my repertoire of VTP troubleshooting (“sh vtp status”, “sh vtp counters”, “debug debug sw-vlan vtp events”, etc.) and I still could not get the damned switches to exchange VTP information.  The links were up and passing frames.  VTP was configured correctly (same VTP domain, no revision conflicts, etc.).  I could not figure out what I had done wrong.

Finally it dawned on me…what does VTP stand for?  VLAN Trunking Protocol.  Well, I had VLANS.  Did I have trunking?  After a quick “show int trunk” I discovered the reason that my switches weren’t passing VTP information.  My links were up, but they were not trunking.  The reason for this was another misunderstanding on my part.  I assumed (always dangerous) that DTP (Dynamic Trunking Protocol) had hooked me up with trunks between these two 3560s.  Wrong!!!  As I mentioned in an earlier post, the default DTP configuration on the 3560 is “switchport mode dynamic auto”.  A connection between two 3560s will NOT form a trunk by default.  The 3550’s default DTP setting is “switchport mode dynamic desirable” which means that it will from a trunk with another switch unless that other switch’s port is in “switchport mode access” or “switchport nonegotiate”.

Even with my confusion over whether DTP would create the trunks for me automatically, I should have verified that trunking was working very early on in my troubleshooting rather than 1.5 hours later. 😦

Fun With VTP Passwords

As I mentioned earlier, I did a ton of VTP labbing last weekend.  I’ll be posting some of the more interesting/strange results.  A lot of this will not be applicable to the lab, but you may come across some of this in real life.  I’ve never worked on a network that actually ran VTP except for using the domain name to identify LANs for CiscoWorks (all of the switches were in transparent mode).  I’ve never run VTP server/client in a production network and my only experience with that type of setup was during my CCNP studies.  ‘Nuff said, on to my adventures with the VTP password.

1) The VTP password can be set from the privileged exec mode:

sw4#sh vtp pass
The VTP password is not configured.
sw4#vtp pass MYPASSWORD
Setting device VLAN database password to MYPASSWORD
sw4#sh vtp pass
VTP Password: MYPASSWORD

2) It makes sense that the VTP password can be removed from privileged exec mode as well:

sw4#sh vtp password
VTP Password: MYPASSWORD
sw4#no vtp password
Clearing device VLAN database password.
sw4#sh vtp password
The VTP password is not configured.

3) You cannot set a VTP password without first configuring a VTP domain.  This makes sense,  you couldn’t really authenticate another switch if it wasn’t in the same VTP domain:

No VTP domain, no VTP password:
sw4#vtp pass MYPASSWORD
 %The VTP password cannot be set for NULL domain
sw4#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
sw4(config)#vtp domain CISCO
Changing VTP domain name from NULL to CISCO
sw4(config)#^Z
*Mar  1 12:58:08: %SYS-5-CONFIG_I: Configured from console by console

We can set the VTP password after setting the VTP domain 
sw4#vtp pass MYPASSWORD
Setting device VLAN database password to MYPASSWORD
sw4#sh vtp password
VTP Password: MYPASSWORD

4) The MD5 hash of a null (default) password and a cleared (“no vtp password”) VTP password are different:

Default switch VTP status:
sw4#sh vtp statu
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Server
VTP Domain Name                 :
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x57 0xCD 0x40 0x65 0x63 0x59 0x47 0xBD
Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00
Local updater ID is 0.0.0.0 (no valid interface found)

VTP status after VTP password cleared (“no vtp pass”):
sw4#sh vtp status
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Server
VTP Domain Name                 : CISCO
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0xD3 0x78 0x41 0xC8 0x35 0x56 0x89 0x97
Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00
Local updater ID is 0.0.0.0 (no valid interface found)

MD5 digest(default)                      : 0x57 0xCD 0x40 0x65 0x63 0x59 0x47 0xBD
MD5 digest(password cleared)    : 0xD3 0x78 0x41 0xC8 0x35 0x56 0x89 0x97

I think that the reason for this is that the switch uses the VTP domain name in its calculation of the VTP MD5 hash.  I think a way to prove this would be to set up two (default) switches in different VTP domains with the same password and then compare the MD5 hashes.  I’ll try this tomorrow and drop the results into this entry.

Update: I was right about the MD5 hash using the VTP domain name in its calculation.

5) This one is obvious from the above entries, but anyone in privileged exec mode can see the VTP password with “show vtp password” command.  Since a switch in VTP server or client mode does not keep the VTP configuration in the running-configuration (more on that later), this is the only way to verify the VTP password on switches running in those VTP modes (switches in VTP transparent mode will show the VTP configuration in the running-configuration)

Cisco Documentation:

Configuring VTP

New Entry-Level Cisco Certification

I received an email from Cisco today announcing a new entry-level certification.  The CCENT will be one step down the certification ladder from the CCNA.  It sounds pretty unnescessary to me, but I do like that they are going to attempt to revise the CCNA “to include a greater breadth of networking topics and more focus on performance-based skills to differentiate Cisco-certified applicants in the IT job market.”

Cisco Introduces New Entry-Level Certification and Updates to CCNA
 

On June 25, 2007, Cisco announced the addition of a new entry-level certification, Cisco Certified Entry Network Technician (CCENT) to the Career Certifications program. Cisco also introduced significant enhancements to the popular CCNA certification. CCENT presents a new point of entry for those just beginning to build a career in networking. As an optional stepping stone to CCNA, CCENT validates the skills required to successfully install and verify basic networks—a requirement for most entry-level network support positions. At the same time, Cisco’s foundational CCNA curricula has been revised to include a greater breadth of networking topics and more focus on performance-based skills to differentiate Cisco-certified applicants in the IT job market.

The CCNA curricula will be available worldwide on July 26, 2007 and exams will be released on August 1, 2007. Both curricula and exams will be initially released in English . Versions in simplified Chinese, Japanese, and Spanish will also be made available in December 2007. Additional languages will be announced as translation plans are finalized.

The CCENT learning path includes:
A required exam: ICND1 640-822
A recommended course: Interconnecting Cisco Networking Devices, Part 1

The CCNA learning path includes:
Required exams: ICND1 640-822 and ICND2 640-816, or CCNA 640-802 composite exam

Recommended courses: Interconnecting Cisco Networking Devices, Part 1 and Interconnecting Cisco Networking Devices, Part 2For more information on the CCENT and CCNA certifications, watch the “CCNA and CCENT Certification Overview” video posted on the CCNA Prep Center. The CCNA Prep Center which is available to anyone with a Cisco.com login helps candidates prepare for the CCENT and CCNA exams.

CCNA Questions and Answers

CCENT Questions and Answers

Additional CCNA and CCENT information.

Regards,
Cisco Career Certifications