DOH! Part 1 of Too Many: send *

I got my 2511 hooked up and running as an access server for my home lab.  I’m loving not having to move the console cable.  One of the other benefits of using the 2511 is that I can issue commands from the access server to be sent to all devices connected to it.

By using the “send *” command on the access server and then entering a list of commands (abort with control+c, send with control+z) you can quickly make changes to all devices at once.

Changing Config on All Devices

a1#send *
Enter message, end with CTRL/Z; abort with CTRL/C:
conf t
router rip
network 10.0.0.0
exit
copy run start
^Z
Send message? [confirm]
a1#
***
***
*** Message from tty0 to all terminals:
***
conf t
router rip
network 10.0.0.0
exit
copy run start

***Verify that the commands took on R4***

a1#4
[Resuming connection 4 to r4 … ]

R4#sh run | b router rip
router rip
 network 10.0.0.0

R4#sh log
Syslog logging: enabled (9 messages dropped, 1 messages rate-limited, 0 flushes,
…
*Mar  1 15:34:16: %SYS-5-CONFIG_I: Configured from console by console
R4#show clock
*15:34:37.754 UTC Mon Mar 1 1993

You can also use the send command to send commands to a specific device.  In this case, I will change the hostname on r1 (connected on tty 1).

Send Commands to a Specific Device

a1#send tty 1
Enter message, end with CTRL/Z; abort with CTRL/C:
conf t
hostname CHANGED
exit
copy run start
^Z
Send message? [confirm]

***Verify Change on R1***

a1#1
[Resuming connection 1 to r1 … ]

CHANGED#sh run
Building configuration…

Current configuration : 948 bytes
!
version 12.3
service timestamps debug datetime
service timestamps log datetime
no service password-encryption
!
hostname CHANGED
!

CHANGED#sh log
Syslog logging: enabled (9 messages dropped, 1 messages rate-limited, 0 flushes,
…

*Mar  1 15:32:15: %SYS-5-CONFIG_I: Configured from console by console
CHANGED#show clock
*15:32:46.512 UTC Mon Mar 1 1993

Using the send command is a great time-saver.  You must respect it’s power though 🙂  My DOH! moment came when I finished one lab and was about to start another.  Instead of going to each router and stripping off the commands so that the devices would be ready for the next lab, I got the bright idea to wipe all of the configurations at once:

a1#send *
Enter message, end with CTRL/Z; abort with CTRL/C:
write erase
yes
reload
yes
^Z
Send message? [confirm]

I was happily watching each of the routers reload until I realized that my Frame Relay switch was reloading as well.  Luckily I had saved out a backup configuration and was able to rebuild the Frame Relay switch.  Rebuilding the melted Frame Relay switch would have been a pain.

Command Reference

send
To send messages to one or all terminal lines, use the send command in EXEC mode.

send {line-number | * | aux number | console number | tty number | vty number}

How to Schedule a Reload

It has happened to all of us at some time: you’re working on an access-list or an interface configuration when suddenly you’re locked out of the router.  While desperately trying to telnet back to the router it slowly dawns on you: you’ve just made a change to the router that has effectively locked you out of the device.  The good news is that you didn’t write the configuration so you just need to reload/power cycle the device.  The bad news is that the device is at a remote site and it’s 2 am.  The call to the site contact to drive to the site and reboot the router is only going to be slightly less pleasurable than the Monday morning meeting with your supervisor where you get to explain why you made a service-affecting change without proper authorization. 🙂

One way to avoid this situation is to schedule a router reload so that if you get locked out of the box, you simply need to wait for the reload to take place and then get back in.

You can schedule the reload to happen at a specific time of day* (reload at) or at the end of a certain interval (reload in):

Reload In x

reload in ?
Delay before reload (mmm or hhh:mm)

r1#reload in 1
Reload scheduled in 1 minute by console
Proceed with reload? [confirm]
r1#
***
*** — SHUTDOWN in 0:01:00 —
***

Reload at xx:xx

r1#reload at 10:01

System configuration has been modified. Save? [yes/no]: y
Building configuration…
[OK]
Reload scheduled for 10:01:00 UTC Sat Apr 28 2007 (in 37 seconds) by console
Proceed with reload? [confirm]
r1#
Apr 28 10:00:26: %SYS-5-SCHEDULED_RELOAD: Reload requested for 10:01:00 UTC Sat
Apr 28 2007 at 10:00:08 UTC Sat Apr 28 2007 by console.

So you’ve set the router to reload in 30 minutes while you complete your change.  You complete the change successfully.  How do you cancel the ticking time-bomb of the impending router reload?  Simply enter the “reload cancel” command:

Reload Cancel

r1#reload cancel
r1#
***
*** — SHUTDOWN ABORTED —
***

Being able to schedule a reload is a great feature and has saved my bacon more than once.  The biggest thing to remember is to disable the router reload once you have successfully completed your change.  To verify whether there are scheduled reloads, run the “show reload” command:

Show Reload

r1#show reload
Reload scheduled in 4 minutes and 55 seconds by console
Reload reason: Scheduled reload

You can optionally provide a reason for the reload.  This will show up with “show reload” but will not survive the reload itself (I was hoping that it would show up in the “show version” output):

Reason For Reload

r1#reload in 1 ?
  LINE  Reason for reload
  <cr>

r1#reload in 2 ROUTER_WAS_NAUGHTY
Reload scheduled in 2 minutes by console
Reload reason: ROUTER_WAS_NAUGHTY
Proceed with reload? [confirm]
r1#sh
*Mar  1 01:37:34: %SYS-5-SCHEDULED_RELOAD: Reload requested for 01:39:33 UTC Mon Mar 1 1993 at 01:37:33 UTC Mon Mar 1 1993 by console. Reload Reason: ROUTER_WAS_NAUGHTY.
r1#sh reload
Reload scheduled in 1 minute and 55 seconds by console
Reload reason: ROUTER_WAS_NAUGHTY
r1#
***
*** — SHUTDOWN in 0:01:00 —
***

r1#
***
*** — SHUTDOWN NOW —
***

r1#
*Mar  1 01:39:34: %SYS-5-RELOAD: Reload requested by console. Reload Reason: ROUTER_WAS_NAUGHTY.

*****After the reload****

r1#sh ver
Cisco Internetwork Operating System Software
IOS ™ C2600 Software (C2600-IS-M), Version 12.3(22), RELEASE SOFTWARE (fc2)
…

r1 uptime is 2 minutes
System returned to ROM by reload
System image file is “flash:c2600-is-mz.123-22.bin”

This command can also be used for evil.  Let’s say – totally hypothetically – that you are in a training class and your buddy did not secure his router.  You could – again this is a hypothetical and cannot be used as evidence of past misuse – enter “write erase” followed by “reload in 1 CUZ_UR_N00B”.  Your buddy had better know about the “reload cancel” command or he may be doing a lot of reconfiguring 🙂

The Last Thing Your Buddy Might See

***
*** — SHUTDOWN in 0:01:00 —
***

r2#
*Mar  1 05:19:43.554: %SYS-5-SCHEDULED_RELOAD: Reload requested for 05:20:29 UTC
 Mon Mar 1 1993 at 05:19:29 UTC Mon Mar 1 1993 by vty0 (155.1.146.1). Reload Rea
son: CUZ_UR_N00B.
r2#
***
*** — SHUTDOWN NOW —
***

r2#
*Mar  1 05:20:43.066: %SYS-5-RELOAD: Reload requested by vty0 (155.1.146.1). Rel
oad Reason: CUZ_UR_N00B.
r2#
System Bootstrap, Version 12.2(10r)1, RELEASE SOFTWARE (fc1)
TAC Support: http://www.cisco.com/tac
Copyright (c) 2002 by cisco Systems, Inc.
C2600 platform with 65536 Kbytes of main memory

program load complete, entry point: 0x80008000, size: 0x1134810
Self decompressing the image : #################################################

No!!!!!!!!!!

Command Reference

reload
To reload the operating system, use the reload command in privileged EXEC mode.

reload [/verify | /noverify] [warm [file url]] [in [hh:]mm | at hh:mm [month day | day month]] [cancel] [text]

* In order for this command to work, you must be running NTP or have the time set on your router.  To set the time use the “clock set” command from the enable prompt (not in configuration mode):

r1#clock set 10:00:00 April 28 2007
r1#show clock
r1#10:00:03.391 UTC Sat Apr 28 2007

If you set the clock with “clock set”, the clock setting will not survive the reload.

Bradshaw Labs Price Increase

It looks like Bradshaw Labs has increased it’s per session price from $25 to $45.  The following announcement is on their homepage:

New Rack Prices –
To meet the growing costs our rate must return to where it was January 1, 2008.
These rates are still under or the same as are competitors.

1-5 sessions $45 per session
6-10 sessions $40 per session
11+ sessions $35 per session

The “January 1, 2008” date is a bit cryptic.  I initially thought that they were announcing a price increase effective January 1, 2008, but when I tried to book a session, the price showed as $45. 

This price increase will effectively catapult Bradshaw Labs from being one of the lowest cost per hour rack rental options to one of the highest.  It looks like I’ll be using a different vendor going forward.  Don’t get me wrong, I think that Bradshaw Labsis a good vendor (I really love their website and the ability to quickly book sessions online), but with their price increase I can book a rack from Internetwork Experts for less.

I’ll update the CCIE Rack Rental Costs posting with the new price.

Home Lab Pix

Here are some pictures of my home lab.  Since it sits on my desk at work, it’s more accurately a “work lab”.  So far it consists of seven 2600s, one 3800 (acting as the frame relay switch), and a pair of 3550s.  This weekend I’ll add another 2600 as a backbone router and a 2511 as an access/backbone router.  I will eventually add two 3560s as well as configure the 3800 as the third backbone router.  That will give me a lab setup to use with the Internetwork Expert labs.

Simple Tcl Script Ping Test

You can use a simple Tcl script to test connectivity between your devices:

foreach address {
x.x.x.x
x.x.x.x
x.x.x.x} { ping $address
}

For example:

R1#tclsh
R1(tcl)#foreach address {
+>(tcl)#172.12.23.2
+>(tcl)#172.12.23.3
+>(tcl)#172.12.23.4
+>(tcl)#172.12.23.6
+>(tcl)#172.12.23.7
+>(tcl)#} { ping $address re 10 si 1500
+>(tcl)#}

Type escape sequence to abort.
Sending 10, 1500-byte ICMP Echos to 172.12.23.2, timeout is 2 seconds:
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 4/4/4 ms
Type escape sequence to abort.
Sending 10, 1500-byte ICMP Echos to 172.12.23.3, timeout is 2 seconds:
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 4/4/4 ms
Type escape sequence to abort.
Sending 10, 1500-byte ICMP Echos to 172.12.23.4, timeout is 2 seconds:
……….
Success rate is 0 percent (0/10)
Type escape sequence to abort.
Sending 10, 1500-byte ICMP Echos to 172.12.23.6, timeout is 2 seconds:
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 1/1/4 ms
Type escape sequence to abort.
Sending 10, 1500-byte ICMP Echos to 172.12.23.7, timeout is 2 seconds:
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 1/3/4 ms
R1(tcl)#tclquit
R1#

Type “tclsh” to enter the Tcl shell.  You can see that you can add extended commands to the ping command (repeat 10 size 1500 in this case).  I included IP address 172.12.23.4 to show the output when a ping is not successful. 

Be sure to type “tclquit” after the script has run.  Usually you will see the “(tcl)” at the command prompt, but I have seen that disappear even though you are STILL in the Tcl shell.  This has happened to me when I go into config mode from Tcl shell.  After [cntrl + z] I exit out to the command prompt, but the “(tcl)” does not show:

r1#tclsh
r1(tcl)#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
r1(config)#^Z
r1# <- still in Tcl shell!!!

I generally keep a copy of the Tcl script in a text file so that I can just paste it into the router to test connectivity at any time.  The extra time it takes to create/modify the text file is much less than the time it would take to type “ping x.x.x.x” for each device.  You are also less likely to forget a device.

Command Reference

Cisco IOS Scripting with Tcl

TCL’ing Your Cisco Router

Saving the Running-Configuration in Tera Term

Tera Term is my telnet client of choice.  I will soon be making the move to SecureCRT as that is the client used on the CCIE lab.  I’ve used Tera Term for years, so the move may be a painful one.

Here’s a step by step method for saving your configurations in Tera Term:

1) Type “terminal length 0”.  This will stop pausing when you do a “show run” so that you don’t have to keep hitting the spacebar.

2) In Tera Term go to “File” -> “Log…” and then enter a filename for your configuration in the form of “filename.txt”.  Remember the .txt or you’ll end up with a file that you cannot open.  If this happens, simply rename the file so that it ends with .txt

3) Click “Open”.  You’ll be returned to your telnet session, but you should notice a new window on your Taskbar.

4) Type “show run”.  As soon as the configuration has been displayed, click on the “Tera Term: Log” window and then click “Close”.

5) Finally, be sure to reset your terminal length (if you like) “term len 24”

You can skip setting the terminal length to zero, but you’ll need to hit the spacebar to scroll through each screen of the configuration.  Also, your configuration text file will/might end up with garbage characters and other text that the router will choke on if you paste it in configuration mode:

Text Without “Term Len 0”

!
interface Serial0/1
 no ip address
 shutdown
!
 –More–         router bgp 123
 no synchronization
 bgp log-neighbor-changes
 neighbor 172.12.123.2 remote-as 123

You can use the above method to save “show tech” or any commands or output from any telnet session.

CCIE Rack Rental Costs

***Updated 02 October, 2007 *** 

Here is a (non-comprehensive) quick rundown on the prices of various CCIE rack rental companies.  Most companies will offer discounts for multiple sessions.  The prices listed do not represent those discounts, but should give you an idea of the cost of renting a single block of rack time.

You can find links to these rack rental providers in the “CCIE Rack Rental” sidebar.

Vendor	Rental Block (in hours)	Cost	Cost Per Hour
CCIE 2 B	4	$25.00	$3.75
CCIERack	6	$17.50	$2.91
CCIE4U	4	$12.00	$3.00
CCOnline Labs	5.5	$20.00	$3.64
CCOnline Labs	11.5	$35.00	$3.04
GigaVelocity	3	$15.00	$5.00
GigaVelocity	6	$25.00	$4.17
GigaVelocity	12	$50.00	$4.17
Mind Tech	10	$49.00	$4.90
Rack Time Rentals	11.5	$45.00	$3.91
Proctor Labs (IPExpert)	5.75	$40.00	$6.96
Internetwork Expert	11.5	$45.00	$3.91
Internetwork Expert (nights)	11.5	$35.00	$3.04
Internetwork Expert (weekend)	11.5	$50.00	$4.35
CCBootcamp	8	$30.00	$3.75

Read The Fine Print

So I’ve built my CCIE lab at work (more on that later) and the only thing that I’m missing is an octal cable so that I can use one of the routers as an access server.  I went on eBay and found a number of octal cables for sale.  Most of them ran about $30 including shipping (and most were shipping from Hong Kong).  I ended up using “Buy It Now” to nab one for $25 with free shipping.

Unfortunately I did not read the auction description closely at all.  The picture showed an octal cable with eight RJ45 ends, but the auction stated (in multiple locations) that this was an octal cable with DB25 ends.

The cable arrived today.  After checking the auction and confirming that I am a bonehead, I was left with three options:

1) Order a new octal cable
2) Order 8 DB25 to RJ45 adapters
3) Try to convert the DB25 cables to RJ45s

The downsides to each option are:

1) At least $25 more dollars and a week to receive.
2) Cost comes to about $30.  It would be cheaper to buy an RJ45 octal cable.
3) By hacking apart the DB25 cable, I might end up with need to go with option 1 and not being able to resell the DB25 cable.

The most sensible option is to simply order/bid on another octal cable and to try to resell the DB25 cable.  This would only end up costing me a few extra bucks, some time, and a little bit of pride.

Since option 2 is not financially prudent and would not net me any extra time, which just leaves me pondering option 3.  After last weekend’s cable building party (more on that later) I am pretty confident in my abilities to convert these cables.  I just need to find out if this is possible and if there is a pin out available.  I am currently searching the Internet for any descriptions of how to convert the DB25 connectors to RJ45s.

ping x.x.x.x repeat x

Ah, the mighty ping.  I use this command every damned day, so you would think that I would know this command very well.  If you did so, you would be completely underestimating how incredibly lame I am.

I know how to send a simple, 5 packet ping a la “ping 10.1.1.1”.  I also know how to send an “extended ping” by simply typing  “ping” + ‘enter’ and filling in the information at the prompts.  The most frustrating thing about the extended ping is that a lot of the time I type the IP address at the “Protocol [ip]” prompt instead of hitting ‘enter’:

Simple Ping:
router#ping 10.1.1.1 Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/7/8 ms
Extended Ping:
router#ping
Protocol [ip]:
Target IP address: 10.1.1.1
Repeat count [5]: 100
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 4/7/8 ms
What I Usually Do:
router#ping
Protocol [ip]: 10.1.1.1
% Unknown protocol - "10.1.1.1", type "ping ?" for help

FUCK!!!!

A couple of months ago I was troubleshooting with a collegue over my shoulder.  I did an extended ping and of course put the IP address in the “Protocol [ip]” prompt.  Argh!!!!  My co-worker then showed me a cool variation of the ping command.  Simply type “ping ip” and then ‘enter’.  This will start the extended ping prompt, but safely bypass the stupid “Protocol [ip]” prompt.

Danger Averted:
router#ping ip
Target IP address: 10.1.1.1
Repeat count [5]: 100
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 4/7/8 ms


Up until this week, that was the coolest ping tip I had ever received.

95% of the time I use an extended ping simply to send more than the standard 5 packets.  “ping ip” + ‘enter’ + IP Address + Number of pings + ‘enter’ + ‘enter’+ ‘enter’ + ‘enter’.  I am no longer a slave to the extended ping prompts now that I have tapped the power of the “ping x.x.x.x repeat x” command.

Sweet:
router#ping 10.1.1.1 repeat 100 Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 4/7/8 ms
If I want to get fancy, I can throw in the datagram size by simply tacking on "size x":

Changing the Datagram Size and Conserving Keystrokes:
router#ping 10.1.1.1 re 100 si 1500
Type escape sequence to abort.
Sending 100, 1500-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 8/8/12 ms

Now I only need to use the extended ping prompts in those rare occasions that I need to use extended commands or sweeps.

Command Reference:
ping ip
To test network connectivity on IP networks, use the ping ip command in privileged EXEC mode.

ping ip {host-name | ip-address} [data [hex-data-pattern] | df-bit | repeat [repeat-count] | size [datagram-size] [timeout seconds] [validate] [verbose]

Bonjour tout le monde!

This post will be a brief introduction as I really hate to write about myself.  I will “fill in the blanks” in future posts. 

As stated in the “About” page, I am a Twin Cities-based network engineer.  I have been “doing the Cisco thing” for just over seven years now.  I’ve been working in IT in one fashion or another for nearly 10 years.  I started out doing graphics and web design and then moved to a corporate Help Desk position.  From there I became certified as a Microsoft MCSE(NT4 – old skool).  I (very) briefly worked in Microsoft networking as well as desktop builds/support.  In March of 2000 I took a job working in the NOC of a very large (actually, the largest in the world) privately-held company.  At the time that I took that position, the sum of my network knowledge (outside of the Microsoft realm) consisted of knowing how to ping from a DOS prompt and also that a T1 contained 24 64K channels (although I don’t think that I fully understood what that meant).  I was definitely not qualified for that position.  I learned later that I was hired based on my “potential”.  Oh, and also, the other candidates were either pretty horrible or wanted way too much money.  This was during the Golden Age of IT when people were getting hired just because they could spell “computer”.

I took the NOC position because I wanted to get a foot in the door and quickly move on to Microsoft server support.  Unfortunately these plans were scuttled when the bottom dropped out of the dot com industry and the IT industry went into a long, dark freeze.  I was stuck doing Cisco.

I spent 5 years working in that position.  While we did get to work on some interesting equipment (Cisco IGX switches, DACS systems, Adran CSU/DSUs, and RNET/Motorola/Whomever-The-Fuck-Owns-Them-Now Vanguard routers) our main duty was supporting a very large, international Frame Relay network.  A lot of this consisted of seeing an alert and turning the circuit into the appropriate carrier to test.  During this time I concentrated on studying database design and .NET programming.  I wanted to be a DBA and/or a web programmer. 

Finally the “IT Freeze” started to thaw (at least internally) in 2005.  I threw out my DBA/.NET plans and took a position (and huge pay/status cut) to join the Router Team.  I effectively decided to stay in networking for the rest of my life.

In the NOC position we were not granted “enable mode” (privileged exec).  As a result, I could trouble-shoot Frame Relay inside and out with “show” commands, but I was way over my head when it came to configuring routers and troubleshooting routing issues.  Also, I had never touched a layer 2 switch in my life.

A few months after I joined the team, the team was split in half during reorganization.  I was placed in the LAN team.  This was actually a boon to me as it effectively cut in half the amount of technology that I had to learn.  We had a mixture of Nortel and Cisco switches (as well as some rogue hubs) in the field.  This made troubleshooting LAN issues “interesting”.  I quickly appreciated the power of CDP.

I eventually became the “wireless guru” as I inherited the WLAN when one of our consultants left.  That became my major passion.  I loved working with wireless.  We rolled out hundreds of access points and supported them with WLSE.

Although I was fully “engaged” (corporate babble) in my work, I was feeling the pinch in my pocketbook.  I took a large pay cut when I moved into that position.  I was quickly promoted, but my pay increase was half of what I had been promised.  I soon left the company and began working in my current position (at a higher salary).

As I have gone on way too long with my (supposedly brief) back story, I will wrap it up here.  My current position is working on a very large CISCO ONLY (hallelujah!) network.  I had to quickly learn BGP, ATM IMA, and other technologies when I moved over.  I decided at that point to start pursuing Cisco certification starting with renewing my CCNA and then working on the CCNP and CCIP certifications.  I have completed the CCNP and the first test of the three remaining CCIP exams (QoS).  I recently decided that I would make a run at the CCIE.