Not On My Game Today

Some days I just don’t have “it”.  Today is one of them.  I’ve been making typos galore, adding subnet masks to Frame map statements, redistributing BGP into EIGRP into BGP, etc.  I’ve done 6 hours of labs and completed about half of what I would normally do.  I’m taking the hint and calling it a day.  Hopefully I don’t have a day like this during the actual lab. 

Mnemonic For BGP Attributes

I found this mnemonic at “maloy & jing apuhin’s 101 & others” blog to help remember the order of BGP attributes:

“We Love Oranges AS Oranges Mean Pure Refreshment”

W   Weight (Highest)
L   Local_Pref (Highest)
O   Originate (local originate)
AS  As_Path (shortest)
O   Origin Code (IGP < EGP < Incomplete)
M   MED (lowest)
P   Paths (External Paths preferred Over Internal)
R   Router ID (lowest)

State By State Internet Speeds

Interesting article and report concerning Internet speeds in the United States.  Also, a cool map with which you can quickly check out your state’s speed:

First-Ever State-By-State Report on Internet Connection Speed Shows U.S. Far Behind Other Industrialized Nations

Report (PDF)

Interactive Map

Article highlights:

Results released today of the first-ever state-by-state report on Internet connection speed reveal that the United States is falling far behind other industrialized nations. The report, based on aggregated data from nearly 80,000 users, shows that the median real-time download speed in the U.S. is a mere 1.9 megabits per second (mbps). The best available estimates show average download speeds in Japan of 61 mbps, in South Korea of 45 mbps, in France of 17 mbps and in Canada of 7 mbps.

“The United States is the only industrialized nation without a national policy to promote universal, high-speed Internet access,” said Larry Cohen, president, Communications Workers of America. “The grim results of the CWA Speed Test illustrate that, without a national policy, we risk losing our competitive edge in today’s global economy—and the jobs that go with it.”

The report also ranks individual states based on median Internet connection speeds. The speediest states?  Rhode Island (5.011 mbps), Kansas (4.167 mbps), New Jersey (3.68 mbps), New York (3.436 mbps) and Massachusetts (3.004 mbps).

Iowa (1.262 mbps), Wyoming (1.246 mbps), West Virginia (1.117 mbps), South Dakota (0.825 mbps) and Alaska (0.545) make up the bottom five. The same 10 megabyte (MB) file that takes 15 seconds to download in Rhode Island would take nearly two and a half minutes to download in Alaska.

The only surprise in the list is Kansas at number two fastest.  Kansas?  Really?  My state (Minnesota) was slightly below the national average.

Rent Your Cisco Certification For Cash

Brad Reese recently posted about an interesting (and disturbing) new business: renting your Cisco certification to companies wanting to qualify as Cisco Gold or Silver partners in exchange for a monthly fee.

Rent A Cert is a new site that will allow you to “rent” your current Cisco and Microsoft certifications to companies that will assocaite your certification with their company in order to get better discounts on Cisco equipment.  I encourage you to read Brad’s posting.  Here are some of the highlights:

Certification	Monthly Income to Cert Holder	Monthly Payment from Cisco Partner	Monthly Profit to Rent A Cert
CCNA	$100	$139	$39
CCNP	$250	$299	$49
CCDP	$250	$299	$49
CCVP	$250	$299	$49
CCSP	$250	$299	$49
CCIE	$1,000	$1,149	$149

That means that once I get my CCIE (I work in an enterprise that has no need of Gold or Silver status) I can rent out my CCNA, CCNP, and CCIE for a grand total of $1,350 per month.  An extra $16,000 a year is nothing to sneeze at.  🙂  Well…it looks like you only get paid for one certification per vendor [You will only be paid for one certification per vendor. For example, if you selected both “MCSE” and “MCDBA,” you will only be paid $200 monthly upon matching. This is because you must associate your entire profile with a company, rather than individual certifications.] , still “free money” for renting your certification sounds tempting.

So, is Cisco down with this?  This is from Rent A Cert’s FAQ:

Do Microsoft & Cisco allow this?
We’ve found nothing so far in Microsoft’s and Cisco’s agreements that prohibit this, but of course review your certification agreements, as Rent A Cert doesn’t have a copy of your particular agreements and isn’t responsible if you break them. We, of course, will not divulge our client list to either company.

As a side note, if people know they can get paid to get certified, they will be more likely to get certified. As Microsoft and Cisco make money every time someone gets certified, as well as have a new person that promotes and knows how to configure their products, we see no reason why this isn’t a win-win scenario.

They’ve found no issues so far.  How nice.  Why the fuck not ASK Cisco and Microsoft if this is on the up and up, then post their replies?  Also, (as Brad points out in his post) the losers in this “win-win scenario” are certified professionals who will see their job opportunities, salary, and certification values plummet if this takes off.

Brad does a good job of disecting the downsides to this “business model”.  I agree with him 100%.  I would also like to add that this looks like a shady venture.  I am willing to bet that Rent A Cert doesn’t have a lot of companies lined up at that moment but will get inundated with thousands of CCNA and CCNx certification holders.  There are also two points that Brad did not touch on that I want to point out:

1)  Twelve Inches Around Corp. is the mother company of Rent A Cert.  You can check out their main site here.  What experience does a modeling/entertainment/catering company have with IT?

2)  Here’s what Rent A Cert requests from you:

If you checked any Cisco certifications, enter your Cisco ID, then from the Certification Traking System, e-mail yourself your transcript and forward that e-mail to transcripts at rentacert.com. Additionally, if you selected CCIE, note which track and your CCIE number. 

Does anyone else think that it’s a bad idea to give these guys this information?  They could “rent” your certification without your knowledge.  Why don’t they contact you once they find a company and then you can decide whether or not to give them your information?

Anyhoo…this seems shady at the very least and possibly illegal.

Cisco Goes To The Dark Side?

Brad Reese has a recent blog entry about Cisco’s involvement in Net Neutrality.  You can read it here.

Basically the posting takes Cisco and – specifically – Jeff Campbell to task.  Jeff Campbell posted the following after a recent FTC report on Net Neutrality [emphasis mine]:

In other words, there is no reason to rush to impose burdensome Net Neutrality regulations in the broadband market. If there is one thing that we have learned from 70+ years of communications regulation, it is that regulation has significant costs and unintended consequences. The FTC clearly recognizes that government should react to actual problems, not hypothetical ones.

Campbell’s comments are of the “the market will handle the situation…this is just hysteria…blah…blah…blah” ilk.  Brad Reese shows the folly of the “invisible hand of the market” line by linking to this post which explains what is obvious to most Americans: most markets in the USA only have very few (usually only one – sometime NONE) broadband providers.  Without competition, the market is unlikely to “correct” issues like this.  If you’re the only man on the planet, you’re probably not going to woo the ladies with roses and poetry.  🙂

As far as Cisco’s position, I feel that it’s probably more business related than ideological.  Cisco sells equipment with the ability to sort and limit traffic, so they would naturally wish for a market where that capability was desired/needed.  It doesn’t make their position correct, but it does make it logical in a possibly greedy, evil way.  🙂

I am personally in favor of Net Neutrality laws, but I do have a problem with some of the examples that Brad Reese links to in his posting in order to show the evidence that the providers are abusing their power.  For instance, he links to a story about Comcast nixing some users because they used too much bandwidth.  On the face of it that seems like typical corporate fuckery, but if you read the article (found here) you’ll see that these were extreme cases and that the provider gave the users warnings as well as asking them to upgrade to a business account.  Here are some quotes from the article [emphasis mine]:

User who was cut off by Comcast:
Admitted “Internet junkie” and Chattanooga resident Cameron Smith also had his service cut off in January for one year. “They said there wasn’t a limit [for downloading] but that I was downloading too much, about 550 gigs. I backed off to about 450 gigs, but they still suspended us.”

Comcast’s response:
“The customers who are notified of excessive use typically and repeatedly consume exponentially more bandwidth than an average residential user, which would include, for example, the equivalent of sending 256,000 photos a month, or sending 13 million e-mails every month (or 18,000 emails every hour, every day, all month). In these rare instances, Comcast’s policy is to proactively contact the customer via phone to work with them and address the issue or help them select a more appropriate commercial-grade Comcast product.”

Anyhoo…I don’t want to get all “political” in a CCIE blog.  I do think that one of the issues that plagues broadband (specifically cable) is that the last mile is essentially a shared segement.  If you have some 450-550 gig a month ass clown on your segment, it will affect your access.  I would like to see cable broadband providers implement a system similar to Frame Relay CIR in which you are guaranteed an amount of bandwidth (and it’s fucking marketed at the CIR rate and not the “possible upper limit” port speed) and if there’s room on the pipe, then you can burst over that limit.  My interest in Net Neutrality is not bandwidth related as much as it’s related to the possibility of service providers filtering my packets based on destination, source, content, etc.

Dynamips Blew Up On Me

It was bound to happen.  I’ve been doing a lot of labs with Dynamips using 4 routers as 1 switch.  Today I started doing some BGP labs that required 5 routers.  Dynamips kept dumping its core (ewww!).  I tried to isolate the issue to a single router instance, but couldn’t do it.  I finally realized that as soon as the IOS loaded on the 6th device (7th if you count the Frame Relay switch) Dynamips would crash.  This is probably due to my laptop.  I only have 512Meg of RAM.  I had planned to drop 2 Gigs in it,  but Dynamips had been chugging along fine up until today.

I dropped the RAM on the Dynamips router instances from 128 to 96.  I was able to get all 6 devices up and working.  Success!!!  Well….until I tried to configure BGP that is:

r1(config)#router bgp 6
r1(config-router)#
*Mar  1 00:42:45.707: %SYS-2-MALLOCFAIL: Memory allocation of 18360 bytes failed from 0x609B8D6C, alignment 0
Pool: Processor  Free: 6300  Cause: Not enough free memory
Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= “Exec”, ipl= 0, pid= 3
-Traceback= 0x603F5B88 0x6053CC4C 0x60543434 0x609B8D74 0x60A08938 0x609B9DE0 0x
60798530 0x607999B8 0x60432DB0 0x60450578 0x604F164C 0x604F1630

I finally stripped the switch out of the equation (I just directly connected the couple of routers on Ethernet segments) and bumped the RAM back to 128.  I started each device one by one and then telnetted to it to make sure that it didn’t crash once the IOS was loaded.  I was finally able to get all 5 routers rockin and rollin with BGP and OSPF redistribution.

I put in an order for 2 gigs of RAM shortly after.  🙂

VTP: Local updater ID on VTP Transparent Switches

Okay, this is the last post of the VTP Local updater ID trilogy.  I promise.  🙂

Here are the first two posts:

VTP: Which VTP Server Generated The Most Recent Update?

VTP: More On The Local updater ID

This last post concerns the behavior of the Local updater ID on VTP transparent switches.  At first blush, it looks like this feature should not appear on a transparent switch.  If you have a transparent switch, then the updates will always be local, but you can still use the Local updater ID feature:

sw3(config)#do sh vtp status
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Transparent
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x13 0x23 0x62 0x16 0x83 0xCD 0x50 0xEC
Configuration last modified by 0.0.0.0 at 9-25-07 09:41:32

Let’s add an IP address and a VLAN to the switch:
sw3(config)#int lo0
sw3(config-if)#ip add 10.0.0.3 255.255.255.255
sw3(config-if)#vlan 666
sw3(config-vlan)#name VLAN_OF_THE_BEAST
sw3(config-vlan)#exit

sw3(config)#do sh vtp statu
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 6
VTP Operating Mode              : Transparent
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x04 0x4E 0xE9 0xBF 0x79 0x81 0x44 0x8F
Configuration last modified by 10.0.0.3 at 9-25-07 09:41:32

sw3(config)#do sh vlan id 666

VLAN Name                             Status    Ports
—- ——————————– ——— ——————————-
666  VLAN_OF_THE_BEAST                active    Fa0/19, Fa0/20, Fa0/21

Notice that you do not see the “Local updater ID is x.x.x.x” line in the “show vtp status” output when the switch is in transparent mode.  BUT you do see that the switch will record the IP address of the switch that updated the VLAN database (which will always be the local switch when the switch is in VTP transparent mode).  Notice that it follows the same rules (first IP address, lowest vlan ip address, vtp interface, etc) as the client/server switches:

sw3(config-if)#int vlan 666
sw3(config-if)#ip add 6.6.6.3 255.255.255.0
sw3(config-if)#do sh ip int br | e ass
Interface              IP-Address      OK? Method Status                Protocol
Vlan666                6.6.6.3         YES manual up                    up
Loopback0              10.0.0.3        YES manual up                    up
sw3(config-if)#do sh vtp statu
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 6
VTP Operating Mode              : Transparent
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x04 0x4E 0xE9 0xBF 0x79 0x81 0x44 0x8F
Configuration last modified by 10.0.0.3 at 9-25-07 09:41:32 <-last update used lo0 IP address

Add another VLAN:
sw3(config-if)#vlan 665
sw3(config-vlan)#name VLAN_OF_THE_NEIGHBOR_OF_THE_BEAST
sw3(config-vlan)#exit
sw3(config)#do sh vtp statu
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 7
VTP Operating Mode              : Transparent
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x9A 0x6E 0x3A 0xCB 0x5F 0x43 0xC9 0xF7
Configuration last modified by 6.6.6.3 at 9-25-07 09:41:32 <-now using SWI vlan 665 address

We can also manually set the Local updater ID:
sw3(config)#vtp interface lo0 only
sw3(config)#no vlan 666
sw3(config)#do sh vtp status
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 6
VTP Operating Mode              : Transparent
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x04 0x4E 0xE9 0xBF 0x79 0x81 0x44 0x8F
Configuration last modified by 10.0.0.3 at 9-25-07 09:41:32 <- lo0 IP address (mandatory)

So even though it’s pretty unneeded, you CAN use the Local updater ID with VTP transparent switches.  This is probably due to either some overlooked code in IOS or to be able to maintain Local update ID settings when switching between VTP modes.

VTP: More On The Local updater ID

This post is a follow up to VTP: Which VTP Server Generated The Most Recent Update?  I didn’t want to make that post any longer than it already was.  Here’s some more information about the “Local update ID” in VTP.

If you have multiple IP addresses on your VTP server, the Local updater will use the first IP address found:

sw1(config-if)#do sh ip int br | e ass
Interface              IP-Address      OK? Method Status                Protocol
Loopback0              10.0.0.1        YES manual up                    up
Loopback1              100.100.100.100 YES manual up                    up
Loopback2              220.0.0.100     YES manual up                    up

sw1(config-if)#do sh vtp statu | i Local updater
Local updater ID is 10.0.0.1 on interface Lo0 (first layer3 interface found)

Just for fun, let’s get rid of lo0 and see what IP address it will choose (either lo1 or lo2):

sw1(config-if)#no int lo0
01:47:42: %LINK-5-CHANGED: Interface Loopback0, changed state to administratively down
01:47:43: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to down
sw1(config)#do sh vtp statu | i Local
Local updater ID is 100.100.100.100 on interface Lo1 (first layer3 interface found)

The Local updater ID will choose the lowest VLAN interface IP address over all IP addresses others:

sw1(config-if)#do sh ip int br | e ass
Interface              IP-Address      OK? Method Status                Protocol
Vlan6                       6.0.0.1         YES manual up                    up
Vlan69                     69.0.0.1        YES manual up                    up
Loopback0              10.0.0.1        YES manual up                    up
Loopback1              100.100.100.100 YES manual up                    up
Loopback2              220.0.0.100     YES manual up                    up

sw1(config-if)#do sh vtp statu | i Local
Local updater ID is 6.0.0.1 on interface Vl6 (lowest numbered VLAN interface found)

If you have multiple IP addresses, you can manually set the Local updater ID:

sw1(config)#do sh ip int br | e ass
Interface              IP-Address      OK? Method Status                Protocol
Vlan6                       6.0.0.1           YES manual up                    up
Vlan69                     69.0.0.1        YES manual up                    up
Loopback0              10.0.0.1        YES manual up                    up
Loopback1              100.100.100.100 YES manual up                    up
Loopback2              220.0.0.100     YES manual up                    up

sw1(config)#vtp ?

  interface  Configure interface as the preferred source for the VTP IP updater address.

sw1(config)#vtp interface ?
  WORD  The name of the interface providing the VTP updater ID for this device. <-word??  really?

sw1(config)#vtp interface lo1 ?
  only  Use only this interface’s IP address as the VTP IP updater address.
  <cr>

sw1(config)#vtp interface lo1
sw1(config)#do sh vtp stat | b Local
Local updater ID is 100.100.100.100 on interface Lo1 (preferred interface)
Preferred interface name is lo1

With “only” keyword:

sw1(config)#vtp interface lo2 only
sw1(config)#do sh vtp stat | b Local
Local updater ID is 220.0.0.100 on interface Lo2 (preferred interface)
Preferred interface name is lo2 (mandatory)

VTP: Which VTP Server Generated The Most Recent Update?

Recently JB left the following comment:

Hi,

I have a VTP question, unrelated to vtp passwords.
I have multiple switches connected by trunks, most on the same vtp domain. Two of the switches are VTP Servers – an Agg pair, and the others are either Transparent or clients.
What command can I use at a Transparent or Client switch, to identify the VTP Server that is managing the VLANs, that sent the last update. Thanks much, hope you can help.

Regards,

My initial answer was to tell him that there was not way of finding out that information without comparing the VTP status on the client switch (transparent switches don’t use VTP for updates) to the same output on the VTP server switch.  Before I responded, I wanted to check out the functionality of the “Local updater ID” in VTP.  I’m glad that I did.

For those of you who don’t want to read this entire post (and I don’t blame you) here’s the quick and easy answer: By configuring an IP address on your VTP server switches you’ll be able to use the “Local updater ID” (on VTP client and server switches) to see which VTP server last updated the VLAN database via VTP.

I think that I’ve mentioned before that I’ve never used VTP in a production environment before.  I’ve used VTP domain names to identify sites for CiscoWorks, but all of our switches are set to VTP transparent mode.  The only time that I’ve used VTP server/client is in the lab.  I tried to find more information on the “Local updater ID” but came back pretty empty.  I decided to lab up a scenario to answer JB’s question:

sw1———-sw2———-sw3———-sw4
server      client     transparent     server
CCIE        CCIE       CCIE            CCIE

All of the switches are in the VTP domain CCIE.  sw1 and sw4 are servers, while sw2 is a client and sw3 is tranparent.

Note: Before labbing this up, make sure that your devices have their clocks synchronized (“clock set” command).

Let’s start by adding a vlan to sw1.  This will propagate to sw2 (client) and sw4(server).  Let’s see if we can tell by looking at sw2 and sw4 where the update came from.

sw1(server):
sw1#sh vtp status
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Server
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x13 0x23 0x62 0x16 0x83 0xCD 0x50 0xEC
Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00
Local updater ID is 0.0.0.0 (no valid interface found)

add vlan on sw1:
sw1(config)#vlan 69
sw1(config-vlan)#name TEST_069
sw1(config-vlan)#exit
sw1(config)#do sh vtp status
VTP Version                     : 2
Configuration Revision          : 1 
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 6  
VTP Operating Mode              : Server
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x38 0x5C 0x9D 0x0F 0x3E 0x6C 0x1F 0x84
Configuration last modified by 0.0.0.0 at 9-25-07 08:59:56
Local updater ID is 0.0.0.0 (no valid interface found)

sw2(client)
sw2#sh vtp statu
VTP Version                     : 2
Configuration Revision          : 1
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 6  
VTP Operating Mode              : Client
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x38 0x5C 0x9D 0x0F 0x3E 0x6C 0x1F 0x84
Configuration last modified by 0.0.0.0 at 9-25-07 08:59:56

sw2#sh vlan id 69

VLAN Name                             Status    Ports
—- ——————————– ——— ——————————-
69   TEST_069                         active    Fa0/13, Fa0/18

sw3 (transparent)
sw3#sh vtp stat
VTP Version                     : 2
Configuration Revision          : 0 
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5  
VTP Operating Mode              : Transparent
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x57 0xCD 0x40 0x65 0x63 0x59 0x47 0xBD
Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00

sw3#sh vlan id 69
VLAN id 69 not found in current VLAN database

sw4(server):
sw4#sh vtp statu
VTP Version                     : 2
Configuration Revision          : 1  
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 6
VTP Operating Mode              : Server
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x38 0x5C 0x9D 0x0F 0x3E 0x6C 0x1F 0x84
Configuration last modified by 0.0.0.0 at 9-25-07 08:59:56 
Local updater ID is 0.0.0.0 (no valid interface found)

sw4#sh vlan id 69

VLAN Name                             Status    Ports
—- ——————————– ——— ——————————-
69   TEST_069                         active    Fa0/19

So, we do see when the updates come in, but we don’t see a layer 2 address associated with it.  In a situation where there are multiple VTP servers in the VTP domain, this doesn’t help us to identify which VTP server iniated the most recent update.  Let’s first remove our test vlan on sw4 (server) and see how that affects the VTP domain:

sw4 (server):
sw4(config)#no vlan 69
sw4(config)#^Z
sw4#sh v
01:22:12: %SYS-5-CONFIG_I: Configured from console by console
sw4#sh vtp status
VTP Version                     : 2
Configuration Revision          : 2
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Server
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0xA4 0x5F 0x11 0x86 0x4D 0x57 0x50 0xC7
Configuration last modified by 0.0.0.0 at 9-25-07 09:06:13
Local updater ID is 0.0.0.0 (no valid interface found)

sw3(transparent):
sw3#sh vtp status
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Transparent
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x57 0xCD 0x40 0x65 0x63 0x59 0x47 0xBD
Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00

sw2(client):
sw2#sh vtp status
VTP Version                     : 2
Configuration Revision          : 2
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5  
VTP Operating Mode              : Client
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0xA4 0x5F 0x11 0x86 0x4D 0x57 0x50 0xC7
Configuration last modified by 0.0.0.0 at 9-25-07 09:06:13

sw2#sh vlan id 69
VLAN id 69 not found in current VLAN databasesw1(server):
sw1#sh vtp statu
VTP Version                     : 2
Configuration Revision          : 2
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Server
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0xA4 0x5F 0x11 0x86 0x4D 0x57 0x50 0xC7
Configuration last modified by 0.0.0.0 at 9-25-07 09:06:13
Local updater ID is 0.0.0.0 (no valid interface found)sw1#sh vlan id 69
VLAN id 69 not found in current VLAN database

Okay…so the timestamp is nice, but in the case where you have multiple servers in your VTP domain, it is useless for finding out which server iniated a change.  In our case we could see that the client’s (sw2) VLAN database was changed at 9:06:13, but we can’t tell if it was sw1 or sw4 that iniated the change.

Let’s add some layer 3 addresses to the mix.  First let’s add an l3 address to sw1 but not to sw4 and repeat the same experiment as above:

sw1(server):
sw1(config)#int lo0
sw1(config-if)#ip add 10.0.0.1 255.255.255.255
sw1(config-if)#do sh vtp statu
VTP Version                     : 2
Configuration Revision          : 2
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Server
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0xA4 0x5F 0x11 0x86 0x4D 0x57 0x50 0xC7
Configuration last modified by 0.0.0.0 at 9-25-07 09:06:13
Local updater ID is 10.0.0.1 on interface Lo0 (first layer3 interface found)

Let’s add a VLAN on sw1:

sw1(config-if)#vlan 69
sw1(config-vlan)#name TEST_069
sw1(config-vlan)#end
sw1#
01:29:49: %SYS-5-CONFIG_I: Configured from console by console
sw1#sh vtp status
VTP Version                     : 2
Configuration Revision          : 3
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 6
VTP Operating Mode              : Server
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0xDF 0x82 0xE1 0x8F 0x9E 0xE4 0x74 0x24
Configuration last modified by 10.0.0.1 at 9-25-07 09:14:33  <-note time and IP address
Local updater ID is 10.0.0.1 on interface Lo0 (first layer3 interface found)

sw2(client):
sw2#sh vtp statu
VTP Version                     : 2
Configuration Revision          : 3
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 6
VTP Operating Mode              : Client
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0xDF 0x82 0xE1 0x8F 0x9E 0xE4 0x74 0x24
Configuration last modified by 10.0.0.1 at 9-25-07 09:14:33  <-booyah!!!
sw2#sh vlan id 69

VLAN Name                             Status    Ports
—- ——————————– ——— ——————————-
69   TEST_069                         active    Fa0/13, Fa0/18

sw3(transparent):
sw3#sh vtp statu
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Transparent
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x57 0xCD 0x40 0x65 0x63 0x59 0x47 0xBD
Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00

sw4(server):
sw4#sh vtp statu
VTP Version                     : 2
Configuration Revision          : 3
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 6
VTP Operating Mode              : Server
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0xDF 0x82 0xE1 0x8F 0x9E 0xE4 0x74 0x24
Configuration last modified by 10.0.0.1 at 9-25-07 09:14:33
Local updater ID is 0.0.0.0 (no valid interface found)
sw4#sh vlan id 69

VLAN Name                             Status    Ports
—- ——————————– ——— ——————————-
69   TEST_069                         active    Fa0/19

SWEET!  We CAN find out which VTP server made the last update to a VTP client switch by just looking at the “show vtp status” output PROVIDED WE HAVE AN IP ADDRESS CONFIGURED ON THE VTP SERVER SWITCH. 

Let’s remove the vlan on sw4 and then see what happens (sw4 does not have a layer 3 address configured).  In the interest of keeping this post under 100,000 words I’m not going to include sw3 (transparent):

sw4 (server):
sw4(config)#no vlan 69
sw4(config)#do sh vtp statu
VTP Version                     : 2
Configuration Revision          : 4
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Server
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x36 0xD3 0xE7 0x16 0xB1 0xF7 0x76 0x54
Configuration last modified by 0.0.0.0 at 9-25-07 09:16:54
Local updater ID is 0.0.0.0 (no valid interface found)

sw2(client):
sw2#sh vtp statu
VTP Version                     : 2
Configuration Revision          : 4
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Client
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x36 0xD3 0xE7 0x16 0xB1 0xF7 0x76 0x54
Configuration last modified by 0.0.0.0 at 9-25-07 09:16:54
sw2#sh vlan id 69
VLAN id 69 not found in current VLAN database

sw1(server):
sw1#sh vtp status
VTP Version                     : 2
Configuration Revision          : 4
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Server
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x36 0xD3 0xE7 0x16 0xB1 0xF7 0x76 0x54
Configuration last modified by 0.0.0.0 at 9-25-07 09:16:54
Local updater ID is 10.0.0.1 on interface Lo0 (first layer3 interface found)
sw1#sh vlan id 69
VLAN id 69 not found in current VLAN database

If you have multiple VTP servers in your VTP domain, you’ll want to make sure that each of the VTP servers has an IP address configured.  We pretty much know how this will turn out, but for the sake of completeness, let’s configure an l3 address on sw4 and then add a vlan to that switch (server).

sw4(server):
sw4(config)#int lo0
01:36:25: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback0, changed state to up
sw4(config-if)#ip add 10.0.0.4 255.255.255.255

sw4(config-if)#do sh vtp statu
VTP Version                     : 2
Configuration Revision          : 4
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Server
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x36 0xD3 0xE7 0x16 0xB1 0xF7 0x76 0x54
Configuration last modified by 0.0.0.0 at 9-25-07 09:16:54
Local updater ID is 10.0.0.4 on interface Lo0 (first layer3 interface found)
*********
WARNING:

Make sure that you “exit” the vlan configuration or else your vlan will NOT be created:

sw4(config-if)#vlan 69
sw4(config-vlan)#name LAST_TEST
sw4(config-vlan)#do sh vtp statu  <- I have not exited the “config-vlan” mode so vlan 69 is NOT created yet
VTP Version                     : 2
Configuration Revision          : 4
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5 <-note
VTP Operating Mode              : Server
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x36 0xD3 0xE7 0x16 0xB1 0xF7 0x76 0x54
Configuration last modified by 0.0.0.0 at 9-25-07 09:16:54 <-old update
Local updater ID is 10.0.0.4 on interface Lo0 (first layer3 interface found)

********
sw4(config-if)#vlan 69
sw4(config-vlan)#name LAST_TEST
sw4(config-vlan)#exit
sw4(config)#do sh vtp statu
VTP Version                     : 2
Configuration Revision          : 5
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 6
VTP Operating Mode              : Server
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0xF4 0x4E 0xDA 0xAA 0x12 0xC1 0x77 0xB1
Configuration last modified by 10.0.0.4 at 9-25-07 09:23:34
Local updater ID is 10.0.0.4 on interface Lo0 (first layer3 interface found)

sw2(client):
sw2#sh vtp statu
VTP Version                     : 2
Configuration Revision          : 5
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 6
VTP Operating Mode              : Client
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0xF4 0x4E 0xDA 0xAA 0x12 0xC1 0x77 0xB1
Configuration last modified by 10.0.0.4 at 9-25-07 09:23:34
sw2#sh vlan id 69

VLAN Name                             Status    Ports
—- ——————————– ——— ——————————-
69   LAST_TEST                        active    Fa0/13, Fa0/18

sw1#sh vtp statu
VTP Version                     : 2
Configuration Revision          : 5
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 6
VTP Operating Mode              : Server
VTP Domain Name                 : CCIE
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0xF4 0x4E 0xDA 0xAA 0x12 0xC1 0x77 0xB1
Configuration last modified by 10.0.0.4 at 9-25-07 09:23:34
Local updater ID is 10.0.0.1 on interface Lo0 (first layer3 interface found)
sw1#sh vlan id 69

VLAN Name                             Status    Ports
—- ——————————– ——— ——————————-
69   LAST_TEST                        active    Fa0/13

By configuring an IP address on your VTP server switches you’ll be able to use the “Local updater ID” when troubleshooting VTP updates.

LFU 4 – Fat Fingers Can Doom You

I was doing a NAT lab today and came to a dead stop because I couldn’t get BGP to work between two routers.  R4 and R5 share two links: a PTP serial link (155.1.45.0/24) and a PTP Frame Relay link (155.1.0.0/24).  I was running OSPF as an IGP and everything was fine until I found that BGP was not working:

r4#sh ip bgp sum
BGP router identifier 150.1.4.4, local AS number 1
BGP table version is 1, main routing table version 1

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
150.1.5.5       4     2       0       0        0    0    0 never    Active

r5#sh ip bgp sum
BGP router identifier 150.1.5.5, local AS number 2
BGP table version is 1, main routing table version 1

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
150.1.4.4       4     1       0       0        0    0    0 never    Active

I went over the BGP config on both routers and couldn’t find any issues:

r4#sh run | sec bgp
router bgp 1
 no synchronization
 bgp router-id 150.1.4.4
 bgp log-neighbor-changes
 neighbor 150.1.5.5 remote-as 2
 neighbor 150.1.5.5 ebgp-multihop 255
 neighbor 150.1.5.5 update-source Loopback0
 no auto-summary

r5#sh run | sec bgp
router bgp 2
 no synchronization
 bgp router-id 150.1.5.5
 bgp log-neighbor-changes
 neighbor 150.1.4.4 remote-as 1
 neighbor 150.1.4.4 ebgp-multihop 255
 neighbor 150.1.4.4 update-source Loopback0
 neighbor 150.1.4.4 default-originate
 no auto-summary

I issues “clear ip bgp *” multiple times on both sides.  I removed the whole BGP configuration on both routers and then re-added them.  Finally, I reloaded both routers.  I still couldn’t get BGP to work.

I debugged BGP events:

r4#debug ip bgp event
BGP events debugging is on
*Sep 25 16:52:58.743: BGP: Regular scanner event timer
*Sep 25 16:52:58.743: BGP: Import timer expired. Walking from 1 to 1

r4#clear ip bgp *

*Sep 25 16:52:58.743: BGP: Regular scanner event timer
*Sep 25 16:52:58.743: BGP: Import timer expired. Walking from 1 to 1
*Sep 25 16:53:04.371: BGP: reset all neighbors due to User reset
*Sep 25 16:53:04.375: BGP(IPv4 Unicast): will wait 60s for the first peer to establish
*Sep 25 16:53:04.375: BGP(IPv6 Unicast): computed bestpaths, table version wentfrom 1 to 1
*Sep 25 16:53:04.375: BGP(VPNv4 Unicast): computed bestpaths, table version went from 1 to 1
*Sep 25 16:53:04.375: BGP(IPv4 Multicast): computed bestpaths, table version went from 1 to 1
*Sep 25 16:53:04.375: BGP(IPv6 Multicast): computed bestpaths, table version went from 1 to 1
*Sep 25 16:53:04.375: BGP(NSAP Unicast): computed bestpaths, table version went from 1 to 1
*Sep 25 16:53:13.743: BGP: Regular scanner event timer
*Sep 25 16:53:13.743: BGP: Import timer expired. Walking from 1 to 1
*Sep 25 16:53:28.743: BGP: Regular scanner event timer
*Sep 25 16:53:28.743: BGP: Import timer expired. Walking from 1 to 1
*Sep 25 16:53:43.743: BGP: Regular scanner event timer
*Sep 25 16:53:43.743: BGP: Performing BGP general scanning
*Sep 25 16:53:43.743: BGP(0): scanning IPv4 Unicast routing tables
*Sep 25 16:53:43.743: BGP(1): scanning IPv6 Unicast routing tables
*Sep 25 16:53:43.743: BGP(IPv6 Unicast): Performing BGP Nexthop scanning for general scan
*Sep 25 16:53:43.743: BGP(1): Future scanner version: 16, current scanner version: 15
*Sep 25 16:53:43.743: BGP(2): scanning VPNv4 Unicast routing tables
*Sep 25 16:53:43.743: BGP(VPNv4 Unicast): Performing BGP Nexthop scanning for general scan
*Sep 25 16:53:43.743: BGP: Import walker start version 0, end version 1
*Sep 25 16:53:43.743: BGP: … start import cfg version = 0

I did a Google search on “BGP: Import timer expired. Walking from 1 to 1” and came across a post suggesting the following:

1) You don’t have a route to it.

2) You need ebgp-multihop but haven’t configured it. (If it’s not on a directly connected network or you’re using update-source loopback, you need ebgp-multihop)

3) (Unlikely, I suspect you’d get a different error) It’s not configured to talk BGP to you.

1 – check.  2 – check.  3 – ummm check.

Actually, number 1 was my issue.  Even though I had looked at the OSPF config, I never did my due diligence and actually verified the loopback addresses from each side of the link(s).  When I finally did that, I found my problem:

r5#sh ip route 150.1.4.4
% Subnet not in table  <-this is a problem  🙂

Although I had glanced at the OSPF configurations, I didn’t notice my problem the first couple of times:

r4#sh run | sec ospf
router ospf 100
 router-id 150.1.4.4
 log-adjacency-changes
 network 155.1.0.4 0.0.0.0 area 0
 network 155.1.4.4 0.0.0.0 area 0  <-DOH!!! 150 not 155!!!
 network 155.1.45.4 0.0.0.0 area 0

r4(config)#router os 100
r4(config-router)#no network 155.1.4.4 0.0.0.0 area 0
r4(config-router)#net 150.1.4.4 0.0.0.0 area 0
r4(config-router)#^Z
r4#
*Sep 25 17:00:39.999: %BGP-5-ADJCHANGE: neighbor 150.1.5.5 Up
*Sep 25 17:00:41.255: %SYS-5-CONFIG_I: Configured from console by console
r4#sh ip bgp sum
BGP router identifier 150.1.4.4, local AS number 1
BGP table version is 1, main routing table version 1

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
150.1.5.5       4     2       2       2        0    0    0 00:00:12        0  <-success!!!!

My OSPF neighbors were established on each router using the router-id which was the same as the loopback address.  I didn’t think the problem through enough to realize that this meant absolutely nothing about the state of the route from each router to the other router’s loopback address.  I had fat-fingered the network address in r4’s OSPF configuration and therefore the network was never advertised into OSPF.  BGP was using the loopback address as the neighbor address.  Since it did not have an IGP route to the loopback, the BGP adjacency never established.  About 45 minutes of head-scratching later, I discovered the problem.

Internetwork Expert advises not to use loopback addresses like 1.1.1.1 (r1) because it is pretty easy for one of the BBC routers to use those types of address and inject some not-so-fun troubles into your lab.  On the same hand, if your loopback addresses are very similar to your active interface networks, it becomes pretty easy to mistype a network statement which will lead to problems like the one that I had.  It also makes it a bit more difficult to find the mistyped statement(s) when you’re quickly trying to troubleshoot.