Cisco Live is obviously the biggest networking event of the year, and Cisco likes to use all the attention to show off some of their new gear.  I must say I was impressed with some of the Enterprise offerings including the 6807-XL, the 6880-X, the 4451-X, and the Sup 8-E for the 4500-E (check out the Nexus 7700, too, even though they aren’t Enterprise class).  Those boxes definitely gave me a bit of a tingle when I was checking them out, but my eyes opened up when I saw the 3850 in one of my sessions and on the show floor.

Did you catch the article on setting up fault tolerance on the CSM?  In that article, I mentioned that Cisco recommends a dedicated trunk for the FT VLAN if you have two HA CSMs in two chassis.  Discuss amongst yourselves while I drone on.

Why should you set up a dedicated trunk for this stuff?  The most obvious reason is to be sure that normal traffic doesn’t step on the syncing traffic.  Since we’re syncing state information as well as configuration, the frames need to arrive in a timely manner.  Any errors could potentially disrupt the FT process, which is bad.  You surely don’t want the primary to fail only to find out that the standby doesn’t have the complete or current config.

There are three different ways that a CSM checks for the health of the servers – active probes, inband health checking, and inband HTTP monitoring.  Let’s talk about active probes.

Active probes (or just probes) typically send traffic to one of the RIPs of a serverfarm, do some stuff, and give a pass or fail grade.  If the probe fails a certain number of times in a row, that server is considered sick and taken out of the pool for use.  The CSM keeps checking the unhealthy until it passes a number of times in a row, at which point it is placed back in the pool for use.  Almost everything is configurable, of course, so let’s look at some of those settings.

We ran into this today, and, though I knew it existed, I never actually saw it in the wild.  I’m talking about MAC access-lists.

In the example setup, we have a DMZ off of a firewall that contains a whole mess of servers – email, web, ftp, etc.  These should all be in the DMZ for sure, but they shouldn’t talk to each other.  If a bad guy was able to own my FTP server, he would have a nice platform to use to attack my email server.  That’s not cool, so we’ve put in MAC access-lists to help out.

Like (nearly) everything in the Cisco world, you can set up your CSM to fail over to another module when the primary dies a horrible death.  You can have two in the same chassis or even have them in separate chassis – the process is the same no matter how you have it set up.  Either way, you have a primary and a secondary module in fault tolerance (FT) mode.

At the office, we reprovision servers like it’s going out of style.  It happens so often that my cabling documentation rarely matches what’s actually out in field, which is a pretty big problem when you’re trying to find to what switch port a server is connected.  I finally relegated myself to asking for the MAC address of the server, having the admin ping something, and then tracing it down through the CAM table entries of the switches.  It works, but the guys really don’t know how a switch populates its CAM table, so they always say “Why can’t you just look on the switch?  I shouldn’t have to ping anything.”  Here’s one just for the aspiring system admin.

We run a large number of LANs all over the country that are “controlled” by the particular business unit. We manage the gear, but, since they have the money and have to pay for anything we do, they make the final decision on what gets put in. Sometimes that gets out of hand, as you can well imagine.

A good terrible example came up a few months ago. It seems that, at some time in the past, one site needed some more LAN ports, but, instead of calling us and having us send them another switch, one of the “technical people” there brought in a hub from home. It really irks me to see a hub on the switched LAN, but we really have no control over those decisions. They plugged the hub into one of the existing drops somewhere in the building and plugged everyone in. It worked…until somebody moved one of the machines. The machine was at a desk near the hub, and the network cable, still with one end plugged into the hub, was just left lying there. A good Samaritan came by, saw that the hub was not plugged into the network (though it was through another path), and plugged it back in for us – providing a nice second link from the hub to the switch stack in the closet. Take one switch stack, add a hub, insert a switching loop, bake at 350F for a few milliseconds, and you have a broadcast storm. If you don’t know already, broadcast storms are bad and eat switch CPU like the yummy cookies we baked. In this case, several 3750s were taken completely down.

In my professional life at some point, I came across someone who had a stack of Catalyst 2950 switches all trunked together with their Internet routers connected to the top of the stack. This was all well and good until they kept adding hosts to the “middle” of the stack, then they had all sorts of latency and packet loss.

The old adage of your chain only being as strong as your weakest length holds true in this case. Here, the weakest link is actually the most-congested trunk, though. Let’s step through to see. A 2950 is a 10/100 switch, so a single trunk can handle 100Mbps of traffic. We have 10 of these guys, Switch1 to Switch10, all trunked to the one above and below. If a server in the center of the stack on Switch5 is sending a lot of data to the Internet routers on Switch1, the trunks off of Switch5 will start to get saturated. Switch4 has a few hosts doing the same thing, so traffic from both Switch4 and Switch5 heads towards Switch1, further filling the trunks. Same for Switch3. Same for Switch2. Next thing you know, there’s 184Mbps or so trying to go across a 100Mbps link.

If you didn’t now already, trunks are connections between switches that carry traffic for all VLANs. It allows you to have, say, VLAN 10 and VLAN 20 on two switches appear as the same network. Unless you’re a really small shop, you’ve already dealt with trunks, so there’s no need for an introduction.

Let’s say we have a Catlyst 2950 switch with multiple VLANs connected to another 2950 configured with those same VLANs. We’ll say we have VLANs 10, 20, and 30 and that the switches are connected to port F0/24 of each switch. First, let’s turn on the trunk.