Switching: Campus Design

icon Switching: Campus Design


This article continues the series on LAN switching and Cisco Catalyst switches. We'll take a look this month at designing campus environments. The key issue here is the appropriate mix of routing (multi-layer switching) and switching activities. Of course, the issue is rendered more complex because routers are acting more and more like switches, and vice versa!

For more detailed info, see the book by Mentor Technologies' Kennedy Clark and Kevin Hamilton: CCIE Professional Development: Cisco Lan Switching (The Cisco Press CCIE Professional Development Series) . (From what Kennedy tells me, switching router and routing switch are the compromise terms the book uses).

Prior articles on switching or related topics:

Hierarchical Design

If you've taken the Cisco Introduction to Design (CID) course, or if you've read Priscilla Oppenheimer's Top-Down Design book (Cisco Press), or if you've attending the right talks at a Networkers, you've run into the hierarchical design theme. In which case you may tune out this paragraph!

The general idea is that you don't want to design networks with everything connected to everything. There can be serious cost and performance issues that can arise from such an architecture. The details vary with the level of network being discussed. Furthermore, such an approach can prevent your network from easily growing or adapting to new technology -- and networks always grow and evolve!

There are other advantages to hierarchical design. It can help with troubleshooting, because the network and the paths taken by packets are more predictable. It can help with operational management of the network, since it may become easier to determine what's going on and where to add bandwidth. Routing can become simpler, because a hierarchical network lends itself to a good OSPF or EIGRP design with route summarization (and understanding 10 routes instead of 600 is a lot easier!). And routers do not do well with large numbers of peer routers that they must exchange routing information with.

If I don't have you convinced, think people. Can a committee of 100 or even 50 people get useful work done? Why do most companies divide things up into workers of various kinds, first level managers, etc.?

Another side of hierarchy is that it helps you migrate a network to new technology. For example, consider campus backbone technology. If you have all your 100 Ethernet switches connected to an ATM core network, then migrating to Gigabit Ethernet (or 10 Gigabit Ethernet, which is coming) is costly. You'd have to put the appropriate board into each switch, and you'd have to reconfigure each switch. If instead you have 10 or 20 smaller switches grouped off a larger switch, then changing the backbone affects only 5 to 10 switches. Less work, lower cost!

I see hierarchy as one reason ATM to the desktop never caught on. The primary reason is probably that protocol stacks and applications that deeply support ATM at the desktop never arrived -- it takes time to rewrite all that code, or develop new code. But the other reason is that you can stick a 10/100 switch in the closet, and connect the desktops up to that. Then you can migrate PC's incrementally, instead of having to buy across-the-board ATM or other NIC's. Furthermore, you can upgrade the building backbone in steps: Fast Ethernet, then Fast EtherChannel, then Gigabit Ethernet, then Gigabit EtherChannel. Ditto for campus backbone. Note the implicit hierarchy here: if you've got a building or a campus backbone, you've already got some hierarchy. And if you don't...

Exercise for the reader: think of 3 other reasons why networking hierarchy is good.

The traditional Cisco hierarchy has been: backbone or core, distribution, and access. We'll come back to hierarchy in a little bit.

End To End VLAN's

For a year or two, the prevailing wisdom was what I'll call End To End VLAN's. The idea was to run VLAN's to random desktops depending on requirements, and to run the VLAN's all the way across campus, perhaps even WAN, to the servers for that workgroup. This approach treats a VLAN as "virtual cabling". There's nothing wrong with this approach, if you're doing it for good reason. We'll see below that there are some pros and cons to doing End to End VLAN's.

The main reason campus design focussed on End To End VLAN's over the past couple of years is Layer 3 Switching performance. It was correctly assumed that routers formed a potential bottleneck between fast clients and fast servers. Cisco and the other vendors did not have the Layer 3 switching performance to address this.

Some trade magazines or journalists seem to have other, somewhat confused, reasons for espousing End To End VLAN's. That's why I wanted to name this approach -- so we can rationally discuss the pros and cons.

One myth is that Desktop protocols require clients and servers to be in the same broadcast domain. (Recall this term from a prior article). That may have once been the case but is no longer true, although you do have to know how to get your desktop network protocol(s) to co-exist with routers.

Another myth is that you need End To End VLAN's for security. Well, VLAN's help you isolate groups of users, so I'll agree there's a tie between VLAN's and security. Why the introduction of a router or layer 3 switch makes things less secure is what escapes me about this line of thought. Even with End To End VLAN's, the users usually have to talk outside their segment of the network.

The biggest deal with End To End VLAN's, for me, is that they are, um, interesting to manage. If you try to draw a map of a campus network with many End To End VLAN's, you end up with something looking like a printed circuit diagram. If you try to troubleshoot such a network, you first spend some time running around to verify that the VLAN follows the desired (or some) path between the user and the server or router they cannot talk to. Since the path is a layer 2 path, you need to check the Spanning Tree for that VLAN. And you need to check the MAC tables for that VLAN, to see if the switches have learned the source and destination MAC addresses on the right ports.

Exercise for the reader: how do you go about verifying that an End To End VLAN is secure?

There is one dramatic plus to End To End VLAN's. If you use this approach, you can potentially support mobile users on your campus. Someone can move to a new port, and you just change the port to the right VLAN. Except: how do you know the user on the phone is who they claim to be? Your users know what VLAN they're on? You wanted to be involved every time someone moves a PC on your campus? You can perhaps see, there might be times when the answer to some or all of these questions is yes -- but that you might need more staff to take advantage of this.

The previous article on VMPS (and the ones on CWSI) might have you thinking "well how about dynamic VLAN's"?  Good question! But I'll point out that whether or not you use dynamic VLAN's is independent of whether those VLAN's run across campus to the switches. DHCP might be another low maintenance approach to mobility.

MultiLayer Campus

The biggest thing that changes with the MultiLayer Campus model is that you throw in one routing hop, or at most two. Well, Layer 3 Switching hop or hops.

Where you put the Layer 3 switching depends on the size of your network. Let's put together some of what's been in the previous articles.

You can have a flat network up to a point. When do you start to need multiple VLAN's? Well, the numbers in a previous article suggested 200, 300, or 500 hosts, depending on chattiness, broadcast level, and perhaps protocol.

As soon as you have multiple VLAN's, you're going to need a router (or Layer 3 switch) to route between them. If you do End To End VLAN's, you try to use the VLAN's to carry the bulk of the traffic, and you put the (hopefully, small amount of) inter-VLAN traffic through a router somewhere on the campus (perhaps connecting to the WAN). The MultiLayer Campus approach realizes that Layer 3 switching is no longer a bottleneck, that Layer 3 switches are less costly now, and so it distributes Layer 3 switches (and the workload) around your campus. In the process of doing so, it takes advantage of the Layer 3 switches to do "risk containment" (thanks to Kevin Hynes for the phrase).

The MultiLayer Campus approach takes the divide-and-conquer logic a step further. You did the multiple VLAN's to reduce the sizes of Spanning Trees (perhaps). It certainly reduced levels of broadcast traffic. But you also might want to reduce risk to your network, to contain the scope of any outages. Layer 2 problems can propagate in flat switched networks. One VLAN with a problem can saturate a trunk, affecting the other VLAN's carried by that trunk. Large Spanning Tree domains can also lead to stability problems ("Spanning Tree weirdness"). With End To End VLAN's, unless you block or prune VLAN's from trunks, you potentially have this problem. Historically, that's one of the reasons routers were used in larger campus networks: segmentation with routers, while costly, proved desirable for stability and control.

So with the MultiLayer Campus, we're going to take a hierarchical approach, and modularize our campus design by inserting Layer 3 switches as the couplers between the components. Where exactly you draw boundaries depends on the size of your network, users and traffic, budget, performance needs, and other requirements. One way to do things would be to have buildings isolated by Layer 3 switching. That is, your distribution layer switches would be Layer 3 switches. You'd use pure Layer 2 switches for access (in the wiring closets), and you might even use pure Layer 2 switches for that Gigabit Ethernet backbone.

The picture attempts to show this.

Note that each access layer switch (wiring closet) is a separate VLAN, indicated by the colors. You might have two per wiring closet if you want to use VLAN prioritization to load balance. This means that there is one or two subnets and they are on a per-wiring closet basis. So the address of a PC tells you something about where it is -- very useful for troubleshooting!

The links to the distribution layer switches are shown in color, but are probably trunks so that you can put all the building switches in a common VTP domain and use CWSI to manage them. That also leaves you more flexibility in case of special future needs or changes. The thick black link between the distribution layer switches is also a trunk, and the VLAN's all run across the trunk. That means your Spanning Tree is very simple: you've got a triangle (plus access ports). If you make the distribution layer switches primary and backup Root Bridge, then one of the colored links will be active for a VLAN and the other backup. The UplinkFast feature was designed for precisely this situation -- configure it only on the access switch, not on the distribution layer switch.

The distribution layer switches route traffic to the backbone (Layer 3 switching). So the yellow links to the backbone are not trunks, they are in a specific VLAN and subnet (or two), different from all the others in the picture. If you wish to keep things very simple (always a Good Thing), then block the yellow VLAN on the trunk between the distribution layer switches (and any other trunks), so that Spanning Tree is not an issue for the backbone.

Exercise for the reader: is verifying the security of a VLAN easier or harder with the MultiLayer approach?

Exercise for the reader: with mobile users, how do you do security? In a hierarchical, manageable way?


That leaves the question of what to do with your backbone. If you've only got one building, maybe that's your network. If you are on a budget and don't need the redundancy, you might use one building distribution switch instead of two.

If you have two buildings, the standard approach would be to repeat the above diagram for each building, and then interconnect the four building switches, either in a ring (Spanning Tree?), or in some combination (pairs?). (Exercise for the reader: what are some of the ways you could do this? Draw pictures!) Conceivably you could just have two large switches and run all connections to them -- except that then you'd probably have a mess of cables going from one building to the other, which somewhat defeats the usual structured cabling practices (and might be a maintenance and future upgrade issue as well). It might also run into distance limitations or get costly.

At some point, as you add buildings to this you need more structure to your backbone. There seem to be two approaches. Both involve star cabling the building distribution switches to central backbone switches. One approach regards each backbone switch as a subnet, and avoids creating possible loops to keep Spanning Tree out of the picture (eliminating Spanning Tree convergence delays from the backbone). The other approach possibly interconnects the backbone switches with a link, perhaps does not block the yellow VLAN from the black trunk (or other trunks) in the above picture. But under the second approach you might tune Spanning Tree (very small possible domain), use Backbone Fast, and use other tricks to minimize Spanning Tree convergence time.

The following picture shows the first, simpler approach. That's the one I prefer.

Wrap Up

The above design is the approach used in the labs in the BCMSN course. So (blatant plug) if you want to learn how to configure all this, attend the class!

I'm thinking that the next article may be about LANE (LAN Emulation), since that seems to fit in nicely with the theme of this series. If you have requests for topics you'd like an article about, please do send me email! See you in a month.

Dr. Peter J. Welcher (CCIE #1773, CCSI #94014) is a Senior Consultant with Chesapeake NetCraftsmen. NetCraftsmen is a high-end consulting firm and Cisco Premier Partner dedicated to quality consulting and knowledge transfer. NetCraftsmen has eleven CCIE's (4 of whom are double-CCIE's, R&S and Security). NetCraftsmen has expertise including large network high-availability routing/switching and design, VoIP, QoS, MPLS, network management, security, IP multicast, and other areas. See http://www.netcraftsmen.net for more information about NetCraftsmen. . New articles will be posted under the Articles link. Questions, suggestions for articles, etc. can be sent to This email address is being protected from spambots. You need JavaScript enabled to view it. .

Copyright (C)  2000,  Peter J. Welcher