Switching: MultiLayer Switching

Peter J. Welcher

Introduction

This article continues the series on LAN switching and Cisco Catalyst switches.

MultiLayer Switching (MLS) is a key component of high performance at Layer 3 with the Catalyst 5000 switches. It also is a marvelous opportunity to review what happens when packets pass through a router. Understanding this is very useful in understanding router and switch performance optimization(s), also in troubleshooting network design and configuration issues.

Once again, for the ultimate reference on this sort of thing, I'm going to refer you to the book by Mentor Technologies' Kennedy Clark and Kevin Hamilton: CCIE Professional Development: Cisco Lan Switching (The Cisco Press CCIE Professional Development Series) . This article is (as always) going to try to briefly give you the idea of what's going on and how to configure it, without getting bogged down in too many details. There is also a chapter and lab on this topic in the BCMSN switching course, which Mentor Technologies has been teaching since September. (I hear most Training Partners are stuck on the less useful old CLSC course, at least until January 2000).
Prior articles on switching or related topics:

Switching: VLAN's ( http://www.netcraftsmen.net/welcher/papers/switchvlan.html )
Switching: Trunks and Dynamic Trunking Protocol (DTP) ( http://www.netcraftsmen.net/welcher/papers/switchvtp.html )
Switching: Dynamic VLAN's, VQP, and VMPS ( http://www.netcraftsmen.net/welcher/papers/switchvmps.html )
Switching: CiscoWorks 2000/CWSI ( http://www.netcraftsmen.net/welcher/papers/cw2000.html )
Switching: MultiLayer Switching ( http://www.netcraftsmen.net/welcher/papers/switchmls.html )

http://www.netcraftsmen.net/welcher/papers/cwsi.html

What is MultiLayer Switching?

MultiLayer Switching (MLS) is where a Layer 2 switch blade does Layer 3 (or pseudo-Layer 3) forwarding of packets between subnets. You can find out about this in the recent documentation by looking for Layer 3 Switching Guides under the 5000 or 6000 models.

The particular form of Layer 3 switching (hardware-assisted routing) we'll be looking at is implemented with the Netflow Feature Card (NFFC) functionality in Catalyst 5000 series switches. This requires that you have either the Supervisor III engine with NFFC daughter-board, or the Supervisor IIG or IIIG card (no daughter-board required). It also requires that you have either a suitable external router or the internal Route-Switch Module (RSM) blade (or Route-Switch Feature Card -- RSFC -- with the Sup IIG or IIIG blade). Suitable external routers are those implementing the router component of MultiLayer Switching, which is now all the models of larger size / more capability, equipped with an IOS image that is MLS-capable. The online documentation specifies that this is the Cisco 7500, 7200, 4700, 4500, or 3600 series router with an IOS image that supports IP MLS. (IPX MLS is also available, but of less general interest. Ditto, IP multicast MLS).

The MLS functionality is also supported (with greater simplicity, and fewer configuration options) in the 6000 series!

To understand what's going on, let's first talk through what happens when a packet is forwarded by a router. There's lots that goes on in addition to what follows, but let's not muddle things with unrelated detail. These steps are the ones we're interested in:

router interface hardware receives the bits and checks the checksum
router strips off the Layer 2 header (de-encapsulates the packet)
router determines outbound interface and next hop IP address from routing table
router builds new Layer 2 header
new Layer 2 header and payload are assembled and queued for the outbound interface
outbound interface sends out the packet, tacking on FCS checksum at end

If you're really into switching performance, you understand that the routers cache the outbound interface and new Layer 2 header information. And the routing table lookup is preceded by cache lookup(s), possibly hardware-assisted and distributed to the interface cards, to get the packet out of the box as fast as possible, with as little central CPU involvement as possible. Let's not go into the details of this, since that's probably a whole article in itself.

The key operation that is needed to understand MLS is the de-encapsulation / re-encapsulation. The router strips off the old Layer 2 header, slaps on the new one, and queues the packet for transmission. The Layer 2 headers are per-hop (from router to router across a subnet). The MAC addresses in those Layer 2 headers are per-hop, local to the Layer 2 segment (broadcast domain; VLAN). IP addresses in the Layer 3 IP header are end to end: the ultimate sender and recipient.

In doing MLS, a NFFC switch blade (or Supervisor IIG / IIIG with built-in comparable functionality) is doing the job of emulating the router, without actually sending the packets to the router. In order to simulate a router transparently, the NFFC has to de-encapsulate and re-encapsulate the packet, and also re-calculate the Layer 2 checksum (FCS).

A requirement for the MLS functionality to work is that you enable it in both the participating router and switch components. (With the 6000, MLS on the switch is enabled by default, and is only disabled when you disable MLS on the switch router component, the MSM or MSFC). Furthermore, MLS only applies to packets where the switch is in position to observe the packet before and after it passes through the router. See the figure below.

My analogy for MLS is a manager with a smart assistant. The manager and assistant agree that certain responsibilities are delegated to the assistant. When the assistant observes how the manager handles certain tasks, the assistant will step in and imitate the manager's previous decision(s). The manager agrees to this, and will let the assistant know when the policy changes.

In the case of MLS, the router is the manager, with packet switching at Layer 3 being delegated to the switch. When a packet passes through the switch, the switch snoops (in effect) to see what the router does with the packet. As it comes back through the switch (post-router, after de-encapsulation and re-encapsulation), the switch learns the new Layer 2 header. When subsequent packets reach the switch bound for the same IP destination, the switch can then just imitate the router, by replacing the Layer 2 header with the cached (stored) Layer 2 header, just as the router previously did. If routing changes, the router does need to notify the switch, which flushes the cached Layer 2 header, since a new routing policy is now in effect.

I hope you're wondering, "but switches make forwarding decisions based on MAC addresses". True! When does a router get involved in forwarding packets? When they are headed for a different subnet. Within the same subnet, MAC address is all that is relevant. That's where switches forward -- within the same VLAN. But when a PC or host computer is sending to another subnet, it does something differently. The destination MAC address is no longer that of the destination computer. It is instead the MAC address of the default gateway router interface.

This is another function of enabling MLS in the router. It can in effect tell the switch (or we can configure into the switch) the MAC address of the router. The switch needs to know that the router's MAC address is special. The frames sent to that MAC address are the frames where the switch needs to subsequently snoop to "observe" the router's imposition of a new Layer 2 header!

Let's re-visit this in more detail now.

Details of MLS Activity in a Switch

When MLS is configured (enabled) in a router (the MLS Route Processor, or MLS-RP), the MLS-RP sends out a multicast announcement that it is doing MLS. The multicast address is the CGMP well-known address, 01-00-0c-dd-dd-dd. This multicast should pass transparently through non-Cisco switches, and tells MLS-capable switches to learn the router's MAC address. The MAC address is internally associated with a one byte XTAG, an internal identifier for fast lookup.

Now suppose a host or PC attached to the MLS switch sends a packet destined for a different subnet. Assuming that the MLS-RP is also the PC's default gateway, the switch observes that the router's MAC address is the destination of the Layer 2 frame. Since the router is a MLS-RP, the switch checks its cache, to see if it already knows what to do for packets from that flow (we'll talk about flows later). If there is a cache entry, the switch replaces the Layer 2 header, then uses normal switching procedures to deliver the frame (Layer 2 destination MAC lookup and forwarding).

If there is no cache entry, the switch creates a temporary cache entry, a place holder for when the post-router frame is seen. (A candidate cache entry). The switch forwards the frame to the router. The router does its thing. The frame comes back to the switch (different VLAN, same switch). The switch sees the candidate entry, records the Layer 2 header in the cache. And forwards the frame.

Subsequently, packets from the same flow have Layer 2 headers rewritten (and checksums recalculated and rewritten) by the switch hardware, and are forwarded without actually having gone through the router.

Should a router get disconnected from the switch, the XTAG provides a fast way for the switch to invalidate entries learned from a particular router.

Configuring Routing Between VLAN's

External Router

If you're routing between VLAN's on a switch with an external router, you are either trunking (see earlier article) or you are not. If you're not trunking, each router interface is a separate subnet, and the router is configured just as usual for IP subnets, on the main interfaces. If you are trunking, then you need to configure the router accordingly, using subinterface. This will look something like the following:

interface fastethernet 0/0
no ip address
interface fastethernet 0/0.100
encapsulation isl 100
ip address 172.16.1.1 255.255.255.0

interface fastethernet 0/1.105
encapsulation isl 105
ip address 172.16.5.1 255.255.255.0

Note that we've made the subinterface (101 or 105) match the VLAN number in the encapsulation isl (or dot1q) statement. This convention means not having to track which subinterface goes with which VLAN. A best practice!

The "encapsulation isl" and the subinterfaces is your clue that this router thinks the FastEthernet is a trunk and that it is doing ISL VLAN's.

Internal Router (RSM/RSFC/MSM/MSFC)

Internal routers in Catalyst switches think their interfaces have names like "interface vlan 1". So the above example becomes merely:

interface vlan 100
ip address 172.16.1.1 255.255.255.0
interface vlan 105
ip address 172.16.5.1 255.255.255.0

Referring to the above figure, the trunk in the picture is now instead the internal backplane of the switch. Both the router and switch components are inside the chassis.

Configuring MultiLayer Switching

Configuring MLS has two components: configuring the router (external or RSM/RSFC blade), and configuring the switch (Supervisor blade). We'll talk about each of these in turn. The following are for a Catalyst 5000.

External Router

You need to globally enable MLS with:

mls rp ip

If the switch has a VTP domain, the MLS physical (primary) interface must agree:

interface fastethernet 0/0
mls rp vtp-domain domain-name

You then enable MLS per interface:

interface fastethernet 0/0.100
mls rp ip

You also need one interface which is connected to the switch set up as the MLS management interface:

interface fastethernet 0/0.100
mls rp management-interface

If the external router has only non-trunk links to the switch, you need to configure an associated VLAN id with each:

interface ethernet 0
mls rp vlan-id 100

(With trunk interfaces, the ISL or 802.1q tag provides VLAN info, needed in the MLS-RP hello).

Internal Router (RSM/RSFC/MSM/MSFC)

With internal routers, basically just the interface names change (and you can omit the vlan-id, since it is implicit with each interface).

mls rp ip
interface vlan 100
mls rp vtp-domain domain-name
mls rp ip

interface vlan 105
mls rp ip
mls rp management-interface

Supervisor

MLS is enabled by default on MLS-capable switches. If disabled, re-enable it on a Catalyst 5000 with the command:

set mls enable

With an external router, you need to tell the switch the router's IP address:

set mls include ip-address

(where ip-address is the address shown on the router with the "show mls rp" command).

Catalyst 6000 MSFC

The only important thing you can configure with Catalyst 6000's is MLS on an interface. It defaults to "on" on the MLS-RP, and cannot be globally disabled (except perhaps by disabling it on all interfaces?). The MLS-SP defaults to "on", and is apparently only disabled if MLS is off on all MSFC interfaces. Sample of how to turn MLS off on an interface:

interface vlan 100
no mls ip

But What Is a Flow?

A flow might be all packets for a certain IP address. Or it might be packets between an IP source and destination. Or it might be packets between a certain IP source and destination on a single port. The meaning varies here because the switch caches based on flows, and it only caches what it has to. With normal routing, only destination matters. Throw in a standard IP access list on the MLS-RP, and suddenly source address matters. So in such situations, the MLS-SP (switch) caches based on source-destination IP pairs. Throw in an extended IP access list and transport protocol and destination port makes a difference. So then the switch caches based on source-destination-transport protocol-port information. The fancy name for this is "flow masks", which are respectively destination-ip, source-destination-ip, and ip-flow in the Cisco documentation.

If you want, you can configure the switch to force it to cache more specifically. This may impact performance, and would be a fairly odd thing to configure. For instance, with no access lists on the router component, you could configure the switch component to force at least source-destination-ip mask:

set mls flow destination-source

The other keyword options for minimal flow mask are destination and full, corresponding to destination-ip and ip-flow above.

Wrap Up

We're out of space, so I'm going to tell you that the relevant show commands mostly start with "show mls". Knowing that, you can use the help system to figure out what's available. And if you try this out, I hope and trust the output will make good sense.

There is also a "clear mls" command (or set of commands).

You can configure cache aging time, which is how long an idle period between packets before a cache entry ages out. You can also tweak the fast aging time, which is the length of time a new entry stays in the cache if the configured switch packet threshold isn't passed. (The idea is to conserve resources by using cache entries for flows with many packets. Those with few packets fail to meet the packet threshold and so their cache entry gets dropped more quickly).

There are also a number of fine points to MLS behavior, some caveats (warnings), and some situations where MLS becomes disabled. For details on all these, see the documentation.

I'm still considering doing an article on switched network design. See you in a month!

Dr. Peter J. Welcher (CCIE #1773, CCSI #94014) is a Senior Consultant with Chesapeake NetCraftsmen. NetCraftsmen is a high-end consulting firm and Cisco Premier Partner dedicated to quality consulting and knowledge transfer. NetCraftsmen has nine CCIE's, with expertise including large network high-availability routing/switching and design, VoIP, QoS, MPLS, network management, security, IP multicast, and other areas. See http://www.netcraftsmen.net for more information about NetCraftsmen. Pete's links start at http://www.netcraftsmen.net/welcher . New articles will be posted under the Articles link. Questions, suggestions for articles, etc. can be sent to pjw@netcraftsmen.net .