Introduction

This article updates an article from 1/2004, which can be found at  http://www.netcraftsmen.net/welcher/papers/metroeth01.html. A lot of that article is still valid. In particular, it contains some useful acronyms, including a couple I invented. What I see as changed since that article is that there is a shift in Cisco's emphasis from optically-based service to Ethernet services based on MPLS transport, possibly with switched networks as Service Provider Points of Presence.  Vendors other than Cisco are pushing whatever delivery approach matches what they have to sell (but that's not news). And Service Providers are starting to get more aggressive about Metro and National scale Ethernet Networks. 

To avoid any puzzlement, if I have to use an acronym for Metro Ethernet Networks, it'll be METN not MEN or ME. (The Metro Ethernet Forum uses "MEN"). Based on that, you can probably figure out NETN. I'll use METN-SP for Metro Ethernet Network Service Provider. But enough about acronyms!

Ethernet Service Architectures

• Optical-Based METN
  
• (Ethernet) Switch-Based METN
• MPLS-Based METN

We'll look at these in turn. I'll mention vendors, where I know of them, and discuss pros/cons and things to be aware of for each of these architectures.

Optical-Based METN

The good news about this approach is that it is fairly simple and static. It's your pipe, dedicated to you, so you don't have to worry about sharing bandwidth. QoS is simpler. Furthermore, provisioning is fairly simple, and may resemble the "classical" SP model. So your circuit will probably be ok unless somebody messes up while provisioning somebody else's new circuit. The simplicity may also pay benefits in terms of stability, Mean Time to Repair when there's a problem, and so on.

Another bit of good news is that each customer has a "dedicated pipe". So if somebody has a bad day and suffers a spanning tree loop, the excess traffic should not affect other users of the optically-based service.

On the down side, since the provider cannot oversubscribe trunks, this approach is arguably a bit wasteful of bandwidth. That may end up costing you more. In addition, DWDM gear isn't cheap. At the very least, you're getting an access port tied directly to DWDM gear, whereas with other approaches, the DWDM gear is either on the SP side of an edge device, or not present. If DWDM or SONET gear is needed at your premises to drive the connection to the SP, then your cost can be considerably higher. Increasing speeds may require a costly hardware swap-out.

Also on the down side, optical provisioning software can be a bit ugly right now. There is motion towards GMPLS for control, but that whole area is immature.

Switch-Based METN

Ethernet switches can be used to deliver METN services. I include optical devices where the edge access device provides Ethernet switching in this category. One advantage of this approach is that lower-cost 1 and 10 Gbps ports can be used to interconnect the switches (unless the edge device has a DWDM or other optical provider-side to it). Long-haul GBIC or SFP modules can be used for the customer-facing side of a provider offering, reducing cost.

Small-scale METN-SP's using switches might provide one or several VLANs per customer. The concern here is that they are sharing Spanning Tree Protocol (STP) domains with their customers. That seems highly unstable and somewhat unscalable.

More sophisticated METN-SP's will probably be using 802.1Q tunneling, also called QinQ. See the Cisco technical article link below for details of how QinQ operates. The basic idea is that an extra 802.1Q tag is applied to your frames. This "outer tag" is then used by the SP switches to switch traffic between your sites, your access ports. When the vendor runs out of VLAN's to use (4096 customer VLAN's?), they can then start using a separate set of Ethernet switches ("red cloud", "blue cloud", etc.), re-using VLAN numbers in each separate set of switches.

Think about how this behaves. Your MAC addresses are still visible, and are used to switch traffic between sites. That can result in flooding of unknown unicasts. IP multicast still has a multicast MAC address for destination (a sharp observation made to me by Keith Boblits) and floods to all your egress ports, as do broadcasts.

If the METN-SP requires you to use routers and not switches for the device connecting to this, then the number of unknown unicasts and broadcasts should be greatly reduced, a Good Thing. Edge routers would also greatly reduce the number of MAC addresses the edge SP switches would have to learn. Practically speaking, how would one enforce such a rule? Legal conditions may also prevent the SP from trying to enforce a "routers only" rule.

One design I've had in the back of my head is a moderately-sized school district, manufacturer, retail chain, etc. Putting switches in schools, all connected by METN to a HQ L3 switch, would provide a relatively inexpensive and simple to manage network. In fact, that model works well for shops with many sites, where the data network is viewed as a cost item rather than as strategic value-add. So this approach might be a managed service model that METN-SP's would find attractive. The central L3 switch would mitigate flooding. The downside would be the greater number of MACs needing to be learned by the SP POP edge switch. I personally think stubby routing in such an environment would be simple enough that I'd use it instead.

As far as pros and cons go, on the plus side this scheme is also relatively simple for the SP to deploy. It does have some scalability issues and the number of customers go up, but partitioning (red/blue) and mixing in MPLS L2 transport may help with that.

This approach does have one significant negative aspect. The SP would be using Spanning Tree Protocol (STP) throughout their delivery network. We have been minimizing use of STP in our designs. This was due to concerns about convergence time and stability. In addition, it limits the size of "failure domains".

Rapid STP greatly mitigates concerns about convergence. It also increases general stability of STP. With traditional STP, we generally saw random instability set in when the Spanning Tree domain included 20 to 30 switches. We had a couple of early designs where we recommended against that approach, the customer insisted or had already built it, and we observed instability. This was apparently partly due to STP domain diameter (number of switch hops), also due to increasing probability of ripple effects whenever a BPDU was delayed or lost. Faster switch CPUs have also helped with this. Since our designs now generally have at most access layer STP in them, there just seems to be little point to having larger scale STP (or potential failure) domains. We haven't been experimenting live with customer networks to test the stability of RSTP. So perhaps our rule of thumb (20 to 30 = likely instability) could be relaxed some.

Thus, I'm not going to say that a 10-20-30 switch local METN-SP switched network will be unstable. I do however view larger scale RSTP as a business risk until more experience is gained. The potential "failure domain" remains the entire switched network.

Furthermore, we have had enough experience with troubleshooting customers' STP loops that I can unequivocably state that diagnosing and fixing such a loop can be excruciating, and can take considerable time. The problem is that you have to simultaneously snapshot the state of the Spanning Tree on every switch participating in the afflicted VLAN, but due to the loop, network management  tools cannot help in this. If you have an out-of-band management network, you may be better off in this regard, unless the switch CPU is getting so spun up it doesn't respond to SNMP (or worse, CLI).

Tight configuration rules and good practices can greatly reduce the likelihood of STP loops. Generally STP loops have two causes: turning off STP on edge ports (bad idea!), and turning off STP on links between switches (worse idea!).

Lately I've had some unintended fun with small rogue switches. My conclusion: Cisco BPDU Guard and Root Guard are Good Things. If a user plugs in a Linksys or home-grade switch, there's generally no CDP to tell you it is there.  It may become root switch for the VLAN if you haven't set root priority elsewhere. It probably has the weakest CPU in the STP domain, so you can then get delayed BPDU's and all the other issues that lead to STP domain-wide instability and/or loops. Having said that, a good SP will have controls, configuration procedures, and configuration audits that may mitigate a good part of the risk. I can't say there's no risk here, but at least it is a fairly-well understood and mature risk.

There is another consideration with switch-based METN. So far, we've discussed SP-side STP. But with a multipoint service, the customer is also running STP, probably with BPDU's transparently tunneled through the SP network. If the customer has a STP meltdown, they may be putting a Gbps of broadcasts into the SP network. What does that do to the SP's trunks and other customers? How does the SP guard against this? The customer is paying for a Gbps of traffic, so the SP can't just slap on some rate limiting.

Finding or installing fiber from your site to the SP Point of Presence (POP) can of course still be a challenge, but that's true with any of these designs.

MPLS-Based METN

Under the hood, core routing among the SP's MPLS routers provides the connectivity between edge devices. The IP routing automatically drives the creation of Label Switch Paths (LSP's) between edge devices. When your Ethernet frame arrives, an MPLS label stack is applied. The frame is sent via a LSP to the appropriate edge device and then switched out the correct port, with changed VLAN tag if appropriate.

As far as the Ethernet switching side of things, behavior depends on whether the service is point-to-point and transparent, or multipoint ("Ethernet Multipoint Service"). In the point-to-point case, transmitted frames are simply forwarded down the "pipe" (pseudo-wire). In the latter, multi-point, case, the SP network is supposed to act like a giant distributed switch. The Provider Edge (PE) devices learn MAC's in a private context. They treat the pseudo-wire links as Ethernet links, and apply the usual rules of switching. Except that they do split-horizoning, i.e. do not re-forward frames received from the MPLS network back out other MPLS pseudo-wires. They do NOT run STP. In the rare case when a pseudo-wire is not working, there are some rather interesting (bizarre!) failure modes to this scheme. Space precludes providing more details.

Yes, that's a bit more complex. The whole pseudo-wire mechanism depends on IP routing and MPLS to operate. On the other hand, that means that the SP gets load-balancing with failover, something not readily available when using Spanning Tree Protocol. See the Cisco technical document below, which makes a strong case in favor of point-to-point links between routers (the design approach I happen to prefer).

Concerning multi-point Ethernet over MPLS, it is potentially a rather desirable service capability.  However, it reminds a number of people (including myself) of ATM LANE (LAN Emulation). Most people regarded LANE as a giant kluge, an accident waiting to happen, and an ugly thing to troubleshoot. Multi-point Ethernet over MPLS  is grappling with the same set of issues. The Metro Ethernet Forum has ruled out a mechanism like LANE-ARP so far (as in "who's got this MAC behind them"), but it is not clear that the alternative is a better approach. In addition, the edge device has to handle replication of flooded frames onto pseudo-wires efficiently.

So in all fairness I have to say there's considerable additional business risk to multi-point MPLS-based Ethernet. This is new stuff, and failure modes and stability issues have yet to be field-tested. In short, "let the other guy be the guinea pig" may apply to MPLS multi-point Ethernet.

In addition, for any form of MPLS-based Ethernet service, the same concern as with multi-point switch-based Ethernet VPN's applies: If one customer of Ethernet VPN is having a bad day, can that spill over and affect others?

Buyer Requirements

Along with QoS you need to consider speed and cost. There can be trade-offs here. But if your VoIP is to share links with data traffic, you need appropriate robustness, QoS, and failover mechanisms. Saving money and having "bad voice days" is probably not career-enhancing. Having to use Policy Based Routing to send VoIP over ATM links and data over METN links is feasible (we have a customer doing it), but increases management hassle and cost. (Is this a de-converged network? Anti-convergence?) And should the ATM fail, kicking the VoIP traffic onto the METN link, absence of QoS could hurt VoIP quality. So does the money you're saving pay for the PBR or other drawbacks, assuming you get METN without QoS.

Another consideration is transparency. Will your BPDU's be passed through the SP network? What about CDP? Real L2 transparency means L2 multicasts pass through the carrier devices. Not having this could range from minor irritant to major problem. What are your needs in this regard?

Then there's stability and oversubscription and inter-customer protections. No matter what the underlying technology, I'd like to understand how the SP designed their network, what they did to address stability, what they did to address customers with STP loops and large broadcast storms, those sorts of issues. If the METN is based on Ethernet switches, how many switches in the STP domain? Are they running vanilla STP or RSTP? Per-VLAN STP or Mono STP ("802.1q standard compliant"), or Multiple Instances of Spanning Tree? What procedural mechanisms do they have in place to protect your service from another customer? What BER or packet loss rate does the SP consider "normal"? (Standards are tightening, but some SP's haven't noticed). How many outages have they had, and what was the Time To Repair for each? Root cause? (Lots of luck getting that data!) What network management tools do they use to manage the METN?

A final consideration is availability and cost of local fiber to the METN PoP. This can be a costly show-stopper. On the other hand, if you've got the right kind of fiber to the right urban location, you've gone a good part of the way towards being able to lease dark fiber or access METN services. Is that a budgeting thought for those who see an upcoming need for METN?

Metro Ethernet and Storage Area Networks

The reason I'm rambling on about SAN is that we're seeing increasing mirroring to DR sites, so far primarily FC variants over DWDM. I expect to see this shift to routed IP traffic over time, using Metro Ethernet links for cheaper high-speed  links.

Minor sidetrack: I'm not seeing network designs for this from the vendors yet. From the QoS point of view, SAN requires very high throughput, absolutely priority and low latency, and very low drop rate. That's the worst case. I'm left with the feeling that even with IP routed SAN traffic, I want to dedicate links to it. I would use PBR (Policy Based Routing) and separate subnets to make that easier.  But doing that seems to be another case of "anti-convergence". How do you see this? What are you doing or planning to do about this? If you have good references (URL's) on this, please let me know and I'll pass them on.

Vendors

For more details and a table of fresh information at the end (as of 7/4/2005), see also the article

Summary

Below, I give the link for the main Cisco METN-SP page, and the major links off it. Curiously, these somewhat correspond to our topics and headings above. For what it's worth, all the vendors seem to be starting to talk about "triple play" (data / voice / video). This usually then segues into a discussion of why their product family is the best way to deliver such services. 

The following also gives the Amazon link for the current Cisco Press Metro Ethernet offering. This book strikes me as being a bit thin. It also has an optical bias, and tails off into rather general theoretical discussion, GMPLS etc. So the Cisco web site is currently reflecting new products and marketing emphasis, and is a somewhat fresher source of information. 

There is also a book by the Dan and Emma Minoli and others titled "Ethernet-Based Metro Area Networks". From the Amazon page, it appears to go into some policy, business, and legal issues, and also contain some SONET content. I won't give a link for this METN book since I don't wish to appear to be recommending it.

The technical marketing document link below is included since it has very good diagrams about frame headers and so on, useful for understanding the underlying technology.

Your comments, questions, and suggestions for future articles are of course welcome! See below to decipher my email address.

Dr. Peter J. Welcher (CCIE #1773, CCSI #94014, CCIP) is a Senior Consultant with Chesapeake NetCraftsmen. NetCraftsmen is a high-end consulting firm and Cisco Premier Partner dedicated to quality consulting and knowledge transfer. NetCraftsmen has ten CCIE's, with expertise including large network high-availability routing/switching and design, VoIP, QoS, MPLS, IPSec VPN, wireless LAN and bridging,  network management, security, IP multicast, and other areas. See http://www.netcraftsmen.net for more information about NetCraftsmen. Pete's links start at http://www.netcraftsmen.net/welcher . New articles will be posted under the Articles link. Questions, suggestions for articles, etc. can be sent to pjw <at> netcraftsmen <dot> net (formatted this way to fool email harvesting software).

7/8/2005
Copyright (C)  2005  Peter J. Welcher