| Configuring the Customer Side of an MPLS VPN WAN, Part 1 |
|
I've been noticing how the requests for MPLS WAN related consulting work has evolved over the years:
I haven't noticed much on the web about how to configure for these sorts of situations. I just googled as a check, and found a couple of sites, plus links to prior writings of mine. The lack of discussion is a bit surprising to me, since the routing can get a little more complicated than your average internal routing does. I've been aware of that since around 2003, when I started consulting on this. (And in presenting at a Cisco Powered Network, got the impression that MPLS Providers seemed to be all thinking that they'd be your only WAN connection, and weren't prepared for some of the issues discussed below.) MPLS WAN ScenariosI see this as three basic WAN settings or scenarios, each requiring mildly different approaches to configuration, and each entailing different important considerations. The settings are:
The last situation, differentially routing certain traffic over the MPLS WAN, with data over the legacy WAN, I see as QoS-related: how do I route traffic based on QoS needs or based on DSCP bit settings? If you have IPT or IPVC or video, you probably do want an MPLS service supporting QoS via DSCP bits (not per-VLAN QoS, which I think is pretty useless). But that's Yet Another Topic, more properly a QoS topic, and not much different than regular QoS. You just have to match the carrier's DSCP bit preferences (if non-negotiable), etc., at the WAN edge. This blog will cover the legacy + MPLS scenario, without or with the differential QoS over MPLS requirement. Others to be written will take a look at the other scenarios. MPLS WAN: The Big PictureIf your area has fiber available (major league cities, etc.), you may find that Ethernet access to L2 or L3 MPLS VPN or to L2 VPLS or Ethernet Private Line (EPL) service is available and fairly cost effective, while providing flexibility as far as ability to rapidly add bandwidth as needed. In a recent set of preliminary quotes from vendors, we saw that there's a bit of a fee for the fiber access, but the incremental costs as you scale up bandwidth are low. If your network is in the T1 range and you're looking at going above that, the alternatives are pretty limited (IMA and ATM has hardware costs and generally only provides enough bandwidth for 1-2 years of use), so cost-effective Ethernet services with 1 or 5 Mbps increments is good news! I'm starting to see more any more of a shift to the Ethernet-based access, which is simple and very useful. It does have the problem of "some sites left behind" -- more rural sites may only be able to get T1 or costly T3 access still, while others are operating at 10, 100, or even 1 Gig speeds. This tends to result in apps that require more bandwidth and don't work so well at the slower sites. The big picture here is that MPLS VPN is great, it can be very cost-effective. Both L3 and L2 MPLS VPN have quirks that make them a bit different than more traditional WAN designs. Nothing terrible, but factors that do have to be taken into account. For L3 MPLS VPN,the challenge is that your Customer Edge (CE) router exchanges routes with the Provider Edge (PE) router, and that's just the way it works. Most providers seem to now grudgingly offer EIGRP, often telling sites you'll be the first customer on it, or back-hauled to a POP with a Cisco MPLS edge router at higher latency. That full disclosure is good and honest, but does seem to have the (intended?) effect of driving customers to NOT using EIGRP for the CE-PE routing. L2 MPLS VPN is attractive to those who want to do their own routing. I'll note in passing that this is slighly illusory, in that routing in the MPLS core still underlies the L2 connectivity. On the other hand, the carrier should be doing some mix of SONET, MPLS Traffic Engineering (TE) Fast Re-route (FRR), fast core IP re-routing based on limited prefixes, tuned timers, etc. And it's all just up / down from your perspective. The bigger gotcha with L2 MPLS VPN or VPLS is that it makes the "WAN cloud" act like a giant Ethernet switch. Routing is not really intended for 25 or more routers on a LAN, let alone 100's. You can do that, but it's not a good design. Legacy + MPLSLet's get technical now, and talk about my first scenario, Legacy WAN plus Layer 3 MPLS VPN for WAN. Generally the legacy WAN routers are talking EIGRP or OSPF to each other, an IGP (Interior Gateway Protocol). If your MPLS carrier provides EIGRP or OSPF handoff in the right form, that too will appear to be IGP routing and life is good (and simple to manage). Cisco has added features to carrier MPLS allowing the carrier MBGP to carry EIGRP or OSPF routing metrics and re-advertise them back to your CE router as internal routes. There are minor quirks to consider and work with your carrier on if you have other connections ("backdoor connections") between your MPLS sites. The main thing that go wrong with the above is for the carrier to mess up on the EIGRP AS matching, at one or more sites. That has the effect of making the routes learned on the MPLS side external EIGRP routes. EIGRP much prefers internal routes (administrative distance). So if that happens, your routing will be steering traffic over the legacy WAN rather than the new MPLS WAN. The other way routing can be designed is to talk EBGP to the carrier. (I'm ignoring static routes, which are useless for dynamic re-routing when you have two connections -- and all my scenarios have two connections.) If your CE-PE routing on the MPLS WAN is EBGP, there are two sub-cases. Sub-Case 1: If you only have one router (e.g. small remote office), then it's talking EIGRP or OSPF on one side, EBGP on the other. Due to administrative distance, EBGP wins, so the router will prefer the MPLS WAN for any prefix learned on that side. That's usually what you want, so lucky you, things work as you'd want!
Sub-Case 2: You have two routers, one (usually the older one) talking to the legacy WAN, the other (sometimes shiny new) talking to the new L3 MPLS WAN. EIGRP for legacy, EBGP for MPLS.
How do the two routers exchange routes? Generally people bi-directionally redistribute the EBGP into EIGRP or OSPF. (Some details follow below.) If you do that, the MPLS router has an IGP and EBGP as route sources, and EBGP wins. So if it receives traffic, it will generally prefer the MPLS WAN. That's good. On the other hand, the legacy router has internal IGP routes plus the external IGP routes learned from the MPLS router's redistribution. For an IGP, internal is better than external, so the legacy router "wants" to send traffic over the legacy WAN. Not so good. So will any other routers or L3 switches at the site. Definitely not so good.
This is why I like doing EIGRP or OSPF as the CE-PE protocol, matching your IGP. It keeps things simpler. From a business perspective, I strongly recommend that the CE-PE routing protocol handoff be a major vendor selection criterion. What can you do about this? If you have no other L3 devices at the site, you can make the MPLS router the HSRP / VRRP primary, with WAN interface tracking, or even whether an important route, like default, is successfully being learned from the MPLS peer. That way, devices' traffic will go to the MPLS router, it prefers the MPLS link, and life is good. If some site falls out of the MPLS cloud (so to speak), the prefix is only learned on the legacy side, so the MPLS router will redirect traffic to that site to the legacy router, which in turn will forward it out the legacy link. Ah, but if you have other L3 devices at the site, then things are a bit more ... interesting. Some ideas:
Adding QoS Preferential Routing to the MixActually, this case is simpler. If you only have one site router, you can just do metric offsets or Policy-Based Routing (PBR) on it. We've done this sort of thing for PBX to PBX IPT traffic in a non-MPLS setting, where you make the IGP metric look better on the side with QoS for traffic headed to a PBX subnet (ATM in one case). More recently, I'm looking at settings with IPT using Microsoft Office Communicator and soft-phones, or soft IPVC clients. In that case, I lean more towards dealing with such device by somehow marking at the edge or trusting DSCP markings. And then you can do DSCP-based PBR on the site legacy router, to "deflect" such traffic to the MPLS router. If you track objects (MPLS neighbor and perhaps a route), you can not deflect traffic via PBR if the MPLS side is down. (Admittedly, a topic worthy of more discussion in and of itself.) If you have two routers at the site, things get more complex. If you only want selected traffic using the MPLS side, then don't redistribute the EBGP into the site IGP on the MPLS router. Do still run the IGP on the MPLS router. If you have default routing advertised on the legacy side, then traffic (one way or another) will go to the legacy router, and you can use PBR as above to "deflect" certain traffic to the MPLS router. If the router doesn't have a route to the destination but the legacy router does, the MPLS router will send it back. You can avoid a routing loop if the preferred path back is different than the one you're doing PBR on. A dedicated router to router link rather than one via the site-interface is one way to do that. You may have to use point-to-point interfaces to the L3 site switch(es), rather than a VLAN, to avoid having the two routers peer with each other over a VLAN they also communicate with the rest of the site or site switch(es) over. Tracking the MPLS router can disable the PBR if the MPLS router goes down.
Note: Loss of default over the legacy WAN will mess this up. You could have the legacy WAN router advertise default for the site, being careful to NOT advertise default into the legacy WAN. An alternative is to have the MPLS router redistribute prefixes into the IGP. Since they're external, the legacy router will be preferred unless a prefix stops being advertised on the legacy side. In that case, all traffic for that site will start being routed to the MPLS side. You might want an access list to prevent data traffic from using MPLS, Or a QoS policy that protects your IPT / IPVC, and gives remaining bandwidth for limited data use. Sites Becoming TransitWe haven't really talked about this yet, but sites tend to redistribute the site IGP into EBGP to get the site prefixes to be known across the MPLS WAN. The alternative is to configure a static site summary route and carefully redistribute that and only that static route into the EBGP. The site IGP redistribution approach amounts to bi-directional redistribution. This can be a receipe for routing loops, and is generally best done with some distribute lists and filtering. The problem becomes, configuring an access list or prefix list to only advertise out the site prefixes is messy, making the static summary redistribution above look simpler. In general, I don't like having to do site-specific filters, it creates extra operational work and can be a source of problems. When you have two WAN connections, conceivably an outage might lead to a remote site (probably with limited access bandwidth) becoming transit, as in traffic going via one WAN to the site to get to a site on the other WAN. Routing metrics generally should take care of this. If you're going IGP into MPLS redistribution with too attractive a metric, the routing metrics may not work in your favor. This is the common problem when doing bi-directional redistribution. One form of insurance against this is to tag routes being learned at a site or redistributed into the IGP. You can then use the EIGRP or OSPF tags to prevent such remote routes from being advertised back into the WAN, so that only the site prefixes are advertised. This does take consistent configuration. The good news is that the route maps and distribute lists needed can be made site-neutral, i.e. part of a cookie-cutter configuration template. Coming TopicsA lot of the above discussion carries over the the other scenarios above, with some note-worthy differences in how it affects you. We'll save specific discussion for the follow-on blog(s) about this. ReferencesIvan Pepelnjak has a great site with lots of information, and I have huge respect for the depth of his knowledge of routing. He's got a number of good and relevant articles, see for instance http://blog.ioshints.info/2008/12/basic-mpls-vpn-design-configuration.html. I come at this a little differently, with less focus on the MPLS provider side, and more on simplicity for the customer side (to the extent possible). I like the validation of seeing some of his discussions of customer side factors (MPLS security, VPLS service, etc.) -- although I slightly disagree with the emphasis of the MPLS security article. MPLS Security takes a little more work by the provider than FR or ATM perhaps -- but should be attainable. (I worry about sites unnecessarily doing IPsec with all its burdens, somewhat defeating the value of high speed MPLS service.) I love the title "VPLS is not Aspirin". Cisco has a good posting about buying MPLS services, the Enterprise Consumer Guide, at http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/L3VPNCon.html. It's a good source of questions to ask vendors about in an RFP, although I recently thought of a number of things I wanted to ask that might not have been in there. (And there was an amazing first in dealing with WAN provider discussions: we actually got some darn good answers, and I hereby award the prize for best technical answers in a customer document to Level 3.) There is a longer (more detailed?) version available as a book from Cisco Press. See http://www.amazon.com/Selecting-MPLS-Services-Chris-Lewis/dp/1587051915. Prior Related ArticlesI seem to have written a lot over the years about MPLS VPN. Recently, not so much. There are a bunch of aging but often still germane articles in my archive. Here are the archived articles that are most relevant to the topic of this blog:
Comments (8)
  written by Pete Welcher, March 31, 2010
The command to use is "distance ... neighbor ... ". Sorry if my word ordering mislead you in where to look. Some references:
https://supportforums.cisco.com/thread/2010047 http://www.cisco.com/en/US/doc...i_pi1.html http://www.cisco.com/en/US/doc...#wp1045238
 written by Wes Tribble, April 01, 2010
So it looks like you can only set it globally or per route for EIGRP. I don't see a way to do it per neighbor like in BGP. Thanks for your help.
written by Pete Welcher, April 01, 2010 Yes, that last URL is the 15.0 Command Reference, there is no document distance ... neighbor for EIGRP, just for BGP. So you can try altering the external Admin Distance within a site, but that could backfire if you have redistributed statics, etc. (Which is I'd prefer to be able to do it per-neighbor, if that were possible.) I don't see a clean simple solution for the EIGRP / BGP mix. The more alternatives I try, the uglier it looks. My answer would be for carriers to hand off EIGRP to customers. Verizon is moving in the opposite direction apparently -- terminating use of Cisco PE routers (Juniper edge, Cisco core). Readers: is there a solution I've overlooked?
written by Wes Tribble, April 01, 2010 I have this labbed up and this is what I have found so far. I tried "distance 20 172.29.1.20 0.0.0.0", and it only effects the internal EIGRP routes learned from 172.29.1.20. It DOES NOT change the AD of external routes learned from 172.29.1.20. The only thing that seems to work is "distance eigrp 90 20" command. However, I am concerned about the implications of changing the AD of all external routes.
written by Pete Welcher, April 01, 2010
That was quick! Thanks for the info.
That sounds like what I recall that I ran into in labbing this combination several years ago. Plus the AD change is purely local, so in principle you have to set it on every site router / L3 switch, if I'm recalling correctly. (AD is certainly not propagated via routing updates.) Having said, that, if all you have is one or two switches front-ending the MPLS BGP WAN and other EIGRP WAN router, they're the ones that really matter. So if MPLS is the preferred path, if you get packets to the MPLS WAN router, you're ok. If the MPLS doesn't have a route, it can hand off to the other router. That suggests policy-based routing towards the MPLS WAN router with CDP check (verify-next-hop) (slow convergence), or alternatives along that line. I'm doing stuff like that now to deflect voice/IPVC to MPLS with data going via the legacy side, but to do it for all your traffic seems a bit of a kluge. And if you want CEF-based load-sharing between two providers, I definitely think you want dual EIGRP or dual BGP. Because you can't get the admin distance problem solved to make the routes comparable between two routing protocols (one redistributed) -- or I sure couldn't find a way, when labbing that up not so long ago.
written by Wes Tribble, April 07, 2010
I have been so focused on lowering the AD of the external routes that I overlooked the simple solution. I CAN raise the AD of the internal routes learned from the VPN router. As long as I make routes learned from the VPN router a higher AD than the external routes learned from the MPLS router, all is golden.
So the fix was "distance 180 172.29.1.21 0.0.0.0". This allows me to prefer external routes from the MPLS router without prefering all external routes.
written by Pete Welcher, April 14, 2010
We did a couple of rounds of followup via email. My first reply was: "Yes, you can do that, it has similar risks of looping. So care is needed. But it's also AD for *all* internal routes on the router you configure it on (and no others)."
Wes then asked about an ACL for VPN routes to change. I replied: "You're probably ok for isolated sites. It's where two MPLS sites have a LAN link on the "backside" you might have some interesting ... loops? That is, where there are cases where you DO want to take the internal route. This leads to you probably want a tag or prefix-list filter on the MPLS side, so that each site only advertises the local prefixes. That way, a site won't learn internal routes to itself via MPLS... I'm leaning more and more towards wanting to only advertise local routes out to the IPsec side (or whatever WAN #1 is), ditto the MPLS side, advertise default from the data center(s) to cover corner cases, and you shouldn't have problems. Doing EIGRP tagging or OSPF tagging (route comes in, tag it as "external / other location") can help. Just ran into 6500 IOS bug where it loses internal EIGRP tags though. Haven't yet tested more recent IOS code." In working in the lab for a consulting customer (EIGRP over TLS, MPLS w/ QoS for voice using EBGP, plan to use PBR to "deflect" voice to the MPLS router), I've been reminded that EIGRP redist into BGP sets metric (MED) and weight. VZN recommends a route-map to reset the weight to 0. If the BGP to EIGRP redist has a lousy metric, then the presence of metric in BGP seems like it should prevent loops (poor metric + additional hops to the BGP router). However, it also seems like you could get some interesting remote site transit traffic under certain failure modes. A BGP and BGP solution (EBGP on both sides, IBGP at sites) at least sort of automatically flags local vs. remote routes. (We're doing different AS number per site, which does increase the config hassle but provides more troubleshooting info in the long run.) We found that with that and bidirectional redist, the local routing showed some oddness unless we diddled the weights. We switched to network statements rather than EIGRP into BGP to see how that looks. I'm trying to avoid building lists of network statements for local routes. Prefix lists are cleaner in that they can be used for both EIGRP and BGP distribute lists out of the site. We're still looking for the "cleanest, simplest" combination. I'm starting to think dual BGP with weight resetting and a "^$" AS PATH filter may be the simplest, site-prefix-free approach. Whew! I'd looked at this before, but I have a new appreciation for its messiness.
Write comment
|
I've recently been doing a lot of consulting work involving variations on a common theme. The general theme is how do I configure my routers to work with the MPLS VPN service I have bought or am about to buy from Provider X. If the service is to be your sole WAN connection, there's little problem or complexity. Do get the carrier to hand routes off to you in your IGP, unless you happen to want EBGP in your life. If you are going to have two WAN connections, well, things can get more interesting. Which is what this article is about. 
Subscribe to this comment's feed
I can't find this command. This seems like the cleanest way to do things, but I can only change the administrative distance per AS. I don't see a way to do it per neighbor.