BGP and MPLS-Based VPNs
Two months ago we started looking at MPLS, Multiprotocol Label Switching, and continued with MPLS last month. This technology is very hot among large Enterprises and Service Providers. If you missed them or want an electronic copy of them, the previous articles can be found at:
This month we complete the MPLS series by looking at MPLS (and BGP) -based VPNs.
References for more information about this:
this is all fairly new stuff, I do not have equipment available to test it with (nor time), and am piecing together information from various sources. Thus the configurations are my best effort but are not guaranteed accurate.
What is a VPN?
A Virtual Private Network or VPN is a network implemented using a shared network infrastructure but so as to provide the security and privacy of a private leased-line network. Older examples would be Frame Relay and ATM. Lately VPN has come to more often refer to IPSec tunnels over the Internet, or perhaps PPTP or L2TP dial VPN connectivity across a shared internetwork.
For our purposes in this article, the VPNs will be IP networks where the WAN core of a corporate network has been outsourced to a Service Provider. The IP VPN connectivity is provided across a shared IP network belonging to the Service Provider. It will turn out the the BGP and MPLS-based VPNs we will talk about are powerful enough to provide secure connectivity (and relatively simple configuration) for both intranets and extranets.
- Intranet -- VPN interconnecting corporate sites
- Extranet -- VPN connecting corporate site or sites to external business partners or suppliers. The Internet is the ultimate insecure Extranet VPN.
- Customer Edge (CE) router -- a router at a customer site that connects to the Service Provider (via one or more Provider Edge routers)
- Provider Edge (PE) router -- a router in the Service Provider network to which Customer Edge Routers connect
- Provider Core (Core) router -- a router in the Service Provider network interconnecting Provider Edge routers but, generally, not itself a Provider Edge Router
- Entry and Exit PE routers -- the PE routers by which a packet enters and exits the Service Provider network
In the figure, imagine the red routers are connected with one VPN, and the blue ones with another. (I tried to draw in some lines to suggest connectivity, but things rapidly got rather cluttered). An extranet is where some red routers connect to some blue routers. The red path with arrow shows traffic from the bottom red CE router to the top one. The first (bottom) gray provider router is the entry PE router, and the final gray provider router is the exit PE router (terms used below).
Understanding MPLS-Based VPNs
I've been thinking of MPLS-based VPNs as basically using long IP addresses. That isn't exactly what's going on, but it is a key part of it.
Each site belongs to a VPN, which has a number. In the Cisco implementation, this number is configured as the 8 byte Route Distinguisher (RD). The route distinguisher number is used to prefix the IP addresses for the site. It is configured on the interface (or subinterface) connecting to the site. This gives us a way to tell duplicate private addresses apart, to distinguish them. For example, subnet 10.1.1.0 for VPN 23 is different than subnet 10.1.1.0 for VPN 109: from the MPLS VPN provider's point of view they are really 23:10.1.1.0 and 109:10.1.1.0, which are quite different. Putting the 8 byte route distinguisher in front of a 4 byte IP address gives us a 12 byte routing prefix. We regard these as the VPN-IPv4 family of addresses.
The multiprotocol extension to BGP4, MBGP, was invented to carry such routing information between peer routers. So once we think in terms of routing 12 byte prefixes, there is a natural way to propagate the information. For security and scalability, MBGP only propagates information about a VPN to other routers that have interfaces with the same route distinguisher value. That reduces the chance of accidentally leaking information about Customer A to Customer B (quite easily done with routing distribute lists in a tunneling approach, or with route maps or distribute lists or prefix lists and ordinary BGP). It also means that each PE router only tracks routes for the customers connected to that one PE router, not for the entire set of long prefixes for all sites and customers connected to the Service Provider. Scalability!
Another aspect of this is that core routers, not being connected to CE routers, don't learn VPN-IPv4 routes. We'll come back to this idea in a moment. This is desirable: it turns out we only need to run an IGP (Internal Gateway Protocol), so that core routers have routes to all PE routers. And from our prior discussions about MPLS, we suspect the IGP might be OSPF or IS-IS, to allow implementation of MPLS Traffic Engineering. Only tracking routes to PE routers keeps the core extremely scalable, and greatly simplifies the size of routing tables for core routers. This too enhances scalability!
So what we've got so far is long addresses, and tracking routing that builds in the VPN ID or route distinguisher as part of the routing prefix. The PE routers that share the long prefix routing information are all speaking MBGP, all within the same AS -- hence internal MBPG, or iMBGP. This behaves very much like ordinary BGP. Well, when iBGP speaking routers propagate routes, they also propagate attributes. One key attribute for Service Providers is the next hop attribute. For iBGP-speaking routers, the next hop is generally the exit point from the Service Provider network, the exit point used to reach the advertised destination prefix.
If we were to actually route based on the long addresses, we'd have to forward the packets hop by hop and do a routing lookup at each PE or core router between the entry PE router and the exit PE router. The problem with that is, we would then have to convert our IP header to use our longer addresses at the entry PE router, we'd have to have internal core routers that knew how to forward this new network-layer protocol, and then we'd have to strip out the longer addressing information at the exit PE router. This probably sounds sort of like what MPLS already does with labels -- but now we'd be doing it with actual network layer headers. Some readers might be thinking "aha! IPv6! Tunneling IPv4!". Nice thoughts, but ... WRONG!
I suppose the network layer code could have been written to support this, or IPv6 could have been used for a form of tunneling. But all of that would have cost time and work and money. Instead, the Cisco engineers who came up with this had a very clever idea. MPLS!
All that the entry PE routers need to do to packets is somehow deliver them to the appropriate exit PE router, the next hop known via the mandatory MBGP next hop attribute. But with MPLS and any IGP carrying routes to the PE routers, we will already have an MPLS Label Switch Path (LSP) from the entry PE to each possible exit PE! And that does it.
When a packet comes in, we look up the long (VPN) destination prefix in the MBGP routing information base (RIB). That tells us the next hop router, the exit PE router. We would normally look up how to get to that router in the IGP, and determine the IP next hop. But this gets short-circuited by MPLS: we find we have a label available for an LSP that delivers packets very efficiently to the MBGP next hop router, the exit PE router. And (here's the clever part) if we use the LSP, the core routers in the core never have to examine IP addresses or headers, they just use the labels to forward the packet!
So MPLS LSPs act as tunnels through the Service Provider core, meaning we can get away with an IGP in the SP core, and thus the SP core routers can remain ignorant of the many, many possible destinations for all subnets in all VPNs.
Route distinguisher 0 and VPN 0 can be regarded as the current Internet.
Note that smart Service Providers might build their AS number into the VPN route distinguisher, as a way to provide uniqueness and allow cooperation in providing MPLS-based VPN services to their customers.
Extended Communities and VRFs
The techniques described so far are enough to build VPNs for a particular SP customer, say Customer A. Suppose the SP is providing VPN services to Customers A and B, and A and B decide they need connectivity between certain sites? The approach above is a little limited. So there is one more piece to this MPLS BGP VPN puzzle. That piece is Extended Communities. This is a long 8 byte version of the 2 byte community attribute already known in BGP.
When the Service Provider connects up a CE router, the route distinguisher is specified on the connecting interface. Routes from the site can be learned by static routing, or dynamic routing exchange with the CE router. (MPLS-speaking CE routers are a special case.) When such IPv4 routes are learned, they are extended using the route distinguisher, so they can be distinguished from the routes from another customer, and so they can be propagated to the other VPN (intranet) sites. This is done by associating the same number with those routes as extended community. The extended community is also called and thought of as target community: it identifies the community of other sites needing routes to this long destination prefix.
To maximize flexibility, a per-site or per-interface routing table is used, the VRF (virtual routing forwarding instance). This is configured by creating it, describing it to the router, and then associating it with one or more interfaces (since the VRF might be shared between corporate sites than connect to the same PE router). We'll see how to do this below.
For an intranet, the VRF contains just the routes from that VPN.
Say we've done all this for Customer A. To connect a Company A site to a business partner B, we import routes for the VPN from B (possibly filtering them, so that we can only route to specified sites within B). So that business partner B can reach Customer A, we also export routes to target community B (or the extended community number for B). We can do this per-location within Customer B's network, providing very fine-grained control over which Customer B sites can reach Customer A. Alternatively, we can use a different VPN ID (route distinguisher and extended community) for the A-B extranet, and then export routes to and import routes from this extranet VPN to the VRF's at the sites that have to communicate with the business partner(s). Note how scalable and extensible this is!
Subinterfaces can be used so that extranet traffic can be forced through a CE firewall or so the CE can filter routes to control what internal sites the extranet partners can get to.
Since the Internet is just RD and extended community 0, the Service Provider can also selectively connect customer sites to the Internet.
The above figure shows some sample VRFs associated with the interfaces on the PE router at the left of the picture. These are suggestive of the situation for the configuration that follows in the next section. The VRF named VRF00001 contains routes to other blue VPN sites (subnets). The VRF named VRF00002 contains red VPN subnets, along with an imported blue VPN subnet. A route map might have been used to provide the fine-grained control over what blue subnets are imported into VRF00002. See below for configuration details.
Configuring MPLS VPNs
This article is getting a bit long, so let's look at a somewhat complete configuration example, instead of tackling the syntax of each of the commands. I've also left out all the MPLS configuration lines, since we've covered those in the previous article(s).
See also http://www.cisco.com/univercd/cc/td/doc/product/software/ios121/121cgcr/switch_c/xcprt4/xcdtagc.htm#xtocid1571049
, which the following configuration is based on.
Suppose as an ISP our AS number is 888. For Customer A, we will create a VRF named vrf00001 and associate it with Route Distinguisher 888:1 (abbreviation for two bytes that are 888 in decimal, followed by six bytes ending in 1). We will also import and export routes to extended community 888:1, namely, other sites in this intranet VPN. For another customer, Customer B, we'll create a VRF named vrf00002 with RD 888:2. This second VRF will import and export extended community 888:2, other sites in Customer B's intranet. However, we'll also import routes from extended community 888:1 accoring to a route map named vrf00002-import-map, so that the site using VRF vrf00002 can reach selected Customer A sites, as extranet partner.
To do all this, configure:
ip vrf vrf00001
route-target both 888:1
ip vrf vrf00002
route-target both 888:2
route-target import 888:1
import map vrf00002-import-map
route-map vrf00002-import-map permit 10
It is important to note that the route map is only needed for fine tuning. Normal import/export with VRFs can just extended communities. The thought of security depending on getting route maps built right rather scares me. Luckily, basic security is provided at the extended community level, making route hiding the normal situation. Then route maps can be used to limit connectivity to extranet partner sites, if the customers don't wish to do that for themselves by speaking BGP to the PE routers.
These VRFs would typically then be associated with interfaces:
interface Fastethernet 0/2
ip vrf forwarding vrf00001
ip address ...
interface Fastethernet 0/3
ip vrf forwarding vrf00002
ip address ...
interface Fastethernet 0/4
ip vrf forwarding vrf00002
ip address ...
VRF vrf00002 is associated with two interfaces that connect to two sites for Customer B. I'm deliberately showing FastEthernet, since some people now think that's how we'll be connecting to SPs in metropolitan settings. (Think BLEC: Building Local Exchange Carrier, providing VPN, Internet, and Voice connectivity).
We need to be speaking MBGP to carry VPN-IPv4 routes and attributes to peer PE routers. We don't need ordinary BGP routes to PE peers however. (On a larger scale, we might use route reflectors vice iMBGP full-mesh peering):
router bgp 888
no synchronization ! don't do IGP synchronization (since
! the IGP won't carry the right routes anyway)
no bgp default ipv4-activate ! don't do ordinary BGP
neighbor 10.60.0.5 remote-as 888 ! identify an iBGP neighbor and AS
neighbor 10.61.0.1 remote-as 888 ! identify another
address-family vpnv4 unicast
neighbor 10.60.0.5 activate ! activate session to some MBGP peer
neighbor 10.61.0.1 activate ! some other MBGP peer
Our design might use eBGP to communicate routes to CE routers in a controlled way, to get routes into each VRF. Or it might use static routing, or some other mix. We can also define per-VRF static routes as shown below.
address-family ipv4 unicast vrf vrf00001
neighbor 10.20.1.1 remote-as 65535 ! private AS number
neighbor 10.20.1.1 activate
address-family ipv4 unicast vrf vrf00002
neighbor 10.20.2.2 remote-as 65535
neighbor 10.20.2.2 activate
ip route vrf vrf00001 22.214.171.124 255.0.0.0 e0/2 10.20.1.1
That's it, a basic MPLS BGP VPN configuration!
Advantages of This Approach
Let's list the advantages briefly, to conserve space. See the above references for details.
- Vast scalability
- Routing provided as a service to customers that don't want to manage their own routing -- configure CE routers with a default route having the PE router as next hop
- No tunnels, instead shared Service Provider IP core network
- IPSec optional (performance boost and cost saving)
- Shared core for SP, not core or tunnels per customer
- Security/ease of configuration (and getting it right)
- Virtual sites (use subinterfaces to connect one site to multiple VPNs)
By the way, using route distinguishers almost lets a Service Provider not have to worry about interoperability with private addressing. However, if you read the fine print, it turns out you cannot interconnect two sets of sites with overlapping addresses. That is, if you have subnet 10.1.1.0 in your VPN, then you can't connect any site in your VPN to another VPN that also contains a subnet 10.1.1.0. The usual ugly quick fix applies: NAT (Network Address Translation). I say "ugly" because you should think of the maintenance and troubleshooting issues for a Service Provider using NAT and BGP/MPLS VPNs to interconnect 10 customers, each with subnet 10.1.1.0.
BGP basics are now covered in the Cisco BSCN course, which has replaced the ACRC course and covers OSPF, EIGRP, and BGP in somewhat more detail than ACRC used to. Mentor Technologies also has longer standalone OSPF and BGP courses that you might find useful in learning more about these protocols, not to mention possible relevance for CCNP and CCIE certification. See our web page for outlines and scheduled locations and dates.
This article concludes our coverage of MPLS and related topics. I may write an article or two explaining IPSec, and I plan a whole series of articles on IP multicast. I'd love to write an article about MBGP, since it is useful for IP multicast, IPv6, and (as we've seen) VPNs. However, there are indications the command line interface is in a state of flux, and so I think we'll wait for command stability before covering that topic.
Your preferences and ideas and suggestions for topics are always more than welcome!
Copyright (C) 2000, Peter J. Welcher