Introduction

OER links in last month's article: http://www.netcraftsmen.net/welcher/papers/oer01.html#links.

Lab Topology

This was discussed at length in last month's article. As a memory refresh, here are the high points:

• I configured OER on only one router, matching one of the expected ways in which OER might be used.
  
• My two OER router outbound interfaces were certain vlan subinterfaces on two trunks. One went to the LAN connecting my lab exit points to my home Linksys router and the Internet. The other went to another lab router (2600) with a connection to the home LAN. 
• See the following diagram.

Other lab info: the lab routers both have static default routes pointing at the Linksys' inside interface (192.168.1.1). These are redistributed into EIGRP. Both do PAT of inside lab addresses to the address of their external interface (192.168.3 and 4, respectively).  The Linksys does PAT again, to whatever address it picked up from Comcast (my ISP) via DHCP.  The lab 2600 acts as DHCP server for the inside lab network.

I can wirelessly connect to the Linksys or to a WAP inside the lab network, so I made sure I was connected to the lab network (with appropriate static routes on my PC). I also made sure my PC was set with only default gateway being the OER router, since normally my lab DHCP supplies both router inside addresses as default gateways to the PC.

Testing OER

I tested OER failover using several web browser windows to observe connectivity or non-connectivity. The web sites used: www.netcraftsmen.net, www.cnn.com, www.wjla.com, and www.abc.com. Nslookup results for these:

To cause a failure, I disallowed the external VLAN on the trunk from the lab switch to the OER router. This caused packet black-holing while leaving the interface up. Since I was using static default routing, there was no way for normal routing to discover the outage. OER to the rescue!

Normal OER with two routers uses redistribution of the OER static routes. I had to deliberately NOT do that in this lab, to avoid a routing loop when I induced VLAN failure as described above.

Test Results

After I induced the packet black holing, the web pages would not refresh for a while, until OER re-routed the outbound traffic. I eventually saw most OER prefixes shift correctly to the other interface, the one that still worked. This was using passive monitoring, via NetFlow data.

Some prefixes did not shift. I believe this was due to changes of host addresses, for example with www.cnn.com. These sites do DNS load balancing, so that at different times the hostname resolves to different addresses.  In particular, CNN appears to be using several /24 blocks, so when DNS name resolution switched to a new block, OER would have to go through its multi-minute learning and holddown cycle all over again. Specifying this prefix or an aggregate CNN prefix manually would presumably have helped with this. I did not test this, as time has been a bit tight for me lately.

Configuring OER

• Documented: VLAN interfaces can only be internal.
• Live labwork (and somewhat obvious): If you're using learned prefixes, you need some live TCP traffic. No traffic, no prefixes, no learning, then no OER activity.
  

Relevant portions of the configuration from my lab OER router:

Show Output

Here is sample output from the "show oer master prefix" command. This is from a different time than the above capture, so they won't match up. I edited the table to cut down its size by about 10 rows.

Here are the key fields in the above show command output:

The "Pas" or "Act" indicates passive or active measurements. The table uses "S" for short-term, "L" for long-term measurements.

Where you see Fa0/0.77 above, the prefix is being routed via the alternative path. Where you see Fa0/1.192, the traffic is being routed directly to the exit router at 192.168.1.1 (which was down at this point). HOLDDOWN is indicating that a new prefix has been learned and the current next hop is being tested for policy compliance. INPOLICY indicates that the route is in policy compliance. When the next hop for INPOLICY is a interface that is a known problem (like Fa0/1.192), this presumably indicates that the alternative route monitoring indicated similar failure rates.

Note that when the next hop is via Fa0/1.192, there are usually many  short-term (S) and long-term (L) unreachables, visible as big numbers four columns over (in the second row in the block of 3 rows for that prefix). I haven't found a great reference on interpreting this show command output. The command reference is enough to get you started.

Here is a sample of output from the "show oer master prefix detail" command, edited for brevity to show part of one prefix block:

rtr1841#show oer master pre det
... <snip> ...
Prefix: 63.208.176.0/24
   State: HOLDDOWN    Time Remaining: 137    
   Policy: Default

   Most recent data per exit
   Border          Interface         PasSDly  PasLDly  ActSDly  ActLDly
  *10.20.254.15    Fa0/0.77               31       31        0        0
   10.20.254.15    Fa0/1.192               0        0        0        0

   Latest Active Stats on Current Exit:
   Type     Target          TPort Attem Comps    DSum     Min     Max     Dly

Prefix performance history records
 Current index 8, S_avg interval(min) 5, L_avg interval(min) 60

Age       Border          Interface       OOP/RteChg Reasons                 
Pas: DSum  Samples  DAvg  PktLoss  Unreach   Ebytes   Ibytes     Pkts    Flows
Act: Dsum Attempts  DAvg    Comps  Unreach
00:00:10  10.20.254.15    Fa0/0.77                                           
       60        2    30        0        0    20313        0       48        2
        0        0     0        0        0
00:01:29  10.20.254.15    Fa0/0.77        Unreachable/Monitor                
       64        2    32        0        0    20439        0       54        8
        0        0     0        0        0
00:03:03  10.20.254.15    Fa0/1.192                                          
        0        0     0        0        1      144        0        3        1
        0        0     0        0        0
--------------------------------------------------------------------------------
... <snip> ...

The above sample is from passive monitoring. Here is one from active monitoring:

Note the differences: data under "latest active stats" plus numbers under the "Act" rather than the "Pas" columns. 

Fancier Variations on OER

OER works by re-routing prefixes to use an alternative exit. But OER also supports link policies, which can be used for load balancing. With a link policy, OER gradually uses prefix re-routing to shift traffic off a link until the link conforms with the configured policy.

OER Application Aware Routing is an enhancement introduced in Cisco IOS Release 12.4(2) T. It allows very fine-grained control based on some rather than all traffic to a prefix. The policy is applied using an extended access list, so among other things it can take into account:

• Source and/or destination IP
• Soure and/or destination port number
• DSCP, IP Precedence, or TOS value
• Protocol flags
• Protocol type  and number

As I read the documentation, OER AAR is doing two things:

1. Conducting per-prefix measurements filtered by the extended ACL
2. Policy-based routing based on the results of such monitoring

Another recent OER feature is GRE over IPsec tunnel optimization, which is supported as of 12.3(11) T.  The GRE tunnel interface is configured as an external interface. You can then use OER to select between using a WAN interface and the VPN tunnel interface.  Or between two tunnel interfaces (e.g. with dual Internet providers).  You can filter prefixes based on the VPN site prefixes, so as to optimize VPN traffic to remote VPN sites.

Dead Box / Link Detection

But more than that, such devices often only react to local outages. So if a connecting box starts behaving oddly, but the link stays up, they do not fail over. If you start getting an asymmetric flow, they probably do not react (example: "zombie switch" where the link is up, and packets are black-holed in one direction, with no return traffic). Some boxes (for example Cisco content switches) allow you to configure monitors to be more aware of outages or odd conditions. Configuring this on a box by box basis can be tricky.  Doing the failure analysis to make sure you've covered the various failure modes can get quite complex. ("Takes a CCIE to build, maintain, and troubleshoot" can be an indication that something is overly complex.)

I find myself wondering whether inbound OER (from Internet routers or their neighborhood towards servers) might help with such situations.  It would require a different design: having alternative "stovepipes" of firewall/switch/content switch, with OER in effect selecting between stovepipes. This is contrary to current design, which tends to do things like HSRP wherever possible. I think of HSRP and most failover mechanisms as providing single box alternatives.  My concern with HSRP and similar approaches is that I've seen or heard anecdotes that indicate that boxes do not fail over when they "should". My thought is that treating a "stovepipe" as a single unit that either passes traffic bi-directionally or does not might provide simpler failover.

On the other hand, I'm not at all sure using OER in this fashion would be a good idea.  Among the drawbacks: much slower failover.  And some additional complexity. Perhaps OER  might be better applied for choosing between two alternative "inbound complexes" of devices, where each complex has box redundancy within it. I've seen this sort of design at e-commerce sites with multiple Internet connections.

Summary

Please let me know if you're using OER and are impressed by it (positively OR negatively). I'd also be interested in hearing about non-standard OER setups, where you're using it "creatively".

The documentation has some brief comments about OER and multicast traffic.  Due to the multicast RPF check, some interesting things might happen to multicast in the presence of OER. On the other hand, how likely are we to be using multicast over the Internet?

OER links in last month's article: http://www.netcraftsmen.net/welcher/papers/oer01.html#links.

As often is the case, I have some questions that don't seem to be answered by the documentation. If you happen to be testing OER or quizzing your Cisco SE and find the answers, please let me know! Unanswered questions:

1. If you have two routers, when OER stuff the static route for a prefix, does it only do it on the exit Border Router, or does it do it on both? (Conjecture: probably just on the exit Border Router, so that redistribution "pulls" packets for that prefix to the correct BR).
2. How can one provide for master controller redundancy or failover? (If you think about it, two active "boss (master controller) routers" stuffing  static routes into one BR would be a Bad Thing).
   
3. What is the full set of issues in multicast-OER co-existence? 

Your comments, questions, and suggestions for future articles are of course welcome! See below to decipher Pete's email address.

Dr. Peter J. Welcher (CCIE #1773, CCSI #94014, CCIP) is a Senior Consultant with Chesapeake NetCraftsmen. NetCraftsmen is a high-end consulting firm and Cisco Premier Partner dedicated to quality consulting and knowledge transfer. NetCraftsmen has ten CCIE's, with expertise including large network high-availability routing/switching and design, VoIP, QoS, MPLS, IPSec VPN, wireless LAN and bridging,  network management, security, IP multicast, and other areas. See http://www.netcraftsmen.net for more information about NetCraftsmen. Pete's links start at http://www.netcraftsmen.net/welcher . New articles will be posted under the Articles link. Questions, suggestions for articles, etc. can be sent to pjw <at> netcraftsmen <dot> net (formatted this way to fool email harvesting software).

2/7/2006
Copyright (C)  2006  Peter J. Welcher