Should you be concerned about a Layer 2 loop in your campus network? My answer is absolutely. In this article, I explain why you should take steps in protecting your campus network. This article also provides practical steps in reducing the effects of a Layer 2 loop on a Cisco Catalyst 6500. Before delving into details, let’s first define what a Layer 2 loop is. A Layer 2 loop occurs in a campus network when more than one Layer 2 forwarding path exists between two given switches. In this scenario, a switch that receives a broadcast frame sends it to all its trunk ports and access ports (same VLAN). In the presence of a loop, when campus switches forward broadcast frames to all their ports, this creates an amplification phenomenon for broadcast frames trapped indefinitely within the loop. This phenomenon is also known as a broadcast storm. It leads to an exhaustion of bandwidth, and CPU overutilization due to the presence of large volumes of broadcast frames. A broadcast storm brings a network to an unusable state, and in certain cases network administrators may lose the capability to access devices by console.

The Spanning Tree Protocol (STP) was designed to ensure loop-free Layer 2 topologies. Despite the use of STP, some situations can create Layer 2 loops such as wiring mistakes, misconfigured hosts (bridged interfaces), switch configuration mistakes, and loss of BPDU keepalives.

You may have a good campus network design, but human errors are still possible. It is critical to look beyond what can be prevented, and ask yourself the following question. If a Layer 2 loop were to occur, would my campus network be able to sustain it? If the answer to this question is no, or if you are not sure, I encourage you to explore a suggested solution presented in this article to mitigate the impact of a Layer 2 loop.

I got involved in exploring solutions to protect the Catalyst 6500 because one of our customers had experienced Layer 2 issues, and they wanted to leverage Cisco existing tools to alleviate the effects of Layer 2 loops on their Cisco Catalyst 6500 distribution switches. Their campus network consists of a three-layer model with Layer 2 connections between access and distribution switches, and Layer 3 connections between distribution and core switches.

This article presents the steps I took to develop a solution to protect the Catalyst 6500 in the presence of a Layer 2 loop. These steps include the following:

• How to simulate of a Layer 2 loop
• How to monitor the control plane traffic
• Control plane policing
• Storm control
• Hardware rate limiting
• Design recommendations

How to Simulate a Layer 2 Loop

Four switches were used in a lab environment to simulate a Layer 2 loop, as illustrated in the figure below. VLAN 10 was created as a user VLAN to simulate communication between two users’ PCs.

The configuration shown below was implemented on all switches to create a Layer 2 loop, specifically a spanning tree loop on VLAN 10.

Distribution-Sw1(config)no spanning-tree VLAN 10.

Within one minute after issuing the command above, the CPU utilization had risen to 99% for all four switches. To display the current CPU utilization, the command “show processes cpu” was used, as shown below.

Distribution-Sw1# show processes cpu sorted | exclude 0.00%__0.00%__0.00%
CPU utilization for five seconds: 99%/92%; one minute: 99%; five minutes: 99%
PID Runtime(ms)   Invoked   uSecs    5Sec    1Min    5Min    TTY Process
------------------------    Output omitted   ----------------------------------------------------

With the CPU reaching 99%, routing adjacencies were flapping, and communication within the user VLAN 10 between PC1 and PC2 was no longer possible. A set of measurements were taken and the results are shown on the figure below.

To develop a good understanding of the type of traffic responsible for high CPU utilization, it was essential to monitor the traffic processed by the control plane.

How to Monitor the Control Plane Traffic

The figure below illustrates the monitoring context for analyzing traffic that is processed by the Catalyst 6500 control plane. What is needed is a Switched Port Analyzer (SPAN), a host running a network protocol analyzer, and an Ethernet cable.

Control Plane Traffic Monitoring using SPAN

The configuration shown below defines port Gigabit 5/24 as a spanned port for CPU traffic. The traffic processed by the Route Processor (RP) and the Switch Processor (SP) is duplicated on Gigabit 5/24.

Distribution-Sw1(config)#monitor session 2 type local
Distribution-Sw1(config-mon-local)#source cpu rp
Distribution-Sw1(config-mon-local)#source cpu sp
Distribution-Sw1(config-mon-local)#destination interface gigabit 5/24

The configuration shown below checks the status of a created SPAN.

Distribution-Sw1# show monitor session 2
Session 2
---------
Type                   : Local Session
Status                 : Admin Disabled
Egress SPAN Replication State:
Operational mode       : -
Configured mode        : -

It is important to note that after the CPU SPAN is created, it defaults to “Admin Disabled”. To make it operational a “no shut” command is needed on the SPAN.

Distribution-Sw1(config)#monitor session 2
Distribution-Sw1(config-mon-local)#no shut

After the “no shut” command is issued, the SPAN is put in Admin Enabled mode as shown in the output below.

Distribution-Sw1# show monitor session 2
Session 2
---------
Type                   : Local Session
Status                 : Admin Enabled
Source Ports           :
Both               : rp,sp
Destination Ports      : Gi5/24
Egress SPAN Replication State:
Operational mode       : Centralized
Configured mode        : Centralized (default)

Now that the Layer 2 loop had been simulated, and that the traffic processed by the control plane was being monitored, the next step was to select what tools, or combination of tools could be used to mitigate the impact of the Layer 2 loop. The possible tools considered for the mitigation were:

• Control plane policing
• Storm control
• Hardware rate limiting

Control Plane Policing

Control Plane Policing is a feature in Cisco routers and switches that enable administrators to configure QoS policies to protect the control plane against reconnaissance, denial-of-service (DoS) attacks, and other scenarios that can lead to exhaustion of CPU resources.

To limit the impact on the switches CPU, control plane policing was applied to limit traffic that could potentially affect CPU utilization and to protect other traffic such as routing against CPU resource starvation. Control plane policing uses the same class-map and policy-map commands that you may be familiar with when you configure Quality of Service. The process of creating and applying control plane policing consists of the following three steps:

1. Use class-map to classify traffic processed by the control plane
2. Use policy-map to apply policing on classified traffic
3. Apply policy-map to the control plane

The traffic targeted for classification included: EIGRP, HSRP, SSH, SNMP, TACACS, DHCP, IGMP, and PIM. I created the following class-map:

class-map match-all class-eigrp
   match access-group name EIGRP

class-map match-all class-mgmt
   match access-group name MGMT

class-map match-all class-hsrp
   match access-group name HSRP

class-map match-all class-pim
   match access-group name PIM

class-map match-all class-igmp
   match access-group name IGMP

class-map match-all class-dhcp
   match access-group name DHCP


ip access-list extended EIGRP
   permit eigrp any host 224.0.0.10

ip access-list extended HSRP
   permit udp any host 224.0.0.2 eq 1985
   permit udp any host 224.0.0.102 eq 1985

ip access-list extended PIM
   permit pim any 224.0.0.0 0.0.0.255

ip access-list extended IGMP
   permit igmp any 224.0.0.0 31.255.255.255

ip access-list extended MGMT
   permit tcp any any tacacs
   permit tcp any any eq 22
   permit udp any any eq snmp
   permit icmp any any

ip access-list extended DHCP
   permit udp any eq bootpc any eq bootps
   permit udp any eq bootps any eq bootpc
   permit udp any eq bootps any eq bootps

I created a policy-map named copp-policy to apply policy restriction to traffic processed by the control place. As an example, for dhcp traffic I created the following policy.

class  class-dhcp
   police 32000 conform-action transmit exceed-action drop

In the policy statement above, the switch processes DHCP traffic up to a threshold of 32000 bits/sec, but any excess above this threshold is dropped by switch.

The values used for policing were based on measurement of actual traffic in the operational network(using the SPAN port). Below is a complete of list of policing statements applied to classified traffic.

policy-map copp-policy
   class  class-eigrp
      police 32000 conform-action transmit exceed-action transmit ! protection of EIGRP traffic
   
   class  class-hsrp
      police 32000 conform-action transmit exceed-action transmit ! protection of HSRP traffic

   class  class-mgmt
      police 512000 conform-action transmit exceed-action drop

   class  class-pim
      police 32000 conform-action transmit exceed-action drop

   class  class-igmp
      police 100000 conform-action transmit exceed-action drop

   class  class-dhcp
      police 32000 conform-action transmit exceed-action drop

   class  class-default
      police 2000000 conform-action transmit exceed-action drop

After the creation of the policy map, I applied it to the control plane as shown below

control-plane
   service-policy input  copp-policy

After applying the policy map to distribution switch control planes, the CPU utilization was reduced from 99% to an average of 92% as shown in the figure below.

With control policing applied, communication was now possible within user VLAN 10 between PC1 and PC2. Despite the CPU utilization reduction, intermittent packet losses were observed.

To alleviate further the impact of the Layer 2 loop, the next step was to use an additional tool. Given the nature of Layer 2 loops, large volumes of broadcast and multicast traffic get amplified, as explained earlier, and storm control was a logical selection for the tool to use.

Storm Control

Traffic storm control is a feature in Cisco switches that can be used to monitor broadcast, multicast, and unicast traffic levels entering a given interface over a 1-second interval. Traffic gets dropped during the monitoring interval when configured thresholds are exceeded.

After observing the amount of multicast and broadcast traffic in the operational network (using the SPAN port) it was estimated that 5% of the total bandwidth was sufficient to accommodate all broadcast and multicast traffic. The following configuration was applied to all switch trunk ports to set the storm control threshold to 5%.

Distribution-Sw1(config)#int  range f0/21 - 24
Distribution-Sw1(config-if)#storm-control multicast level  5
Distribution-Sw1(config-if)#storm-control broadcast level  5

Combining control policing with storm control resulted in a significant improvement of the CPU utilization as shown in the figure below.

Communication within the user VLAN 10 between PC1 and PC2 became normal with no packet losses. Although the CPU was less than 40% on average, CPU spikes for up to 95% were observed for short duration of time. An observation of the traffic processed by distribution switch control planes revealed large amount of BPDU, VTP, STP, and ARP traffic. To reduce the volume of this traffic, the hardware rate-limiting tool available on the Catalyst 6500 was selected as the next improvement step.

Hardware Rate Limiting

The Cisco Catalyst 6500 switches with Supervisor 720 or Supervisor 32 engine provide hardware rate limiters that can be used to limit specific Layer 2 and Layer 3 traffic that are processed by the control plane. The advantage of hardware rate limiters is that their use does not impact CPU utilization.

The following configuration was implemented on distribution switches. The selected parameters were based on measurement of actual traffic in the operational network (using the SPAN port).

Distribution-Sw1(config)#mls rate-limit layer2 pdu 620 10
Distribution-Sw1(config)#mls qos protocol ARP police 64000 2000

The command “mls rate-limit layer2 pdu 620 10” rate limits Layer 2 PDU protocol packets (including BPDUs, DTP, PAgP, CDP, STP, and VTP packets) to a maximum of 620 packets per second with a burst of 10 packets per seconds.

The command “mls qos protocol ARP police 64000 2000” rate limits ARP packets to a maximum of 64000 bps with a burst of 2000 bps

By combining control policing, storm control, and hardware rate-limiting, the CPU utilization was reduced to an average of 5% with a maximum peak of 28% as shown in the figure below.

With such a reduction of CPU utilization, no perceptible impact was observed on the performance of switches and the user VLAN 10 despite the presence of the Layer 2 loop.

Results from All Scenarios

The figure below depicts the results from all scenarios. This diagram provides a representation that enables a visual comparison of the results achieved for these scenarios.

Design Recommendations

The following are some basic recommendations that can help reduce the risks of occurrence of Layer 2 loops:

• Limit VLANs to a single wiring closet, whenever possible.
• Use UniDirectional Link Detection (UDLD) aggressive mode to prevent occurrence of spanning tree loops as a result of unidirectional links.
• As applicable, use BGPU guard, loop guard, and root guard switch features to protect against undesirable changes in the spanning tree topology.
• Disable the use of Dynamic Trunking Protocol (DTP) by using the “no negotiate” command on switch ports to prevent automatic negotiation of trunks and access ports.
• Set unused ports to an undefined VLAN, and set the administrative mode to shutdown to prevent unauthorized users to connect devices to unused ports.

Conclusion

The results that I have presented show that by combining control plane policing, storm control, and hardware rate limiters, significant and consistent reduction of CPU utilization can be achieved in the presence of Layer 2 loops. Configuration parameters used to define control plane policing, storm control, and hardware rate limiters were based on a 72-hour measurement of traffic processed by switch control planes in the operational network using the SPAN port. These results are based on tests performed in lab settings, but the conclusions reached are representative of what can be achieved in operational environments.

Even in a well-designed campus network, Layer 2 loops may occur due to wiring mistakes, misconfigured hosts (bridged interfaces), switch configuration mistakes, and loss of BPDU keepalives. By implementing mitigating tools presented in this article, you can prevent the Catalyst 6500 from being overwhelmed in the presence of a Layer 2 loop.

You should follow Cisco design guidelines in campus network design to minimize risks of spanning tree loop occurrence. One important goal in the campus network design is to minimize, as much as possible, the span of broadcast domains by reducing VLANs to a single wiring closet, and using high-speed Layer 3 switching for the core layer instead Layer 2 switching. By reducing the scope of broadcast domains within a campus, a shorter diameter can be obtained for spanning-tree, which inherently reduces the risk and scope of Layer 2 loops.

References

1. Protecting the Cisco Catalyst 6500 Series Switches Against Denial-Of-Service Attacks

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd802ca5d6.html

2. Configuring Control Plane Policing (CoPP)

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_paper_c11_553261.html

3. Protecting Cisco Catalyst 6500 Series Switches Using Control Plane Policing, Hardware Rate Limiting, and Access-Control Lists

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_paper_c11_553261.html

4. Configuring Traffic-Storm Control

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/storm.pdf

Device Registration and VCS Clustering

Introduction and Background

Introduction

When running a VCS cluster, some consideration needs to be given to the endpoint configuration parameters for client registration since providing a fault-tolerant solution for client registration and re-registration should be a critical element of any production VCS design.  This article discusses several approaches for managing SIP and H.323 device registrations to a clustered VCS.  (I previously provided some general information on the Cisco VCS Clustering Configuration.)

This article covers the following areas:

1. Background
2. Example Environment
3. Leveraging DNS
4. Endpoint Registration Approach: H.323
5. Endpoint Registration Approach: SIP
6. Other Considerations

Background

A Cisco VCS cluster can contain up to 6 member peers. From a call routing perspective, a VCS cluster acts a a single Local Zone.  In a fully provisioned cluster, only 4 VCS peers can be used for active registrations. The 2 remaining VCS peers are intended for failover. Endpoints and IPVC service providers can register to any VCS peer by IP address or hostname. Providing a  redundant mechanism for handling client registration requests should be a critical element of any production design. Equally important is the ability for registered clients to handle failure events that impact communication between the client and the attached VCS cluster member.

To facilitate our discussion we will lay out an example configuration. Then we will go into design options related to client registrations. Since the Cisco VCS can be a H.323 gatekeeper and a SIP proxy/registrar server we shall look at each standard independently.

For this discussion, we are focusing on VCS version X6.

Example Environment

The discussion in this article is based on the following example VCS configurations.

VCS Control (Internal):

The VCS Control cluster is the primary call processing solution for endpoints and IPVC infrastructure resources (e.g. MCU, ISDN GW) deployed on the internal or corporate network.

The VCS Control cluster name is: vcsc.netcraftsmen.net

VCS Expressway (External):

The VCS Expressway cluster is used as the registration point for endpoints deployed outside of the organization’s autonomous network (e.g. the Internet). The Expressway can also peer with call processing systems in other autonomous domains. The Expressway is also used to facilitate firewall traversal between the VCS Control and Expressway.

The VCS Control cluster name is: vcsc.netcraftsmen.net

Methodology

Diverse Methodology

In my humble opinion, if you have a diverse environment with different endpoint types (vendor or version) then you will need to leverage multiple techniques to ensure registration fault tolerance.

You will also need to pay attention to the nuances between devices or clients from the same manufacturer. For instance, the E20 leverages Name Authority Pointer (NAPTR) queries to determine whether it is "internal" or "external". Movi, on the other hand, has two separate entries for an "internal" and "external" SIP proxy.

Leveraging DNS

A key component of the design approaches outlined in this article is DNS. For a design to address the needs of different end point models and versions it must leverage DNS SRV Resource Records (RRs), DNS host ("A") RRs, and DNS round-robin techniques. For some devices Name Authority Pointers (NAPTR) RRs may be required. Finally, leveraging a "split dns" approach may help optimize client registrations for clients such as Cisco Movi.

DNS SRV Resource Records

RFC 2782 defines the DNS SRV resource record. A SRV record takes the form of:

_service._proto.name TTL class SRV priority weight port target

• service: The symbolic name of the hosted service (e.g. sip)
• proto: The transport used to access the hosted service (i.e. tcp or udp)
• name: The domain name or FQDN identifying the domain of the hosted service
• ttl: The "time to live" field (standard DNS TTL)
• class: The standard DNS class field. For SRV RRs this is always "IN"
• priority: The priority for the target host associated with the SRV RR. A lower priority value means "more preferred"
• weight: The weighted value for a target host. Leveraged when target hosts have equal priority value. This is a relative weight.
• port: The TCP or UDP port number used to access the hosted service
• target: The DNS A or AAA record de-referencing the IP address of the target host (NOTE: CNAME records are not permitted)

Wikipedia provides a decent discussion on DNS SRV records here.

H.323 and DNS SRV

H.323v5 Annex O extended the H.323 standard protocol stack to support DNS procedures for gatekeeper discovery. This is useful when statically provisioning endpoints with gatekeeper information. Leveraging mechanisms available in DNS SRV records allows gatekeeper redundancy and load balancing schemes to be deployed transparently.

H323 Annex O defines the following symbolic names to be used in the service field of the SRV record:

SIP and DNS SRV

IETF RFC 3263 defines how SIP agents can leverage DNS to resolve a SIP Uniform Resource Identifier (URI) into the IP address, port, and transport protocol of the next hop contact. The method in RFC 3263 may be applied to SIP register messages. RFC 3263 defines the following symbolic names to be used in the service field of the SRV record.

Endpoint Registration Approach: H.323

We are going to assume that gatekeeper configurations on IPVC endpoints will be manual. Auto-gatekeeper discovery is beyond the scope of this article. Based on Cisco best practice, IPVC system designers should leverage the following methods for identifying gatekeepers to endpoints and infrastructure devices (in order of preference):

1. DNS SRV Resource Records
2. DNS A records using a round-robin methodology
3. Static IP address assignments

Initial Registration DNS SRV

Using DNS SRV records is the preferred approach to providing redundant gatekeeper lists to IPVC endpoints and infrastructure devices for initial registration. From the endpoints perspective, you configure a hostname such as vcsc.netcraftsmen.net. You don't configure the service or protocol fields on the client. You will notice that the hostname configured is identical to the VCS cluster name. Using the cluster name isn't necessarily required but it is preferred.

When an endpoint that supports DNS SRV is configured with a gatekeeper hostname it will issue a DNS SRV request on startup. The assigned DNS resolver returns the SRV record with all VCS peer IP addresses, associated priorities, and associated weights. The endpoint will then attempt to register with the VCS peer which has the highest priority.

SRV records can be used to provide redundancy to VCS cluster peers as well as identifying backup or secondary VCS clusters. In very large deployments this can be used in conjunction with geographic proximity of the requesting endpoint to provide all kinds of fancy registration scenarios.  For our purposes, we are going to keep it simple. Building on our example, we could have the following SRV records.

_h323ls._udp.vcsc.netcraftsmen.net.     86400 IN SRV 10 0 1719 vcsc02.netcraftsmen.net.
                                        86400 IN SRV 10 0 1719 vcsc01.netcraftsmen.net.
_h323rs._udp.vcsc.netcraftsmen.net.     86400 IN SRV 10 0 1719 vcsc02.netcraftsmen.net.
                                        86400 IN SRV 10 0 1719 vcsc01.netcraftsmen.net.
_h323cs._tcp.vcsc.netcraftsmen.net.     86400 IN SRV 10 0 1720 vcsc02.netcraftsmen.net.
                                        86400 IN SRV 10 0 1720 vscc01.netcraftsmen.net.

In this example, we are using an equal priority and weight for all VCS peers. This should result in a balanced distribution of registrations over both cluster peers. We are also specifying a TTL value of 24 hours (86400 seconds). This is recommended because the endpoint will cache ALL addresses returned by the SRV request. Having a higher TTL will minimize the number of requests that need to be handled by the DNS server(s).

Initial Registration DNS Round-Robin

Your IPVC design should account for both DNS SRV and DNS Round-Robin if you have endpoints which do not support DNS SRV for H.323 registration/location services. It is also worth noting that endpoints that do leverage SRV will fall back to requesting DNS "A" resource records if there is no SRV records.

If the endpoint doesn't support SRV or there is no SRV record response, then the endpoint will perform a DNS A-record lookup. The DNS server responds with an IP address and the endpoint attempts registration. If that IP address doesn't respond, then the endpoint will submit another DNS A-record lookup. This will be repeated until the endpoint can register to a gatekeeper (VCS peer).

Using our example, we may have the following A-records configured in a round-robin fashion:

vcsc.netcraftsmen.net.       60     IN     A      10.1.1.32
vcsc.netcraftsmen.net.       60     IN     A      10.2.2.32

Again, we used the cluster name for the round-robin configuration. This is a good idea because you can then have a standard gatekeeper configuration for all endpoints, regardless of whether they support SRV record lookups or not. When using DNS A-records in a round-robin configuration, it is recommended to use relatively short TTL values. In our example, we use 60 seconds. I like to set the TTL to a value which is identical to my SIP and H.323 re-registration interval (which I adjust to 60 seconds).

Using DNS round-robin has the drawback of requiring the endpoint to make multiple DNS requests in the event that the VCS associated with the first DNS query response is unreachable.

Initial Registration IP Address

This is the least preferred approach because it represents a single point of failure with gatekeeper registration. You specify an IP address and hope the VCS device is online.

Re-Registration and Fault Tolerance

No matter which method is used, when an H.323 endpoint registers to a VCS cluster member the gatekeeper process provides the endpoint with an alternate gatekeeper list. The VCS builds this alternate gatekeeper list from the list of VCS cluster peer members. This alternate gatekeeper list will be used by the endpoint for all further re-registration transactions. The advantage of the alternate gatekeeper functionality is that endpoints will use this list in the event the gatekeeper holding the active registration should become unavailable. IOW, a H.323 endpoint that has an alternate gatekeeper list won't rely on DNS in the event of failover.

The endpoint still needs to detect the failure that means that there is an impact to service. The actual impact is roughly equivalent to the re-registration timer value.

Endpoint Registration Approach: SIP

Based on Cisco best practice, IPVC system designers should leverage the following methods for identifying SIP registrars to endpoints and infrastructure devices (in order of preference):

1. SIP Outbound (RFC 5626)
2. DNS SRV Resource Records
3. DNS A records using a round-robin methodology
4. Static IP address assignment

Registration Using SIP Outbound

Some Cisco endpoints can leverage a parameter called SIP Outbound. Using this option, an endpoint is configured to have more than one SIP registrar/proxy address and has the "SIP Outbound" optional parameter enabled. When configured in this manner, the endpoint will actually open separate TCP connections to each configured SIP registrar/proxy server. The endpoint keeps all sessions open (assuming there is an available network path).

An example endpoint configuration:

• Proxy 1
  • Server discovery: Manual
  • Server address: vcsc01.netcraftsmen.net (or 10.1.1.32)
• Proxy 2
  • Server discovery: Manual
  • Server address: vcsc02.netcraftsmen.net (or 10.2.2.32)
• Outbound: On

Registration Using DNS SRV

Using DNS SRV for SIP registrations is very similar to the method described for H.323. From the endpoints perspective, you configure a hostname such as vcsc.netcraftsmen.net. You don't configure the service or protocol fields on the client. You will notice that the hostname configured is identical to the VCS cluster name. Using the cluster name isn't necessarily required but it is preferred.

When an endpoint that supports DNS SRV is configured with a SIP proxy hostname it will issue a DNS SRV request on startup. The assigned DNS resolver returns the SRV record with all VCS peer IP addresses, associated priorities, and associated weights. The endpoint will then attempt to register with the VCS peer which has the highest priority.

SRV records can be used to provide redundancy to VCS cluster peers as well as identifying backup or secondary VCS clusters. In very large deployments this can be used in conjunction with geographic proximity of the requesting endpoint to provide all kinds of fancy registration scenarios.  For our purposes, we are going to keep it simple. Building on our example, we could have the following SRV records.

_sip._tcp.vcsc.netcraftsmen.net.     86400 IN SRV 10 0 5060 vcsc02.netcraftsmen.net.
                                     86400 IN SRV 10 0 5060 vcsc01.netcraftsmen.net.
_sips._tcp.vcsc.netcraftsmen.net.    86400 IN SRV 10 0 5061 vcsc02.netcraftsmen.net.
                                     86400 IN SRV 10 0 5061 vscc01.netcraftsmen.net.

The SRV records defined above facilitate SIP sessions that leverage TCP or TLS transport mechanisms. We don't specify a UDP configuration because the call control channel for SIP conversations involve messages that are too large to be carried on a packed based (vs. stream based) transport.

In this example, we are using an equal priority and weight for all VCS peers. This should result in a balanced distribution of registrations over both cluster peers. We are also specifying a TTL value of 24 hours (86400 seconds). This is recommended because the endpoint will cache ALL addresses returned by the SRV request. Having a higher TTL will minimize the number of requests that need to be handled by the DNS server(s).

Registration Using DNS Round-Robin

As with H.323, when an endpoint doesn't support method 1 or method 2 then we fall back to using a DNS round-robin approach. In our example, SIP clients would use the exact same configuration presented for H.323 clients.

Registration Using Static IP Address

This is the least preferred approach because it represents a single point of failure with gatekeeper registration. You specify an IP address and hope the VCS device is online.

Re-Registration and Fault Tolerance

The closest analog to the H.323 "Alternate Gatekeeper" feature in SIP is found in RFC 5626. An endpoint that supports the ability to establish and maintain divergent SIP connections offers the optimum fault tolerant model.

Endpoints that do not support this capability (or are not configured to support it) must rely on one of the other three approaches. Using the DNS SRV approach, an endpoint that loses connection to its primary VCS will use the cached DNS SRV response to establish a connection to an alternate SIP proxy/registrar. With DNS round-robin, the endpoint must query DNS again, which can add considerable lag time.

Other Considerations

Cisco Movi Clients

The Cisco Movi client is a SIP-only client that supports DNS SRV. Therefore it shall prefer SIP registration option 2 (SIP Outbound is not supported). Movi clients are actually provisioned with separate parameters for “internal” and “external” registrations. This approach means that some thought must be given to how DNS records are resolved by internal and external DNS clients.

When the Movi client is initiated it will always attempt to register to the VCS host configured as the "internal" VCS. If this registration attempt fails then the VCS will try the external VCS. If you are using DNS names for your internal/external VCS parameters then the first step Movi will use to register to the internal VCS is to query DNS to resolve the hostname to an IP address.

Therefore, NetCraftsmen recommends that DNS be configured so that queries from external (i.e. Internet) clients cannot resolve DNS SRV or host records for the VCS Control cluster. This will optimize the time it takes for a Movi client on the Internet to determine it should register to the VCS Expressway (i.e. "external") cluster.

SRV for Registration vs. SRV for Call Routing/Searching

The design approach outlined in this article is focused on facilitating the registration process only. The registration process should be treated separately from call processing. DNS SRV and NAPTR records supporting call routing may be different than those used to support the registration process.

Chesapeake NetCraftsmen has been supporting our health care customers with designs and implementations of 'Medical Grade Networks'. In this whitepaper we will describe medical grade network design and discuss some of the problems that we find in real networks.

Background

Most health care organizations have an existing network infrastructure - often there are several physically separate networks, supporting clinical data, non-clinical data, voice, research, educational equipment, and departmental staff. For several reasons (manageability, efficiency, costs), there is a desire to converge these separate networks into one physical infrastructure, while still providing the isolation, security, and responsiveness needed by the organization.

High Level Converged Network Infrastructure Design

Our design of the converged network infrastructure for a health care organization is based on the hierarchical, three-layer model: core, distribution, and access layers. This hierarchy establishes the foundation and connectivity for the entire network, as shown below. It is a resilient network that is easy to understand and easy to troublnaceshoot.

Within each layer are redundant modules that serve a specific role in that layer. The hierarchy allows changes or upgrades to be performed at one layer in the hierarchy without disruption or significant changes to the other layers. End-do-end connectivity uses the ISO network model’s network layer (also called the routing layer or Layer 3), which provides network stability, fault isolation, fast response to failures, and maximizes redundant path utilization. Large Layer 2 switching domains are avoided, because of their inherent susceptibility to operational failures. (For example, a large Layer 2 network that failed and caused a four-day network-wide failure at the CareGroup Healthcare System in Boston is described here: http://www.cio.com.au/article/6511/all_systems_down/).

We overlay the network infrastructure foundation with multiple virtual networks to create the connectivity and the isolation previously provided by the separate physical networks. These virtual networks we call Virtual Network Overlays (VNO). By using VNOs, separate logical networks can be built to support clinical, voice, research, and guest users/devices while maintaining appropriate isolation from each other. Connectivity between VNOs is controlled at clearly identified connection points where security and connectivity policies control access into each VNO. The gray area in the figure below illustrates the domain of the Clinical General Purpose VNO across a healthcare network.

Along with Layer 3 routing, we recommend a structured addressing plan based on IPv4, augmented with groundwork for implementation of IPv6. The structured IP addressing plan incorporates stability and security features, making it easy to implement Network Admission Control for network sign-on and easy identification of IP telephony endpoints.

The network Quality of Service (QoS) design is developed to support delay sensitive network applications and to improve application performance. It provides end-to-end differentiated service levels, ensuring that applications like Voice over IP (VoIP), clinical systems, and telemedicine receive preferential treatment over bulk data applications like email, system backups, and back-office applications. The generalized QoS design, as shown in the figure below, is flexible, easily extending to support new services, and is based a Cisco QoS model.

To implement high availability throughout the network infrastructure, we design with redundant pairs of devices, especially for the core and distribution layer. We find that replicating a standard redundant design, based on pairs, is predictable, easy to implement, scalable, and relatively simple to maintain.

Security is a crucial part of the health care network designs. In addition to standard infrastructure security practices, our designs include a security module to control all communication between the global network and the virtual networks. IPS/IDS devices and firewalls customized with organization specific policies and rules comprise the nucleus of the security module. All traffic that passes between the virtual networks and the global network travels through these network security devices. In addition, the perimeter network is secured to meet appropriate health care policies and regulations.

Network Operations

Once the network is built, network management and operations makes sure that any problems are quickly identified and corrected. We have used the network management architecture containing the functionality shown in the following diagram in multiple organizations. It provides the core functionality that is needed without overwhelming the network operations staff with new tools to learn and use.

The combination of event processing and performance data collection provides real-time notification of network problems. NCCM (Network Change and Configuration Management) identifies inconsistent configurations that create compliance problems and security holes, in addition to performance problems and errors. IP Address Management tracks devices on the network, so that when a problem is identified, the affected device can be quickly located.

Performance monitoring systems report on high utilization interfaces, identifying hot spots in the network where links are oversubscribed. These same systems also identify interfaces with high numbers of errors, allowing the network staff it correct the problem and increase productivity of anyone using applications whose data transits the interface.

Dashboards that show network health and operational problems are key to identifying and correcting network problems before they impact healthcare applications. A combination of real-time alerts and the network health metrics dashboard, shown in the following diagram, informs the network team of immediate and long-term problems that need to be corrected.

Conclusion

The Medical Grade Network has become a critical infrastructure element for healthcare organizations, incorporating significant redundancy and security components. With a few good network management tools, the network team can work proactively to keep the network operating smoothly.

About Chesapeake NetCraftsmen

Authors: Terry Slattery (CCIE #1026), Carole Warner Reece (CCIE #5168), and Steve Meyer

Section 1 topics are outlined in this document, which is one of a set of six documents that correlates CCIE R&S Blueprint v4.0 from the Cisco Learning Network DOC-4375 "CCIE Routing and Switching Lab Exam Topics (Blueprint) v4.0" with the expanded topic list from the DOC-6864 "CCIE Routing and Switching Lab Exam Checklist".

Numbering and topics from the DOC-4375 blueprint are shown in blue, numbering and expanded topics from the DOC-6864 checklist are shown in red. When the topic appears on both blueprints, it is shown in purple.

_____________________________________________________________________________________________

PDF of Combined RS Topics from DOC-4375 and DOC-6864

Excel File of Combined RS Topics from DOC-4375 and DOC-6864

• Section 1 Implement Layer 2 Technologies
• Section 2 Implement IPv4
• Section 3 Implement IPv6
• Section 4 Implement MPLS Layer 3 VPNs
• Section 5 Implement IP Multicast
• Section 6 Implement Network Security
• Section 7 Implement Network Services
• Section 8 Implement Quality of Service
• Section 9 Troubleshoot a Network
• Section 10 Optimize the Network

Section 2 topics are outlined in this document, which is one of a set of six documents that correlates CCIE R&S Blueprint v4.0 from the Cisco Learning Network DOC-4375 "CCIE Routing and Switching Lab Exam Topics (Blueprint) v4.0" with the expanded topic list from the DOC-6864 "CCIE Routing and Switching Lab Exam Checklist".

Numbering and topics from the DOC-4375 blueprint are shown in blue, numbering and expanded topics from the DOC-6864 checklist are shown in red. When the topic appears on both blueprints, it is shown in purple.

_____________________________________________________________________________________________

PDF of Combined RS Topics from DOC-4375 and DOC-6864

Read more...

• Section 1 - Implement Layer 2 Technologies
• Section 2 - Implement IPv4
• Section 3 - Implement IPv6
• Section 4 - Implement MPLS Layer 3 VPNs
• Section 5 - Implement IP Multicast
• Section 6 - Implement Network Security
• Section 7 - Implement Network Services
• Section 8 - Implement Quality of Service
• Section 9 - Troubleshoot a Network
• Section 10 - Optimize the Network

Sections 3, 4, and 5 topics are outlined in this document, which is one of a set of six documents that correlates CCIE R&S Blueprint v4.0 from the Cisco Learning Network DOC-4375 "CCIE Routing and Switching Lab Exam Topics (Blueprint) v4.0" with the expanded topic list from the DOC-6864 "CCIE Routing and Switching Lab Exam Checklist".

Numbering and topics from the DOC-4375 blueprint are shown in blue, numbering and expanded topics from the DOC-6864 checklist are shown in red. When the topic appears on both blueprints, it is shown in purple.

_____________________________________________________________________________________________

PDF of Combined RS Topics from DOC-4375 and DOC-6864

• Section 1 - Implement Layer 2 Technologies
• Section 2 - Implement IPv4
• Section 3 - Implement IPv6
• Section 4 - Implement MPLS Layer 3 VPNs
• Section 5 - Implement IP Multicast
• Section 6 - Implement Network Security
• Section 7 - Implement Network Services
• Section 8 - Implement Quality of Service
• Section 9 - Troubleshoot a Network
• Section 10 - Optimize the Network

Section 6 topics are outlined in this document, which is one of a set of six documents that correlates CCIE R&S Blueprint v4.0 from the Cisco Learning Network DOC-4375 "CCIE Routing and Switching Lab Exam Topics (Blueprint) v4.0" with the expanded topic list from the DOC-6864 "CCIE Routing and Switching Lab Exam Checklist".

Numbering and topics from the DOC-4375 blueprint are shown in blue, numbering and expanded topics from the DOC-6864 checklist are shown in red. When the topic appears on both blueprints, it is shown in purple.

_____________________________________________________________________________________________

PDF of Combined RS Topics from DOC-4375 and DOC-6864

• Section 1 - Implement Layer 2 Technologies
• Section 2 - Implement IPv4
• Section 3 - Implement IPv6
• Section 4 - Implement MPLS Layer 3 VPNs
• Section 5 - Implement IP Multicast
• Section 6 - Implement Network Security
• Section 7 - Implement Network Services
• Section 8 - Implement Quality of Service
• Section 9 - Troubleshoot a Network
• Section 10 - Optimize the Network

Section 7 and Section 8 are outlined in this document, which is one of a set of six documents that correlates CCIE R&S Blueprint v4.0 from the Cisco Learning Network DOC-4375 "CCIE Routing and Switching Lab Exam Topics (Blueprint) v4.0" with the expanded topic list from the DOC-6864 "CCIE Routing and Switching Lab Exam Checklist".

Numbering and topics from the DOC-4375 blueprint are shown in blue, numbering and expanded topics from the DOC-6864 checklist are shown in red. When the topic appears on both blueprints, it is shown in purple.

_____________________________________________________________________________________________

PDF of Combined RS Topics from DOC-4375 and DOC-6864

• Section 1 - Implement Layer 2 Technologies
• Section 2 - Implement IPv4
• Section 3 - Implement IPv6
• Section 4 - Implement MPLS Layer 3 VPNs
• Section 5 - Implement IP Multicast
• Section 6 - Implement Network Security
• Section 7 - Implement Network Services
• Section 8 - Implement Quality of Service
• Section 9 - Troubleshoot a Network
• Section 10 - Optimize the Network

Section 9 and Section 10 topics are outlined in this document, which is one of a set of six documents that correlates CCIE R&S Blueprint v4.0 from the Cisco Learning Network DOC-4375 "CCIE Routing and Switching Lab Exam Topics (Blueprint) v4.0" with the expanded topic list from the DOC-6864 "CCIE Routing and Switching Lab Exam Checklist".

Numbering and topics from the DOC-4375 blueprint are shown in blue, numbering and expanded topics from the DOC-6864 checklist are shown in red. When the topic appears on both blueprints, it is shown in purple.

_____________________________________________________________________________________________

PDF of Combined RS Topics from DOC-4375 and DOC-6864

Updated 02/22/2010: A big thank-you to all the folks that were gracious enough to speak with us! We are no longer collecting candidate names. We really appreciate the help of every one who participated!

Chesapeake NetCraftsmen would like to invite recent Routing and Switching CCIEs and CCIE candidates to participate in a survey of CCIE preparation methods. In the next few weeks, Terry Slattery and Carole Warner Reece are looking to interview about 100 people who attempted or passed the CCIE lab exam between April 1, 2010 and Dec 31, 2010.

During the interview process, we will gather information about your preparation methods and  learning materials, and get your feedback on what tools or processes you think are most helpful in preparing for the CCIE lab exam. We will also look at data from the Cisco CCIE lab and the Cisco 360 Learning Program (where available).

CCIE Candidate Survey Privacy Policy:

Chesapeake NetCraftsmen respects your privacy. We will use the information we gather only in support of a Cisco Systems funded survey. We may gather information from you directly, or with your permission also through Cisco sources. We will not release the specific information gathered except to Cisco Systems.