The Stress Is On QoS



	The Stress Is On QoS

	Peter J. Welcher

Introduction

There are 10 (binary) kinds of people, those who have been reading these articles and those who haven't . The second group will probably never see these words. Those in the first group may have noticed I like to talk about things I've been doing. I hope this keeps the topics fresh and real-world. It certainly helps me out with having to do less preparation before writing. This month is another article relating to work I've been doing.

I and some of our other folks have been working on a QoS project for a large network. We're describing the architecture, the configurations, things to consider, initial testing and deployment, etc. in the customer's specific environment. One of the tasks was testing and demonstrating QoS in the lab. (Stress-testing QoS, hence the title). We put together a set of freeware and used it for this. Feedback suggests the demo is effective.

The original idea for this sort of thing is something I first saw with a Cisco SE in San Francisco, Eric Mar, who had a road show QoS Boot Camp a few years back. He was kind enough to run through it for me and some others. It provided the core idea for the DQOS course labs when I and some other folks wrote the initial outline. (No, I didn't write the actual course slides and materials). Since then, I've borrowed ideas from the now-retired DQOS course labs, and some other sources, including some quick and dirty demos I've done over time.

The key idea here is to vividly demonstrate the benefits of QoS. For Cisco, the intent was indubitably sales and showing off the new capabilities that differentiated Cisco routers. For you and me, the goal might be to demonstrate the value and efficacy of QoS to management, consulting customers, etc. If you want your management chain to allocate time, training, staff, or money for QoS, they need to understand what it does for the network and why its important.

Vivid here means either showing something or (as a fallback) listening to something. Seeing is believing. Dry numbers in show command output or some test tool window may be more informative, but are less effective, particularly with not-so-technical managers.

This article describes in general terms how you can put together your own QoS demo. I don't want to give away all our secrets and added value, but a lot of what I've been doing is something you can do on your own. This is also a great way to work with QoS in a lab, looking at the technical information and seeing how that correlates with what you're seeing or hearing.

The Basic Idea

The way the demo works is to congest a link that carries loading traffic plus the demo traffic. You then show how ugly things are without QoS, and then apply QoS to the interface to make some things better. It can be particularly effective if you have two video streams or two phone calls, and only one is improved. That makes it evident that the congestion and packet drops are still present, but that QoS has protected the important traffic.

Important thing to note: QoS is not a panacea, and doesn't make up for not having enough bandwidth. QoS provides priority access to the bandwidth -- winners, which implies there are also losers. Don't let your boss leave the demo thinking QoS will make any amount of video or audio work. Nor will it replace buying more bandwidth, although QoS may allow you to delay having to buy more bandwidth for a little while. The point to make is that QoS can protect important traffic, IF you have enough bandwidth for that traffic plus some left over for the remaining Best Effort Traffic.

You do have to tune what you demo to the bandwidth on the link. I suggest you not do demos on Gigabit Ethernet. You would probably need some costly power tools or several computers to generate enough traffic to congest it. Rather, work with fractional-T1 to say 6 Mbps links. This reflects most real-world WAN links today, except perhaps at corporate HQ or data center. Congesting links at this speed doesn't take a lot of effort, and yet you can run some fairly interesting multi-media traffic over them for demo purposes. Serial links are good for the bottleneck link, especially if you want to cause congestion by adjusting the clockrate downwards.

The following figure shows that you can do a demo with a rather minimal set of equipment. Using multiple laptops attached via a switch can help if you want to classify traffic based on IP address.

qos demo lab diagram

My definition of "Interesting" (as in "interesting demo") varies with speed. If you're working with a 128-256 Kbps link, forget video and stick to audio. Run some $10 handsets into FXS ports on a pair of 2600's and you can demo QoS interacting with VoIP. I've steered clear of mixing IPT (IP Telephony) and QoS demos, just due to complexity. With the VoIP approach, you can pre-configure and not have to fuss with the telephony side of things while demoing QoS. If you're working in the T1 or above range, consider video. For ideas, see below.

If your critical application is TCP-based or web-based, there's a way to make QoS impact visible there too. See below ("web").

Free Tool Toolkit

The starting point in building your own QoS demo(s) is to have a toolkit of free or inexpensive programs you can use. Here's what's in my bag of tricks now.

Tool Name	Where To Get It
TTCP	See Netcordia's tools page, http://www.netcordia.com/tools/tools.html. See also http://ftp.arl.mil/~mike/ttcp.html.
TTCP Java	See Netcordia's tools page, http://www.netcordia.com/tools/tools.html
TTCPW	http://renoir.csc.ncsu.edu/ttcp/W95-98-NT/ttcpw.zip
iperf	http://dast.nlanr.net/Projects/Iperf/
jperf	http://dast.nlanr.net/Projects/Jperf/
VideoLAN	http://www.videolan.org/
apache web server	http://www.apache.org/
Microsoft IIS or Personal Web Server	Included with Windows NT or 2000.
Microsoft Web stress testing tools	http://support.microsoft.com/default.aspx?scid=kb;en-us;231282
Special web pages	http://www.netcraftsmen.net/welcher/misc/bigimage4.html. See description below.
net-snmp	http://net-snmp.sourceforge.net/
NetMeeting	Included with Windows. Under Programs --> Accessories --> Communications in Windows 2000 or XP.
PING or Cisco extended PING	Included on your PC, Linux box, and there's always extended PING in the Cisco IOS.
Ethereal	http://www.ethereal.com/
Java WAN simulator	http://tanmaykm.tripod.com/work/javawork.html
cbqos + RRFW (aka Torrus)	http://rrfw.sourceforge.net/plugins/cisco-cbqos.pod.html, http://rrfw.sourceforge.net/

Cisco used to provide the lovely QDM tool for free, which you put into flash in your router. It allowed CBWFQ to be configured with a GUI, and provided graphs of the Cisco Class-Based QoS MIB. As you can probably tell from my prior article, I really liked this applet. Regretfully, Cisco declared it End of Life a while back.

Other things you might want to buy to use with this:

2 Cisco 1700, 2600 routers or better with 1 LAN and 1 WAN interface for QoS demo lab.
2 sets of FXS ports for the above (optional). Get two or three $5-10 analog handsets to use with them.
Videocam(s) to use with NetMeeting on your PC. Two for $30.00 with rebate the other day.
Microphone(s), ditto.

The newest Cisco IOS images don't fit into the 2600's unless they're the XM models (more RAM). So at some point, using the new 1800 or 2800 model routers may be a good idea. They certainly make a more flexible lab with better future-proofing.

Examining the free tools

Tool Name	Description and Comments
TTCP	TTCP is code from the 80's by Terry Slattery (now of Netcordia) that measures maximum network throughput, minimizing disk I/O and other performance-inhibiting factors. Cisco Enterprise builds of IOS from 11.2 through about 12.2 included TTCP. I prefer TTCP on a laptop, possibly with windowing front end, for simplicity.
TTCP Java	Java version of the above, with simple windowing interface (requires getting the CLASSPATH environment variable right). The command "java ttcp" runs it if your environment is set up properly. Can also be run with CLI arguments. Java doesn't mean slow: one of our guys had it doing 700 Mbps on a medium-sized Solaris box with Gigabit interface. I've consumed most of a FastEthernet link with it myself.
TTCPW	Windows version of TTCP.
iperf	IPERF is like TTCP with enhancements. It can send TCP or UDP with various CLI settings.
jperf	JPERF is a Java-based window front end for IPERF making it easier to figure out all the settings. At the end of a test run, JPERF produces a graph of the traffic levels during the run. You do need to put the files mentioned into your CLASSPATH. Hint: not being a Java programmer, I had figured out that the directory containing .class files goes in the CLASSPATH. It then took some experimenting before figuring out that the actual .jar filename, not the directory it is in, also has to be in the CLASSPATH. This was the only real gotcha in trying to use JPERF.
VideoLAN	Free streaming video server. You do have to find MPEG trailer files on the web, or otherwise match the format(s) it understands. It took me some fiddling to get it working, but the price is right. VideoLAN can also be used as a multicast video source for a multicast lab! Firing off a moderate to high resolution video is also a great way to eat up 1 Mbps or more on a link. You might want a 384K video stream (i.e. small window or lower resolution) for demos with T1 or slower links.
apache web server	If you want a good webserver that's well-documented.
Microsoft IIS or Personal Web Server	You probably have this too. Use the help and it's pretty easy to figure out how to get IIS to serve up files or CGI scripts. (Ok, they left out the critical "%s %s" you need to get PERL working under IIS, but google found that tidbit for me rather quickly).
Microsoft Web stress testing tools	I haven't used this, but it's certainly a "web sucker" that pulls down web pages, useful if you need a background web load to congest a link.
Special web pages	I used Visio to create a web page with some arrows pointing downwards. Saving it at various sizes and resolutions created JPEG images from 144 KB to 28 MB. I then built a web page that includes the image with a couple of lines of JavaScript added to cause the page and image to keep reloading every second, defeating local caching. See below, or follow the link above. Note: to avoid DoS'ing our web server, the posted copy is a 144 KB image with 10 second refresh, instead of 1 second as in the code snippet below. If your Internet link is fast, you may not even see the blink as it refreshes. On the congested 2 Mbps link in our lab, this was about the right combination. Hints: Change the image file size to use more or less bandwidth. Making copies of the files as "fast." and "slow." lets you demonstrate the Cisco NBAR feature.
net-snmp	Free SNMP CLI software. Also available as a PERL package from CPAN. Allows you to create a simulated network management server, in case you want to guarantee some bandwidth to say SNMP, PING, telnet from certain servers. We've been doing this sort of thing lately to try to ensure we can manage the network despite a worm/virus storm or other congestive event. Note: you need to make sure you'll be able to telnet TO the central NMS box, if that's the one with protected telnet to network devices.
PING or Cisco extended PING	Network management software uses PING to track device availability. PING is also good for some extra loading when you need it, or another type of traffic (e.g. if you want something to match yet another class, or use ping to crudely demo packet drops).
NetMeeting	Add a pair of microphones and you can do G.711 VoIP between two laptops. Add videocams and you can do low-resolution video between two laptops. Since you can hear and see the speaker in the demo lab, any delays become utterance and display in NetMeeting are fairly noticeable.
Ethereal	The great free packet capture and analysis tools. Vastly improved with versions 10.4 through 10.7. See below for a sample graph. Useful for seeing and documenting how bandwidth is being used in your demo lab.
Java WAN simulator	This is a small Java snippet of code, but not a bad idea. It listens on one port, forwards on another after a fixed delay. It would be rather easy to add random delays and / or packet drops. Run multiple copies to simulate delays on multiple ports. Or use a PERL or Tk wrapper? Every programming shop should either have a pair of router or this code, for testing development tools or application WAN-readiness before deployment!
cbqos + RRFW (aka Torrus)	This might be used for graphs showing router queuing internals while demoing QoS. RRFW or Torrus is nifty freeware, user-friendly MRTG/Cricket-like front end for RRD database and Net SNMP. It polls and then lets you pull up graphs via a web front end. The user friendly part is that web scripts do most of the work, whereas Cricket and MRTG require more editing of configuration files. The cbqos plug-in adds Cisco CB QoS MIB support. We did have substantial difficulties recently trying to get this to work on Cygwin over Windows 2000. There were one or two snags building it with Fedore Core 3 Linux (default libpng package version, for example) but that went more smoothly.

Commercial Tools

If you've got some money for a "real" QoS lab, then you either want to introduce WAN delay and jitter simulation or load up on connections or traffic. There are a vast number of vendors and tools. Here are some we've noticed:

Vendor	URL
Shunra	http://www.shunra.com/
Ixia	http://www.ixiacom.com/
Spirent Communications	http://www.spirentcom.com/
Candela Technologies	http://www.candelatech.com/

For example, if you're doing serious testing ability of an IPSec VPN box to hold up under a large number of connections or high traffic load, you might use something like one of Ixia's product. My earlier article on that particular topic was focussed more on the cheap "do it yourself" quick testing approach.

How to Build Your Demo

Start with only the key applications running. Make sure they use less bandwidth than your link has. Use Ethereal to determine average traffic level for each application. Another option: if you set up classification and marking, the "show policy interface" command will tell you the average traffic in each class. That tells you how much bandwidth to provide in your output QoS policy using priority and bandwidth commands.

Then add background traffic to overload the link. By adding or removing the "service-policy out policy-name" command from the relevant interface, you can ensure that the video or audio is bad until QoS is applied. Test thoroughly before demonstrating, because it may take a few iterations to get it working smoothly.

Hint: if you're using G.729 voice compression, it's such a small stream I found it hard to clobber with other traffic. G.711 uses more bandwidth and is easier to "step on".

I used a small PERL script to launch several applications at one time. This was mostly for convenience in starting the demo.

A Little JavaScripting Goes a Long Way

Here's the source code for the magic web page mentioned above. It uses a smidge of JavaScript to defeat caching and cause repeated loading of the web page image. I commented each line out below to ensure it doesn't do anything when you view this document! So if you're trying to cut and paste, paste into Wordpad, remove the "//" and leading spaces, and save. Then change the file extension to .html and you're ready to load.. load... load that image!.

//    <!DOCTYPE html PUBLIC "-//w3c//dtd html 4.0 transitional//en">
//    <html>
//    <head>
//    <title>bigimage</title>
//    <script language="JavaScript1.2"> 
//    </script>
//    </head>
//    <body onload="doit()">
//    <h2>Now is the time to load an image... </h2>
//    <img src="bigimage4.jpg" title="" alt="big image 4"
//    style="width: 400px; height: 500px;">
//    <br>
//    </body>
//    </html>

For what it's worth, I tend compose basic pages using Netscape Composer, then add scripting using WordPad. I've been avoiding MS Word, it adds way too much junk to HTML pages, trying to control appearance. That tends to make them impossible to hand-edit when you want to.

Ethereal

Here's a sample of the grapher in Ethereal. Every time I show this to someone who hasn't seen it, they say "where do you download that from", so it's certainly an eye-catcher. The graphs show traffic from capturing myself hitting www.cnn.com and www.cnbc.com with a web browser. The red and green graphs show me alternately clicking on refresh. The blue graph is a different address, my guess is a separate server for the graphics for one of the two sites.

This can be very useful for baselining a QoS demo lab, since you can put in a filter such as address or protocol ("http") and easily graph all the matching traffic. Caution: it's in Bytes / second, not bps, and you do have to switch it from Packets/second. I've never understood why anyone would want Packets/second, they rarely seem all that interesting to me.

This shows captured packets. If you capture inbound to a router and (separately) outbound, you can compare the two. The Cisco Class-Based QoS MIB does all this with one set of polls. The CBQOS tool mentioned earlier takes some work to build and get working, but displays this data. The commercial software I know of that reports on the CB QoS MIB is unfortunately costly.

Summary

Several of my prior articles discussed QoS. For configuring QoS, you'll want to use the newest cut at it, CBWFQ and the Modular QoS CLI. For more about this, see the article, New Quality of Service Features in Cisco IOS 12.1, http://www.netcraftsmen.net/welcher/papers/newqos121.html.

If you don't have time to roll your own QoS demo, we'd of course be glad to come do it as a brief consulting engagement. It might also be useful to talk about QoS-readiness, and what your organization's QoS needs might be.

When we started experiencing problems with the CBQOS freeware, I wrote some PERL. I have a rough tool that offers you a set of choices, then polls CB QoS data and displays it in real time using gnuplot. This may or may not be the subject of a future article.

Your comments, questions, and suggestions for future articles are of course welcome! See below to decipher my email address.

Dr. Peter J. Welcher (CCIE #1773, CCSI #94014, CCIP) is a Senior Consultant with Chesapeake NetCraftsmen. NetCraftsmen is a high-end consulting firm and Cisco Premier Partner dedicated to quality consulting and knowledge transfer. NetCraftsmen has ten CCIE's, with expertise including large network high-availability routing/switching and design, VoIP, QoS, MPLS, IPSec VPN, wireless LAN and bridging, network management, security, IP multicast, and other areas. See http://www.netcraftsmen.net for more information about NetCraftsmen. Pete's links start at http://www.netcraftsmen.net/welcher . New articles will be posted under the Articles link. Questions, suggestions for articles, etc. can be sent to pjw <at> netcraftsmen <dot> net (formatted this way to fool email harvesting software).