The Stress Is On QoS
There are 10 (binary) kinds of people, those who have been reading these articles and those who haven't . The second group will probably never see these words. Those in the first group may have noticed I like to talk about things I've been doing. I hope this keeps the topics fresh and real-world. It certainly helps me out with having to do less preparation before writing. This month is another article relating to work I've been doing.
I and some of our other folks have been working on a QoS project for a large network. We're describing the architecture, the configurations, things to consider, initial testing and deployment, etc. in the customer's specific environment. One of the tasks was testing and demonstrating QoS in the lab. (Stress-testing QoS, hence the title). We put together a set of freeware and used it for this. Feedback suggests the demo is effective.
The original idea for this sort of thing is something I first saw with a Cisco SE in San Francisco, Eric Mar, who had a road show QoS Boot Camp a few years back. He was kind enough to run through it for me and some others. It provided the core idea for the DQOS course labs when I and some other folks wrote the initial outline. (No, I didn't write the actual course slides and materials). Since then, I've borrowed ideas from the now-retired DQOS course labs, and some other sources, including some quick and dirty demos I've done over time.
The key idea here is to vividly demonstrate the benefits of QoS. For Cisco, the intent was indubitably sales and showing off the new capabilities that differentiated Cisco routers. For you and me, the goal might be to demonstrate the value and efficacy of QoS to management, consulting customers, etc. If you want your management chain to allocate time, training, staff, or money for QoS, they need to understand what it does for the network and why its important.
Vivid here means either showing something or (as a fallback) listening to something. Seeing is believing. Dry numbers in show command output or some test tool window may be more informative, but are less effective, particularly with not-so-technical managers.
This article describes in general terms how you can put together your own QoS demo. I don't want to give away all our secrets and added value, but a lot of what I've been doing is something you can do on your own. This is also a great way to work with QoS in a lab, looking at the technical information and seeing how that correlates with what you're seeing or hearing.
The Basic Idea
The way the demo works is to congest a link that carries loading traffic plus the demo traffic. You then show how ugly things are without QoS, and then apply QoS to the interface to make some things better. It can be particularly effective if you have two video streams or two phone calls, and only one is improved. That makes it evident that the congestion and packet drops are still present, but that QoS has protected the important traffic.
Important thing to note: QoS is not a panacea, and doesn't make up for not having enough bandwidth. QoS provides priority access to the bandwidth -- winners, which implies there are also losers. Don't let your boss leave the demo thinking QoS will make any amount of video or audio work. Nor will it replace buying more bandwidth, although QoS may allow you to delay having to buy more bandwidth for a little while. The point to make is that QoS can protect important traffic, IF you have enough bandwidth for that traffic plus some left over for the remaining Best Effort Traffic.
You do have to tune what you demo to the bandwidth on the link. I suggest you not do demos on Gigabit Ethernet. You would probably need some costly power tools or several computers to generate enough traffic to congest it. Rather, work with fractional-T1 to say 6 Mbps links. This reflects most real-world WAN links today, except perhaps at corporate HQ or data center. Congesting links at this speed doesn't take a lot of effort, and yet you can run some fairly interesting multi-media traffic over them for demo purposes. Serial links are good for the bottleneck link, especially if you want to cause congestion by adjusting the clockrate downwards.
The following figure shows that you can do a demo with a rather minimal set of equipment. Using multiple laptops attached via a switch can help if you want to classify traffic based on IP address.
My definition of "Interesting" (as in "interesting demo") varies with speed. If you're working with a 128-256 Kbps link, forget video and stick to audio. Run some $10 handsets into FXS ports on a pair of 2600's and you can demo QoS interacting with VoIP. I've steered clear of mixing IPT (IP Telephony) and QoS demos, just due to complexity. With the VoIP approach, you can pre-configure and not have to fuss with the telephony side of things while demoing QoS. If you're working in the T1 or above range, consider video. For ideas, see below.
If your critical application is TCP-based or web-based, there's a way to make QoS impact visible there too. See below ("web").
Free Tool Toolkit
The starting point in building your own QoS demo(s) is to have a toolkit of free or inexpensive programs you can use. Here's what's in my bag of tricks now.
Cisco used to provide the lovely QDM tool for free, which you put into flash in your router. It allowed CBWFQ to be configured with a GUI, and provided graphs of the Cisco Class-Based QoS MIB. As you can probably tell from my prior article, I really liked this applet. Regretfully, Cisco declared it End of Life a while back.
Other things you might want to buy to use with this:
- 2 Cisco 1700, 2600 routers or better with 1 LAN and 1 WAN interface for QoS demo lab.
- 2 sets of FXS ports for the above (optional). Get two or three $5-10 analog handsets to use with them.
- Videocam(s) to use with NetMeeting on your PC. Two for $30.00 with rebate the other day.
- Microphone(s), ditto.
The newest Cisco IOS images don't fit into the 2600's unless they're the XM models (more RAM). So at some point, using the new 1800 or 2800 model routers may be a good idea. They certainly make a more flexible lab with better future-proofing.
Examining the free tools
||Description and Comments
||TTCP is code from the 80's by Terry Slattery (now of Netcordia) that measures maximum network throughput, minimizing disk I/O and other performance-inhibiting factors. Cisco Enterprise builds of IOS from 11.2 through about 12.2 included TTCP. I prefer TTCP on a laptop, possibly with windowing front end, for simplicity.
||Java version of the above, with simple windowing interface (requires getting the CLASSPATH environment variable right). The command "java ttcp" runs it if your environment is set up properly. Can also be run with CLI arguments. Java doesn't mean slow: one of our guys had it doing 700 Mbps on a medium-sized Solaris box with Gigabit interface. I've consumed most of a FastEthernet link with it myself.
||Windows version of TTCP.
||IPERF is like TTCP with enhancements. It can send TCP or UDP with various CLI settings.
||JPERF is a Java-based window front end for IPERF making it easier to figure out all the settings. At the end of a test run, JPERF produces a graph of the traffic levels during the run. You do need to put the files mentioned into your CLASSPATH.
Hint: not being a Java programmer, I had figured out that the directory containing .class files goes in the CLASSPATH. It then took some experimenting before figuring out that the actual .jar filename, not the directory it is in, also has to be in the CLASSPATH. This was the only real gotcha in trying to use JPERF.
||Free streaming video server. You do have to find MPEG trailer files on the web, or otherwise match the format(s) it understands. It took me some fiddling to get it working, but the price is right. VideoLAN can also be used as a multicast video source for a multicast lab! Firing off a moderate to high resolution video is also a great way to eat up 1 Mbps or more on a link. You might want a 384K video stream (i.e. small window or lower resolution) for demos with T1 or slower links.
|apache web server
||If you want a good webserver that's well-documented.
|Microsoft IIS or Personal Web Server
||You probably have this too. Use the help and it's pretty easy to figure out how to get IIS to serve up files or CGI scripts. (Ok, they left out the critical "%s %s" you need to get PERL working under IIS, but google found that tidbit for me rather quickly).
|Microsoft Web stress testing tools
||I haven't used this, but it's certainly a "web sucker" that pulls down web pages, useful if you need a background web load to congest a link.
|Special web pages
Hints: Change the image file size to use more or less bandwidth. Making copies of the files as "fast.*" and "slow.*" lets you demonstrate the Cisco NBAR feature.
||Free SNMP CLI software. Also available as a PERL package from CPAN. Allows you to create a simulated network management server, in case you want to guarantee some bandwidth to say SNMP, PING, telnet from certain servers. We've been doing this sort of thing lately to try to ensure we can manage the network despite a worm/virus storm or other congestive event. Note: you need to make sure you'll be able to telnet TO the central NMS box, if that's the one with protected telnet to network devices.
|PING or Cisco extended PING
||Network management software uses PING to track device availability. PING is also good for some extra loading when you need it, or another type of traffic (e.g. if you want something to match yet another class, or use ping to crudely demo packet drops).
||Add a pair of microphones and you can do G.711 VoIP between two laptops. Add videocams and you can do low-resolution video between two laptops. Since you can hear and see the speaker in the demo lab, any delays become utterance and display in NetMeeting are fairly noticeable.
||The great free packet capture and analysis tools. Vastly improved with versions 10.4 through 10.7. See below for a sample graph. Useful for seeing and documenting how bandwidth is being used in your demo lab.
|Java WAN simulator
||This is a small Java snippet of code, but not a bad idea. It listens on one port, forwards on another after a fixed delay. It would be rather easy to add random delays and / or packet drops. Run multiple copies to simulate delays on multiple ports. Or use a PERL or Tk wrapper?
Every programming shop should either have a pair of router or this code, for testing development tools or application WAN-readiness before deployment!
|cbqos + RRFW (aka Torrus)
||This might be used for graphs showing router queuing internals while demoing QoS.
RRFW or Torrus is nifty freeware, user-friendly MRTG/Cricket-like front end for RRD database and Net SNMP. It polls and then lets you pull up graphs via a web front end. The user friendly part is that web scripts do most of the work, whereas Cricket and MRTG require more editing of configuration files.
The cbqos plug-in adds Cisco CB QoS MIB support. We did have substantial difficulties recently trying to get this to work on Cygwin over Windows 2000. There were one or two snags building it with Fedore Core 3 Linux (default libpng package version, for example) but that went more smoothly.
If you've got some money for a "real" QoS lab, then you either want to introduce WAN delay and jitter simulation or load up on connections or traffic. There are a vast number of vendors and tools. Here are some we've noticed:
For example, if you're doing serious testing ability of an IPSec VPN box to hold up under a large number of connections or high traffic load, you might use something like one of Ixia's product. My earlier article on that particular topic was focussed more on the cheap "do it yourself" quick testing approach.
How to Build Your Demo
Start with only the key applications running. Make sure they use less bandwidth than your link has. Use Ethereal to determine average traffic level for each application. Another option: if you set up classification and marking, the "show policy interface" command will tell you the average traffic in each class. That tells you how much bandwidth to provide in your output QoS policy using priority and bandwidth commands.
Then add background traffic to overload the link. By adding or removing the "service-policy out policy-name" command from the relevant interface, you can ensure that the video or audio is bad until QoS is applied. Test thoroughly before demonstrating, because it may take a few iterations to get it working smoothly.
Hint: if you're using G.729 voice compression, it's such a small stream I found it hard to clobber with other traffic. G.711 uses more bandwidth and is easier to "step on".
I used a small PERL script to launch several applications at one time. This was mostly for convenience in starting the demo.
For what it's worth, I tend compose basic pages using Netscape Composer, then add scripting using WordPad. I've been avoiding MS Word, it adds way too much junk to HTML pages, trying to control appearance. That tends to make them impossible to hand-edit when you want to.
Here's a sample of the grapher in Ethereal. Every time I show this to someone who hasn't seen it, they say "where do you download that from", so it's certainly an eye-catcher. The graphs show traffic from capturing myself hitting www.cnn.com and www.cnbc.com with a web browser. The red and green graphs show me alternately clicking on refresh. The blue graph is a different address, my guess is a separate server for the graphics for one of the two sites.
This can be very useful for baselining a QoS demo lab, since you can put in a filter such as address or protocol ("http") and easily graph all the matching traffic. Caution: it's in Bytes / second, not bps, and you do have to switch it from Packets/second. I've never understood why anyone would want Packets/second, they rarely seem all that interesting to me.
This shows captured packets. If you capture inbound to a router and (separately) outbound, you can compare the two. The Cisco Class-Based QoS MIB does all this with one set of polls. The CBQOS tool mentioned earlier takes some work to build and get working, but displays this data. The commercial software I know of that reports on the CB QoS MIB is unfortunately costly.
Several of my prior articles discussed QoS. For configuring QoS, you'll want to use the newest cut at it, CBWFQ and the Modular QoS CLI. For more about this, see the article, New Quality of Service Features in Cisco IOS 12.1
If you don't have time to roll your own QoS demo, we'd of course be glad to come do it as a brief consulting engagement. It might also be useful to talk about QoS-readiness, and what your organization's QoS needs might be.
When we started experiencing problems with the CBQOS freeware, I wrote some PERL. I have a rough tool that offers you a set of choices, then polls CB QoS data and displays it in real time using gnuplot. This may or may not be the subject of a future article.
Your comments, questions, and suggestions for future articles are of course welcome! See below to decipher my email address.
Copyright (C) 2004, Peter J. Welcher