|The Stress Is On QoS|
|Saturday, 04 December 2004 21:00|
There are 10 (binary) kinds of people, those who have been reading these articles and those who haven't . The second group will probably never see these words. Those in the first group may have noticed I like to talk about things I've been doing. I hope this keeps the topics fresh and real-world. It certainly helps me out with having to do less preparation before writing. This month is another article relating to work I've been doing.
I and some of our other folks have been working on a QoS project for a large network. We're describing the architecture, the configurations, things to consider, initial testing and deployment, etc. in the customer's specific environment. One of the tasks was testing and demonstrating QoS in the lab. (Stress-testing QoS, hence the title). We put together a set of freeware and used it for this. Feedback suggests the demo is effective.
The original idea for this sort of thing is something I first saw with a Cisco SE in San Francisco, Eric Mar, who had a road show QoS Boot Camp a few years back. He was kind enough to run through it for me and some others. It provided the core idea for the DQOS course labs when I and some other folks wrote the initial outline. (No, I didn't write the actual course slides and materials). Since then, I've borrowed ideas from the now-retired DQOS course labs, and some other sources, including some quick and dirty demos I've done over time.
The key idea here is to vividly demonstrate the benefits of QoS. For Cisco, the intent was indubitably sales and showing off the new capabilities that differentiated Cisco routers. For you and me, the goal might be to demonstrate the value and efficacy of QoS to management, consulting customers, etc. If you want your management chain to allocate time, training, staff, or money for QoS, they need to understand what it does for the network and why its important.
Vivid here means either showing something or (as a fallback) listening to something. Seeing is believing. Dry numbers in show command output or some test tool window may be more informative, but are less effective, particularly with not-so-technical managers.
This article describes in general terms how you can put together your own QoS demo. I don't want to give away all our secrets and added value, but a lot of what I've been doing is something you can do on your own. This is also a great way to work with QoS in a lab, looking at the technical information and seeing how that correlates with what you're seeing or hearing.
The Basic IdeaThe way the demo works is to congest a link that carries loading traffic plus the demo traffic. You then show how ugly things are without QoS, and then apply QoS to the interface to make some things better. It can be particularly effective if you have two video streams or two phone calls, and only one is improved. That makes it evident that the congestion and packet drops are still present, but that QoS has protected the important traffic.
Important thing to note: QoS is not a panacea, and doesn't make up for not having enough bandwidth. QoS provides priority access to the bandwidth -- winners, which implies there are also losers. Don't let your boss leave the demo thinking QoS will make any amount of video or audio work. Nor will it replace buying more bandwidth, although QoS may allow you to delay having to buy more bandwidth for a little while. The point to make is that QoS can protect important traffic, IF you have enough bandwidth for that traffic plus some left over for the remaining Best Effort Traffic.
You do have to tune what you demo to the bandwidth on the link. I suggest you not do demos on Gigabit Ethernet. You would probably need some costly power tools or several computers to generate enough traffic to congest it. Rather, work with fractional-T1 to say 6 Mbps links. This reflects most real-world WAN links today, except perhaps at corporate HQ or data center. Congesting links at this speed doesn't take a lot of effort, and yet you can run some fairly interesting multi-media traffic over them for demo purposes. Serial links are good for the bottleneck link, especially if you want to cause congestion by adjusting the clockrate downwards.
The following figure shows that you can do a demo with a rather minimal set of equipment. Using multiple laptops attached via a switch can help if you want to classify traffic based on IP address.
My definition of "Interesting" (as in "interesting demo") varies with speed. If you're working with a 128-256 Kbps link, forget video and stick to audio. Run some $10 handsets into FXS ports on a pair of 2600's and you can demo QoS interacting with VoIP. I've steered clear of mixing IPT (IP Telephony) and QoS demos, just due to complexity. With the VoIP approach, you can pre-configure and not have to fuss with the telephony side of things while demoing QoS. If you're working in the T1 or above range, consider video. For ideas, see below.
If your critical application is TCP-based or web-based, there's a way to make QoS impact visible there too. See below ("web").
Free Tool ToolkitThe starting point in building your own QoS demo(s) is to have a toolkit of free or inexpensive programs you can use. Here's what's in my bag of tricks now.
Cisco used to provide the lovely QDM tool for free, which you put into flash in your router. It allowed CBWFQ to be configured with a GUI, and provided graphs of the Cisco Class-Based QoS MIB. As you can probably tell from my prior article, I really liked this applet. Regretfully, Cisco declared it End of Life a while back.
Other things you might want to buy to use with this:
Examining the free tools
If you've got some money for a "real" QoS lab, then you either want to introduce WAN delay and jitter simulation or load up on connections or traffic. There are a vast number of vendors and tools. Here are some we've noticed:
For example, if you're doing serious testing ability of an IPSec VPN box to hold up under a large number of connections or high traffic load, you might use something like one of Ixia's product. My earlier article on that particular topic was focussed more on the cheap "do it yourself" quick testing approach.
How to Build Your DemoStart with only the key applications running. Make sure they use less bandwidth than your link has. Use Ethereal to determine average traffic level for each application. Another option: if you set up classification and marking, the "show policy interface" command will tell you the average traffic in each class. That tells you how much bandwidth to provide in your output QoS policy using priority and bandwidth commands.
Then add background traffic to overload the link. By adding or removing the "service-policy out policy-name" command from the relevant interface, you can ensure that the video or audio is bad until QoS is applied. Test thoroughly before demonstrating, because it may take a few iterations to get it working smoothly.
Hint: if you're using G.729 voice compression, it's such a small stream I found it hard to clobber with other traffic. G.711 uses more bandwidth and is easier to "step on".
I used a small PERL script to launch several applications at one time. This was mostly for convenience in starting the demo.
// <!DOCTYPE html PUBLIC "-//w3c//dtd html 4.0 transitional//en">
// function doit()
// // wait time in msec
// setTimeout("document.location.reload(1)", 1000);
// // Flag is 1: force reload, no matter what user preferences are re cache
// // end hiding contents from old browsers
// // -->
// <body onload="doit()">
// <h2>Now is the time to load an image... </h2>
// <img src="/images/stories/welcher/bigimage4.jpg" title="" alt="big image 4"
// style="width: 400px; height: 500px;">
For what it's worth, I tend compose basic pages using Netscape Composer, then add scripting using WordPad. I've been avoiding MS Word, it adds way too much junk to HTML pages, trying to control appearance. That tends to make them impossible to hand-edit when you want to.
EtherealHere's a sample of the grapher in Ethereal. Every time I show this to someone who hasn't seen it, they say "where do you download that from", so it's certainly an eye-catcher. The graphs show traffic from capturing myself hitting www.cnn.com and www.cnbc.com with a web browser. The red and green graphs show me alternately clicking on refresh. The blue graph is a different address, my guess is a separate server for the graphics for one of the two sites.
This can be very useful for baselining a QoS demo lab, since you can put in a filter such as address or protocol ("http") and easily graph all the matching traffic. Caution: it's in Bytes / second, not bps, and you do have to switch it from Packets/second. I've never understood why anyone would want Packets/second, they rarely seem all that interesting to me.
This shows captured packets. If you capture inbound to a router and (separately) outbound, you can compare the two. The Cisco Class-Based QoS MIB does all this with one set of polls. The CBQOS tool mentioned earlier takes some work to build and get working, but displays this data. The commercial software I know of that reports on the CB QoS MIB is unfortunately costly.
SummarySeveral of my prior articles discussed QoS. For configuring QoS, you'll want to use the newest cut at it, CBWFQ and the Modular QoS CLI. For more about this, see the article, New Quality of Service Features in Cisco IOS 12.1.
If you don't have time to roll your own QoS demo, we'd of course be glad to come do it as a brief consulting engagement. It might also be useful to talk about QoS-readiness, and what your organization's QoS needs might be.
When we started experiencing problems with the CBQOS freeware, I wrote some PERL. I have a rough tool that offers you a set of choices, then polls CB QoS data and displays it in real time using gnuplot. This may or may not be the subject of a future article.
Your comments, questions, and suggestions for future articles are of course welcome! See below to decipher my email address.