Chesapeake NetCraftsmen

Tracking syslog events is very important in network operations.  But the syslog stream at many sites is often chatty enough that a few important events may be easily overlooked.  I became aware of a neat perl script that summarizes syslog through Phil Koontz.  Phil found the neat script on the Internet, but doesn't recall where.  Several people have done minor modifications to it over time, but it still has the original characteristics, which is to summarize the events of the day in a way that makes it easy to find key events.  The script and supporting files may be downloaded from the links at the end of this page.

Below is an example of the syslog summary.  The first section is the summary by message type.  This is where you can find the low frequency, but critical message that tells you about impending disaster.  An example is Pinnacle errors, which is an indication of an ASIC going bad on either the Supervisor module or on a line card in a Cisco Cat 6500.  I have found power supply or fan problems here that other NMS systems failed to report.  Also look at the few most common messages.  The OSPF-5-ADJCHG messages are likely due to changes in the line protocol changes.  Note that in the example there were two line protocol changes when a link went down and then back up.  There were then nine corresponding  OSPF adjacency changes, one for each line protocol transition.

The second section is the frequency, device and interface summary.  This section tells you which devices and interfaces generated the counts that appear in the first section.  A useful addition to the detail section would be the description on each interface that is reporting a problem.  If you have good interface descriptions, this would help you quickly rank the importance of the interfaces without having to go elsewhere to look it up.  Once you identify an important message, you'll need to go to the syslog logging file to see any additional messages that might be associated with the individual log messages.  You may want to have this summary emailed to you daily, which is handy if you receive them on a smart phone and have the opportunity to scan them at your leisure.

==========

Summary of Cisco syslog Messages on    Sun Oct 11 23:59:01 2009

Cisco Messages:

18  LINEPROTO-5-UPDOWN
9  OSPF-5-ADJCHG
7  SNMP-3-AUTHFAIL
2  BGP-5-ADJCHANGE
2  LINK-3-UPDOWN
1  BGP-3-NOTIFICATION

Messages sorted by frequency and source device:

8       d04-3550-03      d04-3550-03       LINEPROTO-5-UPDOWN FastEthernet0/13
4       d19-3400-01      d19-3400-01       LINEPROTO-5-UPDOWN FastEthernet0/19   
2       d02-2811-01      d02-2811-01       SNMP-3-AUTHFAIL
2       d03-2811-01      d03-2811-01       SNMP-3-AUTHFAIL
2       d45-3560-01      d45-3560-01       LINEPROTO-5-UPDOWN GigabitEthernet0/17  
2       d19-3400-01      d19-3400-01       LINK-3-UPDOWN FastEthernet0/19
2       d48-7604-01      d48-7604-01       OSPF-5-ADJCHG
2       d16-7604-01      d16-7604-01       BGP-5-ADJCHANGE
2       d16-7604-01      d16-7604-01       SNMP-3-AUTHFAIL
2       d64-3550-05      d64-3550-05       LINEPROTO-5-UPDOWN FastEthernet0/2
2       d22-7604-01      d22-7604-01       OSPF-5-ADJCHG
1       d14-6504-01      d14-6504-01       OSPF-5-ADJCHG
1       d38-7604-01      d38-7604-01       OSPF-5-ADJCHG
1       d38-7604-01      d38-7604-01       SNMP-3-AUTHFAIL
1       d89-3560-01      d89-3560-01       LINEPROTO-5-UPDOWN Vlan3264
1       d89-3560-01      d89-3560-01       OSPF-5-ADJCHG
1       d16-7604-01      d16-7604-01       BGP-3-NOTIFICATION
1       d27-3560-01      d27-3560-01       LINEPROTO-5-UPDOWN Vlan3265
1       d27-3560-01      d27-3560-01       OSPF-5-ADJCHG
1       d92-3560-01      d92-3560-01       OSPF-5-ADJCHG

=============

Co-worker Pete Welcher incorporated some modifications to the script and created the CGI scripts that allow the summary to be accessed via a web page.   We've included them in the archive file in case you want to use them too.  The unpacked directory hierarchy contains the following files.  Note that the cgi-bin files will need to have the path modified to match where you install the perl summarization script.

summarize-nmslog2.pl: The summarization perl script.  It could be improved to take an argument to allow it to run for 'yesterday' and automatically calculate the start and stop days.  When run at 23:59, it may miss events that occur in the last minute of the day on a busy log (it depends on system load and how long it takes the script to startup to know how much it will miss).  See the Usage message in the script for arguments.

summarize-nmslog.pl.orig: The original summarization perl script from Phil Koontz.  It won't work with some of the web links below because it doesn't support the arguments that the cgi scripts use nor the calculation of time deltas.

www/html/summarize-nmslog.html:  An html file that provides web access to different views of the syslog summary.

www/cgi-bin/syslog-lasthour.pl: shows syslog summary for the last hour.  It depends on the Date::Calc perl library to calculate the time differences for displaying log messages between specifics times - something that the original syslog summary script didn't do.

www/cgi-bin/syslog-tail-f.pl: An automatically refreshing page that shows the last few syslog messages.  For an active syslog server, it may miss some events.

www/cgi-bin/syslog-today.pl: Shows the syslog messages for today, up to the current time.

www/cgi-bin/syslog-yesterday.pl: Shows the syslog messages for yesterday.  It depends on the crontab entry to run summarize-nmslog2.pl each day.

crontab: A crontab entry for running sumarize-nmslog2.pl.  It could be improved to run for 'yesterday' and run it at 00:01 each day.  When run at 23:59, it will miss the events that happen in the last minute of the day.

And finally, the scripts themselves, in a zip archive: summarize-syslog.zip