| Audit Logs on Cisco Unified Communications Manager (CUCM) |
|
Background
I will keep the background discussion to a minimum, because I am sure most readers of this blog understand the need for an auditing solution. To summarize, admins need a way to look at an audit trail for administrative changes that occur in their CUCM cluster. Basically, in large organizations there is a need to look at "what has changed" so that if there is an issue reported that is related to human error, the error can be quickly identified and fixed. There is also an accountability aspect as it is likely that senior support folks want to identify education gaps in their lower tier support teams. Software Revisions The ability to enable auditing for administrative tasks was included in CUCM version 7.1(2) and later. In my lab, I tested this feature using a CUCM 7.1(3b)SU2 cluster. Enabling Audit Logs By default, audit logs for administrative level tasks are enabled. To access this feature, you will need to browse the audit log parameters from Cisco Unified Serviceability. The following procedures can be used to accomplish this task: 1. Connect to Cisco Unified Serviceability on your CUCM cluster: https://mypublisher/ccmservice/ 2. Go to Tools>Audit Log Configuration
3. Depending on your particular operational model, you may have people with different levels of authorization. Authorization levels are implemented via the Groups and Roles configuration objects. If you have a need to customize roles or Group/Role assignments you will want to look at the "Standard Audit Log Administration" role to understand the level of access required for users/groups that need to look at audit logs.
Accessing Audit Logs At this point, you have enabled audit logs for administrative tasks. As with other logs and trace files on the system, you can access these logs in several ways. For this example, we will walk through using RTMT to view audit logs. 1. Load RTMT on your admin workstation 2. Connect to your CUCM cluster using a user ID that has the "Standard Audit Log Administration" role (or equivalent) assigned. 3. Go to System>Tools>Trace>Trace and Logs Central 4. Under "Trace and Logs Central", you will see an "Audit Logs" folder. Double click the folder to start the log collection wizard.
One of the benefits in using RTMT is that you can collect and download AuditLogs either as a one time request or you can schedule the download. For those that like to run pro-active analysis (via scripts and the like) you may want to look at the "Schedule Download" option. As with other traces/logs, once you specify the search criteria RTMT will provide a folder tree structure that can be used to navigate to the audit logs stored on the server.
Deciphering the Audit Log The log file entries are lengthy but parsing them is not at all difficult. Let's look at an example. I have a dummy phone on my CUCM cluster. The phone was already created prior to testing the audit log feature. What I did was add a phone number to the device. In the AuditLog this action is captured as follows: 04/06/2010 11:03:15.295 |LogMessage Now, the log entry itself will be a single line in the trace file but I have broken it out in something a tad more legible. You will see that the log identifies the user who performed the change (UserID), the IP address of the machine the user was using (ClientAddress) along with other pertinent information. Of particular interest is the AuditDetails section of the logged event. In this example, we can see that I added a directory number. Before dissecting this section, we should address an obvious question: which device was this performed on? This is where things are tricky. If you look at previous log entries, you will find when the user (ccmadministrator in our example) connected to the CCMadmin web portal. However, the AuditLog will not track this user's movements through the various web pages. So, if I clicked on a device, and then clicked on "Add new line". Those events are not recorded. Only when I actually "save" the change will it be recorded in the AuditLog. Why? Well, it boils down to the fact that the Audit Log is tracking changes to the database. It's focus is on what has changed in the database, not what has changed in the admin interface. This is actually a necessary distinction, since changes to the database can come from a variety of sources: BAT, CCMAdmin, AXL/SOAP (e.g. UCCX creates a JTAPI trigger). If you look at our example above, you will see that the AuditDetails says: record in table numplan with key field dnorpattern=2025552222 added Numplan is a database table and if we wanted to find out what device this number was added to, we would need to take a few extra steps. One step could be to run an ad hoc query from the command line to see what gives: admin:run sql select d.name as device,n.dnorpattern,dmap.numplanindex From this output, we have identified the device and the line appearance the new number is assigned. Yeah, I know the query is ugly. I can't resist you see, I do most of my admin tasks via custom queries. That's just me. You could certainly figure this out via the CCMAdmin web pages by going to Call Routing>Route Plan Report. Then search for the directory number "2025552222". So, you still have some work to do when you want to analyze the audit log, but the data now available is a great improvement over what you would have to do in the past. I won't even waste your cycles on that discussion. Other Things That Are Logged As you play around with the Audit Log, you will find that it captures other interesting things as well. Such as commands entered by users on the CLI. This is handy if you want to find out who restarted the TomCat service or who reloaded the server. Small things like that. Of course, if you want that data to have any meaning, you may want to investigate the CLI command: set account name. The Audit Log will also capture events like a user uploading a custom banner page to the CUCM system. The custom banner page is another new security feature added to CUCM 7.1(2). Where is the Audit Log For those folks who don't dig on using RTMT for certain tasks, and would just rather download the file or view it directly on the console I have two pieces of information: 1. The files are stored here: activelog audit/AuditApp/* 2. If you go this route, I recommend downloading the files to your workstations and then viewing/parsing/filtering. Especially during production hours. Conclusion Well, the AuditLog feature isn't 100% perfect as you still have to tear yourself away from CNN or ESPN to actually do some work. But, it is getting there and the information that is recorded in the AuditLog is very handy when you need to determine the "how", "when", and "why". Hmmm, maybe it won't tell you the "why" but it sure will point you to the person that can!
Comments (15)
  written by William Bell, April 14, 2010
Steve,
Sorry for the delay in getting back to you. I missed this comment in my e-mail stream. The audit logs outlined in the article can be used for this purpose. Back in the day, I would use the IIS logs (pre-appliance model) to do something similar but that was a chore. You can see similar data with tomcat. On the CUCM system (usually the publisher node) you can view a daily access record for the tomcat service: admin:file list activelog tomcat/logs/localhost* detail You can then view one of these files to see if it is close to what you need. I am not sure if there is any more value in tomcat that the audit logs don't already provide but you can determine what best meets your requirements. (NOTE: You can also browse the tomcat logs using the same methods described in this article). Now, if you are looking for a way to pull or push the traces off box and then post process using perl or something similar, then you have a few hoops to jump through. You can push tomcat traces to a syslog server. Take a look at the CUCM Serviceability configurations (Alarm configurations). Tomcat traces can be configured under the platform services for your server. I am not sure if there is a way to push auditlogs to a syslog server but you can pull them on a scheduled basis using RTMT. There are other methods, but using RTMT is pretty straight forward. Regards, Bill
 written by Sree, July 21, 2010
Hi Steve,
i just started working with the CUCM 6.0, i need your valuable advice regarding how and where i can get the logs of the daily activities and where can i get the information about the root cause for a crash in CM, do i have to configure anything before for getting this or will it be available by default. It may seem like a simple thing but i don't have much idea about this as i am a total fresher to this work.So kindly help me out in this. Thanks in Advance Sree
written by William Bell, July 22, 2010
Sree,
You may want to look through the CUCM troubleshooting guide: http://www.cisco.com/en/US/doc.../trbl.html If you are new to CUCM and you are trying to determine root cause of a system crash then I recommend opening a Tac case. HTH. Regards, Bill
 written by Davorin, August 18, 2010
Bill -
I was wondering if you have noticed any changes with CPU usage on the Call Manager server after you have turned on this audit logging. I am trying to avoid turning on logging on my production server and then have the CPU jump to very high usage and do affect my production calls etc. Thanks, Davorin
written by William Bell, August 31, 2010
Davorin,
No, I have not noticed an increase in CPU utilization. I/O would be the area I would check before CPU utilization. I haven't seen a problem their myself. Though, I have not studied that aspect in detail. Sorry for the delay in responding. I have been off radar for a while. HTH. Regards, Bill
 written by Guy Kelly, September 16, 2010
Hi William,
Actually this article is incorrect. The logs you are enabling above are the database logs and are filed under the informixauditlogs on RTMT. Application audit logs are enabled by default and are the checkboxes listed in the Application Audit Log Settings above. Cheers Guy Kelly
written by William Bell, October 27, 2010
Guy,
Yup, you are absolutely correct. I have nothing to say except thanks. I have made the correction and plan to come back and expand upon blog later. Thanks, Bill
 written by Akos, October 29, 2010
Many thanks for this post it is very useful.
My question is if there is a similar way to do auditing on the CUP server too? I created a test user on the CUP server but I could not logon to the web administration interface until I added the test account to the “Standard CUP Super Users” Group. The issue is that this user group assignment gives more rights which is needed by us. Basically I just want to limit what different users can do and audit their activity. I found the following document but this has not fully answered my questions: http://docwiki.cisco.com/wiki/Cisco_Unified_Presence,_Release_7.x_--_Managing_Users_on_Cisco_Unified_Presence Many thanks in advance Akos
 written by Mohammed Riyas AT, February 24, 2011
Hi Bill,
Is there any way to get the audit logs from CCM 6.1.5 because i cannot see the option for audit logs in RTMT in my CCM. We have one billing system integrated to our CUCM making changes to the extensions on the phones unnecessarily, But the vendor is asking for logs or traces taken from the CCM proving the changes were made by the billing application and not the CCM administrators. If it is possible please let me know how can i get the traces. I tried with many traces in RTMT but did not get anything useful regarding this.
written by William Bell, February 24, 2011
Mohammed,
AFAIK 6.1(5) does not incorporate the features discussed in this blog. The features discussed here were added in 7.1(2). There is an audit facility in 6.1(5) but I don't think it captures the web-based admin activities. However, you may be able to find what you need in the tomcat logs. It will be a little more tedious and I can't guarantee you will find what you are looking for. The tomcat logs are stored in tomcat/logs/. You can see the pertinent log files using the following command: file list activelog tomcat/logs/localhost* detail There should be several files sorted by date. The content of these files is very chatty. But you can find edit events. For example, here is one from one of my lab systems: [24/Feb/2011:09:40:51 -0500] 192.168.1.250 192.168.1.250 wjbell - 8443 GET /ccmadmin/lineGroupEdit.do ?key=649448a2-2dbe-ed84-17f3-2c901d60f4ec&order=1&devkey=0cfea004-ea65-6e77-4479-6b3a97ce2061 HTTP/1.1 200 73598 1519 The client in this case has the IP address 192.168.1.250 and the user account is wjbell. So, you can use this information to "grep" through the files. From the command line: admin:file search activelog tomcat/logs/local* wjbell Will bring back a slew of output (not all of it useful). You will still have some parsing to do and may want to pull the tomcat logs off box so you can parse easier/quicker. If you had an idea on what was actually changed then you could look for the pkid (database) of the object that was modified and then search the tomcat logs for that pkid. In the example log above that crazy long hex string (?key=649448a2-2dbe-ed84-17f3-2c901d60f4ec) is the pkid for whatever line group (linegroupedit.do) I was looking at. Let's say you had an issue with a specific line group. You would know the name of that line group and you can find the key as follows: 1. You can open CCMAdmin and browse to the line group. Look at the URL in your web browser and you will notice the key is plain as day. 2. You can run an ad hoc query on the command line against the linegroup table: admin: run sql query select pkid,name from linegroup where name = 'mylinegroup' Either way, you will find what pkid/key matches what linegroup. You can then search the tomcat logs for that pkid. Even better, search for that pkid and the user ID you assigned to your CDR application. Hopefully, you created a special user for your CDR app to make things like this easier. On that note, I always recommend that customers avoid over using the generic administrator account assigned to the CUCM cluster. Each human should have their own account and it should be different than their regular user account. Further, each application should have their own account as well. This will help with accountability because it is easier to search. You could also differentiate account capabilities with different authorization levels. For example, you could customize the CDR account (in your case) to have only the permissions it needs to do its job. You could give the CDR account permissions to Read stations/lines but not write. Then the question of whether the CDR account is the culprit never arises. HTH. Regards, Bill
 written by Vladimir, May 12, 2012
Hi!
Thank you for this very usefull article! Is there any way to delegate to any user a permission to manage only some pool of phones? In large organisation with many Regions anf offices we have to give to local admins roles: Standard CCM end users Standard CCM phone management Standard CCM user management Every thing works fine, local admin can add and change phones in his office. The problem that ye can can change and delete ANY phone in Organization. Is there any way to resolve it? Thanks Vladimir
written by William Bell, May 12, 2012
Vladimir,
The behavior you are observing is by design. The CUCM authorization model only provides for what web pages you can access and the permission level on those pages. There is no way you can provide for a mult-tenant admin model using the native CUCM software. There are third-party applications that address this need. I haven't looked at Cisco's Unified Provisioning Manager, but it may provide what you are looking for (and then some). You could also look at Unimax, 2Ring, and similar partners. I cannot recommend a solution on this site. Perhaps query: "cucm multi-tenant provisioning" or "cucm multi-tenant administration" in Google. HTH. Regards, Bill
 written by ahmed, September 12, 2012
commenting on response of :William Bell, February 24, 2011 ,
Really this is good info ,,,pls include some examples sql commands , i appreiciate for good work 
written by William Bell, September 12, 2012
Ahmed,
I have a series that goes through various SQL queries. This series is hosted on my personal blog: http://ucguerrilla.com. HTH. -Bill (@ucguerrilla)
Write comment
|
Ever since I started working with Cisco Call Manager (circa 1999 - yikes) I have always wanted a way to implement an Authentication/Authorization/Accounting model. Now, I finally got everything I wanted. The Authentication/Authorization need was addressed somewhere around CM 3.1.4b with a web application add-on called Multi-Level Access (MLA). Since the initial release, MLA has been incorporated into the CM software (starting around CM 4.1.3). With 5x/6x/7x the MLA term has gone by the wayside but the feature is alive and well (and quite robust). Now, with CUCM 7.1(2) we have a decent Accounting or Auditing solution available.
Subscribe to this comment's feed
Humbly,
Steve Sutton