Useful Debugs

There's a multitude of debugs on Cisco routers, below are some that I frequently find useful for VoIP troubleshooting:

Command	Purpose
debug ccsip messages	Shows the SIP messages being sent & received
debug gatekeeper call 5	Shows gatekeeper call processing, including bandwidth requests
debug gatekeeper main 5	Shows gatekeeper address resolution, including zone/prefix matching
debug h225 asn1	Shows H225 messages being sent & received
debug isdn q931	Shows ISDN Q931 messages, including calling & called number
debug ras	Shows summary of H225 RAS messages being sent & received
debug voice ccapi individual 2	Shows a summary of call setup messages regardless of protocol, including calling & called number
debug voip ipipgw	Shows the CUBE processing, such as when it needs to invoke a transcoder

Don’t forget to add voice iec syslog to your configuration too, this enables a lot of useful information to be logged.

UCCX Demo Licence

If you're building a lab, UCCX won't let you proceed through the initial set up without a valid licence file. Fortunately Cisco allow you to use a 30 day demo licence for any Standard, Enhanced or Premium, these are located in the DemoLicense folder on the install DVD. However if you don't have an install DVD or ISO to hand there is another way - you can download a UCCX update patch, open it with a tool like 7-Zip & extract the relevant demo licence file from it. Just remember that the licences are tied to the major UCCX version, so a UCCX 8.x install requires a UCCX 8.x licence file.

Troubleshooting Database Replication

Whilst wrangling with a particularly thorny database replication failure I found an in depth Cisco troubleshooting guide to database replication. Well worth a read if dealing with issues that utils dbreplication repair all doesn't fix.

HLog and 6900 Series Phones

Unfortunately the 6900 series phones don't have any visible notification of whether or not the phone is logged into hunt groups. This means that once the message displayed after pressing the HLog softkey disappears, there's no way of telling if your phone will take calls from hunt groups, other than to press HLog again. This is a rather bizarre omission given that queuing on hunt pilots is one of the big new features in CUCM 9.x & that in 9.0 it actually automatically HLogs you out if you don't answer a call from a hunt pilot with queuing enable. Thus it's easy to get into a situation where no one is receiving calls from hunt groups & they have no idea why.
Thankfully in CUCM 9.1 you can disable the auto HLog out of phones via "Automatically Logout Hunt Member on No Answer" within the line group settings. But this still leaves 6900 series phones with no indication of HLog status. The solution is to use a custom phone button template with one of the buttons being the HLog PLK, for example:

The LED on that button then indicates whether or not the phone is logged in to hunt groups. This is actually pretty handy for 7900 series phones too, as their "Logged out of Hunt Group" status message gets overwritten by the number of missed calls status message.

Installing Updates With Invalid Licensing

If you try to install an update on to CUCM without valid licences installed it will block it & display the error message "Upgrades are prohibited during the licensing grace period". When you're midway through doing physical to virtual migration & major upgrade this can be a pain as Cisco GLO can take up to 72 hours to respond to licence migration requests.
Fortunately there is a way round this to get the patches installed & then go back to Cisco for licence migration. You'll need to download a CentOS minimal install ISO & boot the CUCM server from it, then follow these steps:

1. Select rescue installed system, skip the media check & select the appropriate language, keyboard layout, etc.
2. Select local CD/DVD & don't start networking
3. Select continue to mount the file system & continue through the information messages. If in doubt about which partition is active, you can boot off the CUCM Recovery DVD & use it to list the partition layout
4. Start the shell

From the shell run the following commands:

chroot /mnt/sysimage
cp /usr/local/platform/conf/licexpiry.txt licexpiry.txt.bak
rm /usr/local/platform/conf/licexpiry.txt

Next you need to stop SELinux from choking on the changes made to the filesystem, so we need turn off enforcing. Run the following command to locate the grub.conf files:

find ./ -name "grub.conf"

Now use vi to edit the grub.conf files & add enforcing=0 to the end of the line which says kernel, for example:

kernel /boot/vmlinuz-2.6.18-274.12.1.el5PAE ro root=LABEL=/partB clock=pmtmr divider=4 crashkernel=128M@16M enforcing=0

Now you can exit to reboot the server. Without licexpiry.txt present you can continue installing patches to get the desired version of CUCM.

Upgrades from 6, 7 & 8 to 9 Best Practices

Handling an upgrade to CUCM 9.x involves wrangling with half a dozen different documents, such as the compatibility guide, supported servers matrix and the relevant version's upgrade guide. The process is made all the more complicated if you need to migrate from a physical MCS server to a virtualised install. Fortunately Cisco have written a succinct guide to the best practice for an upgrade, including physical to virtual migration: Operational Guide for Cisco Unified Communications Manager

Troubleshooting Lack of Music on Hold

A quick run down of common causes for not receiving music on hold, symptoms include tone on hold or silence.

Multicast MoH
Multicast routing must in place across the entire path to the endpoint receiving the MoH stream, the MoH server in CallManager must set the TTL value to a high enough value to reach the endpoint, the default is 2 hops.
If the endpoint is connecting via the PSTN, the PSTN gateway must have the "ccm-manager music-on-hold" command configured to stream multicast MoH to the PSTN interfaces.

MoH Settings
No MoH configured for the device putting the endpoint on hold, or invalid MoH source configured.

Media Resources
The MoH server selected by the endpoint's MRGL lacks the MoH source file, as these must be uploaded to each server separately.

Unicast MoH
The location that the endpoint and/or MoH server reside in has reached the maximum specified bandwidth, therefore the MoH stream is rejected by call admission control (CAC). Note that Multicast MoH is exempt from CAC.

Region
The region configuration for the endpoint and MoH server selected a CODEC that MoH isn't transmitted in. By default MoH is only sent in G711 format, you must enable G729 under the IP Voice App Streaming service parameters.

Access Lists
There must be no access lists that block the multicast MoH IP address (e.g. 239.1.1.1), or traffic from the MoH server's IP address in the case of unicast MoH.