UCCX Demo Licence

If you're building a lab, UCCX won't let you proceed through the initial set up without a valid licence file. Fortunately Cisco allow you to use a 30 day demo licence for any Standard, Enhanced or Premium, these are located in the DemoLicense folder on the install DVD. However if you don't have an install DVD or ISO to hand there is another way - you can download a UCCX update patch, open it with a tool like 7-Zip & extract the relevant demo licence file from it. Just remember that the licences are tied to the major UCCX version, so a UCCX 8.x install requires a UCCX 8.x licence file.

Troubleshooting Database Replication

Whilst wrangling with a particularly thorny database replication failure I found an in depth Cisco troubleshooting guide to database replication. Well worth a read if dealing with issues that utils dbreplication repair all doesn't fix.

HLog and 6900 Series Phones

Unfortunately the 6900 series phones don't have any visible notification of whether or not the phone is logged into hunt groups. This means that once the message displayed after pressing the HLog softkey disappears, there's no way of telling if your phone will take calls from hunt groups, other than to press HLog again. This is a rather bizarre omission given that queuing on hunt pilots is one of the big new features in CUCM 9.x & that in 9.0 it actually automatically HLogs you out if you don't answer a call from a hunt pilot with queuing enable. Thus it's easy to get into a situation where no one is receiving calls from hunt groups & they have no idea why.
Thankfully in CUCM 9.1 you can disable the auto HLog out of phones via "Automatically Logout Hunt Member on No Answer" within the line group settings. But this still leaves 6900 series phones with no indication of HLog status. The solution is to use a custom phone button template with one of the buttons being the HLog PLK, for example:

The LED on that button then indicates whether or not the phone is logged in to hunt groups. This is actually pretty handy for 7900 series phones too, as their "Logged out of Hunt Group" status message gets overwritten by the number of missed calls status message.

Installing Updates With Invalid Licensing

If you try to install an update on to CUCM without valid licences installed it will block it & display the error message "Upgrades are prohibited during the licensing grace period". When you're midway through doing physical to virtual migration & major upgrade this can be a pain as Cisco GLO can take up to 72 hours to respond to licence migration requests.
Fortunately there is a way round this to get the patches installed & then go back to Cisco for licence migration. You'll need to download a CentOS minimal install ISO & boot the CUCM server from it, then follow these steps:

1. Select rescue installed system, skip the media check & select the appropriate language, keyboard layout, etc.
2. Select local CD/DVD & don't start networking
3. Select continue to mount the file system & continue through the information messages. If in doubt about which partition is active, you can boot off the CUCM Recovery DVD & use it to list the partition layout
4. Start the shell

From the shell run the following commands:

chroot /mnt/sysimage
cp /usr/local/platform/conf/licexpiry.txt licexpiry.txt.bak
rm /usr/local/platform/conf/licexpiry.txt

Next you need to stop SELinux from choking on the changes made to the filesystem, so we need turn off enforcing. Run the following command to locate the grub.conf files:

find ./ -name "grub.conf"

Now use vi to edit the grub.conf files & add enforcing=0 to the end of the line which says kernel, for example:

kernel /boot/vmlinuz-2.6.18-274.12.1.el5PAE ro root=LABEL=/partB clock=pmtmr divider=4 crashkernel=128M@16M enforcing=0

Now you can exit to reboot the server. Without licexpiry.txt present you can continue installing patches to get the desired version of CUCM.

Upgrades from 6, 7 & 8 to 9 Best Practices

Handling an upgrade to CUCM 9.x involves wrangling with half a dozen different documents, such as the compatibility guide, supported servers matrix and the relevant version's upgrade guide. The process is made all the more complicated if you need to migrate from a physical MCS server to a virtualised install. Fortunately Cisco have written a succinct guide to the best practice for an upgrade, including physical to virtual migration: Operational Guide for Cisco Unified Communications Manager

Troubleshooting Lack of Music on Hold

A quick run down of common causes for not receiving music on hold, symptoms include tone on hold or silence.

Multicast MoH
Multicast routing must in place across the entire path to the endpoint receiving the MoH stream, the MoH server in CallManager must set the TTL value to a high enough value to reach the endpoint, the default is 2 hops.
If the endpoint is connecting via the PSTN, the PSTN gateway must have the "ccm-manager music-on-hold" command configured to stream multicast MoH to the PSTN interfaces.

MoH Settings
No MoH configured for the device putting the endpoint on hold, or invalid MoH source configured.

Media Resources
The MoH server selected by the endpoint's MRGL lacks the MoH source file, as these must be uploaded to each server separately.

Unicast MoH
The location that the endpoint and/or MoH server reside in has reached the maximum specified bandwidth, therefore the MoH stream is rejected by call admission control (CAC). Note that Multicast MoH is exempt from CAC.

Region
The region configuration for the endpoint and MoH server selected a CODEC that MoH isn't transmitted in. By default MoH is only sent in G711 format, you must enable G729 under the IP Voice App Streaming service parameters.

Access Lists
There must be no access lists that block the multicast MoH IP address (e.g. 239.1.1.1), or traffic from the MoH server's IP address in the case of unicast MoH.

German (Overlap Receiving) ISDN With CME

In Germany they use overlap sending & receiving on ISDN circuits, meaning that call setup messages don't contain the entire dialled number (i.e. en bloc dialling), instead the dialled digits will be split across several call setup messages. This is complicated by the fact that this isn't done for all calls - calls from international numbers & mobiles use en bloc dialling.
The Cisco configuration guide for overlap receiving says to enable it on the ISDN interface, disable direct-inward-dial & use T in your dial-peers to force a T302 timeout. However all the documentation I've seen covers a voice gateway with inbound & outbound dial-peers with timeout, not CME. This causes a problem as having just an inbound dial-peer with timeout doesn't result in digits after the first call setup message being captured, as the virtual dial-peers for the phones registered with CME don't have a timeout. The solution is to use a voip dial-peer pointing to the CME router to loop back the call, then translate the dialled number so that the phone's virtual dial-peer is matched.
Below is example configuration using a BRI interface, where the direct dial range is 8693920X & internal numbers are 920X. After experimentation I found the 6000ms value for the overlap receiving T302 timeout to work the best (the default is 10000ms).

interface BRI0/1/0
 no ip address
 isdn switch-type basic-net3
 isdn overlap-receiving T302 6000
 isdn point-to-point-setup
 isdn incoming-voice voice
 isdn send-alerting
 isdn sending-complete
 isdn static-tei 0
!
voice translation-rule 1
 rule 1 /^8693\(920.\)/ /\1/
voice translation-profile BRI_Called
 translate called 1
!
dial-peer voice 3000 pots
 description BRI0 inbound
 incoming called-number 869392T
 port 0/1/0
dial-peer voice 3001 voip
 description Loopback for overlap receive T302 timeout
 translation-profile outgoing BRI_Called
 destination-pattern 869392T
 session target ipv4:172.23.12.130
 incoming called-number 92..
 dtmf-relay h245-alphanumeric
 codec g711ulaw
 no vad

Looking at the debug isdn q931 output, we can see the first call setup message arrive, followed by the 2nd containing the last digit & then the call connects correctly:

001218: May  3 11:49:04.136: ISDN BR0/1/1 Q931: TX -> RELEASE pd = 8  callref = 0x2B
001219: May  3 11:49:04.188: ISDN BR0/1/1 Q931: RX <- RELEASE pd = 8  callref = 0xAB
001220: May  3 11:49:20.569: ISDN BR0/1/0 Q931: RX <- SETUP pd = 8  callref = 0x01
        Bearer Capability i = 0x9090A3
                Standard = CCITT
                Transfer Capability = 3.1kHz Audio
                Transfer Mode = Circuit
                Transfer Rate = 64 kbit/s
        Channel ID i = 0x89
                Exclusive, B1
        Progress Ind i = 0x8283 - Origination address is non-ISDN 
        Calling Party Number i = 0x00A3, N/A
                Plan:Unknown, Type:Unknown
        Called Party Number i = 0xC1, '8693920'
                Plan:ISDN, Type:Subscriber(local)
001221: May  3 11:49:20.573: ISDN BR0/1/0 Q931: TX -> SETUP_ACK pd = 8  callref = 0x81
        Channel ID i = 0x89
                Exclusive, B1
001222: May  3 11:49:21.125: ISDN BR0/1/0 Q931: RX <- INFORMATION pd = 8  callref = 0x01
        Called Party Number i = 0xC1, '0'
                Plan:ISDN, Type:Subscriber(local)
001223: May  3 11:49:27.149: ISDN BR0/1/0 Q931: TX -> CALL_PROC pd = 8  callref = 0x81
001224: May  3 11:49:27.193: ISDN BR0/1/0 Q931: TX -> ALERTING pd = 8  callref = 0x81
        Progress Ind i = 0x8188 - In-band info or appropriate now available
001225: May  3 11:49:29.297: ISDN BR0/1/0 Q931: TX -> CONNECT pd = 8  callref = 0x81
        Channel ID i = 0x89
                Exclusive, B1
001226: May  3 11:49:29.377: ISDN BR0/1/0 Q931: RX <- CONNECT_ACK pd = 8  callref = 0x01
001227: May  3 11:49:29.381: %ISDN-6-CONNECT: Interface BRI0/1/0:1 is now connected

Communications Manager Tools Part 2

Troubleshooting using SDI/SDL traces from CUCM is a pain, in part because the tool within RTMT for doing this is pretty lame. Cisco do make the Voice Log Translator, which tidies up the presentation of traces & provides explanations of many of the fields, very useful! Unfortunately it's not been updated since 2010 & when I tried using it with CUCM 9.1 recently it didn't seem to understand many of the traces. Fortunately there are a couple of alternative tools for wrangling with traces, both made by Cisco employees: TranslatorX & Triple Combo.

Also, don't forget to give the tools from Communications Manager Tools Part 1 a go.

ATA 186 & 187 2nd FXS Port

On the now discontinued ATA 186 & the current ATA 187, to use the 2nd FXS port you have to create another device, take the MAC address & tack 01 on the end & remove the first 2 digits. For example 54781A1D7D43 becomes 781A1D7D4301.

Jabber Video for Telepresence & AnyConnect VPNs

I recently spent some time troubleshooting a customer's video call quality issues when their staff working from home on an AnyConnect VPN make video calls using Jabber Video for Telepresence. The symptoms include low resolution being used & frequent periods of high packet loss (upto 70%) reported by Jabber Video, despite fast Internet connections in use at both ends. After much head scratching it turns out Jabber Video doesn't get on with SSL VPNs, there's an official notice from Cisco on the matter here. My testing shows that both IPsec & L2TP/IPsec VPNs don't suffer from this problem.

Unhelpful Error Messages

This error message appeared after trying to restart a CUCM server via the OS Administration GUI. Where is the Save button that it refers to?!