New Subscriber Installation Failing With 10.5 After DRS Restore

I recently encountered this bug in 10.5 which breaks the ability to install subscribers after a DRS restore. As Cisco's official P2V method is to do a DRS backup from the physical machine & then a DRS restore to an identically configured virtual machine, this largely applies to upgrade projects where you have to P2V CUCM as it was running on a physical MCS 7800 series server.

With the 10.5.2.12901 upgrade I was working on, the subscriber installation would get stuck after validating network connectivity to the publisher with a very helpful error message:

There is a bug report (CSCus35964) for this & a patch ciscocm.Fix_DRS_Recovery_v1.0.k3.cop.sgn that must be installed on the publisher, then the publisher rebooted.

4300 Series Routers Bugs

Having deployed a number of the new 4331 routers to replace 2800 series routers as voice gateways, there's been some quirks...

The first being that unicast music on hold stopped working. Identical configuration to the 2800 routers, but PSTN callers just hear silence. Eventually found that setting the CallManager service parameter Duplex Streaming Enabled to true fixes this problem:

I also hit a nasty bug in that 4000 series IOS XE releases 3.15.0, 3.15.1 & 3.16.0 aren't fit for purpose as voice gateways, whenever the router reloads the dial-peer configuration (along with NBAR & some ACLs) disappears. Why these versions aren't deferred releases yet I don't know?!

UPDATE: Now to cap it all off the 4300 series is undergoing a recall.

UCCX Editor Script Repository and Reactive Debugging Not Working

In UCCX 10.0 Cisco decided to change the way UCCX works so that hostnames not IP addresses are used for inter-server communication. The UCCX Editor was also changed to use hostnames, resulting in the odd situation where you can authenticate against the UCCX server by IP address, but browsing the script repository or using reactive debugging doesn't work. This is because the UCCX Editor will try to use the hostname of the UCCX servers, if the DNS A records don't exist or you're connected via a VPN without correct DNS configuration then the DNS resolution will fail & these features won't work. The error message provided is pretty vague on this, for example if you try to use reactive debugging it will get stuck with the message "Engine hostname list is empty. Check the ldap for enabled unified ccx engines".
The solution is to kludge it by editing your local hosts file to contain the hostname & FQDN of the UCCX server(s). On Windows edit the file C:\Windows\System32\drivers\etc\hosts, example entries below:

10.15.0.5      perry
10.15.0.5      perry.somewhere.com

7936 Phones and Problems With DTMF on SIP Trunks

The CUCM  SRND states that the best way to implement a SIP trunk is to set the DMTF Preference on the Trunk in CUCM to "No Preference" and use dtmf-relay sip-kpml rtp-nte on the dial-peers pointing to the CUCM server(s). This allows CUCM to decide which method to use, depending if a phone supports out of band (e.g. KPML, SIP notify, etc.) or RFC2833 & thus avoid invoking an MTP to handle DTMF inter-working unless unavoidable.
Unfortunately this doesn't work with the old 7936 conference phones, you'll get no DTMF in calls via the SIP trunk, bug reference CSCuc80321 explains in detail. In short the workaround is to use DTMF Preference "RFC2833" on the Trunk, but this will always invoke an MTP. This in turn can cause a problem if you're using G729, as the MTPs running on the CUCM server(s) as part of the IP Voice Media Streaming App service only support G711. You'll need a router to be able to provide an MTP that can handle G729, as the example configuration snippet shows:

dspfarm profile 1 mtp 
 codec g729r8
 codec g729ar8
 maximum sessions software 60
 associate application SCCP

Unity Connection Intra-Site Networking Stalled

In some situations when adding server(s) to Unity Connection's intra-site networking, replication can get stuck and constantly show errors on the Voice Network Map. Symptoms include not seeing other server's partitions, search spaces, users, distribution lists, etc. Looking at the traces for Connection Digital Networking Replication Agent service shows many entries like "18:34:13.772 |20016,StalledReplicationDetector,,-1,-1,Detected stalled replication sending to location <location name>. Waiting for acknowledgment of USN <number>. This situation may indicate network connectivity problems."
Follow these steps to get replication unstuck:

Remove the stuck servers from Locations via "remove self from site".

On all servers restart the following services:
Connection Digital Networking Replication Agent
Connection SMTP Server

Add the servers back, don't forget to put in the Connection Location Passwords on all servers & activate the Connection Digital Networking Replication if necessary.

6900 & 7800 Series Phone Directories Failing

Sadly the 7800 series phones have multiple known issues with directories (e.g. CSCuq12291 & CSCus20047), resulting in the Personal Directory and Corporate Directory displaying "host not found". I encountered this during a large deployment of 7821 & 7841 phones with CUCM 10.5(2), Cisco TAC issued an engineering special firmware (78xx.10-2-1ES6-1) which failed to resolve the problem. Eventually the problem was resolved by manually editing the service URLs for Personal Directory and Corporate Directory from the default values.

Directory	Default value	Amended value
Corporate Directory	Application:Cisco/CorporateDirectory	http://10.15.0.2:8080/ccmcip/xmldirectoryinput.jsp
Personal Directory	Application:Cisco/PersonalDirectory	http://10.15.0.2:8080/ccmpd/pdCheckLogin.do?

Put your publisher IP address into the amended service URL, click save, click update subscription & then reset your phones.
To avoid affecting other phones I did try setting these URLs in Enterprise Parameters & then setting the 7821/7841 phones to use external service provisioning, however they don't seem to support external service provisioning & thus get stuck showing "requesting". Fortunately using the amended service URLs have no negative affect on other models of phone.

Update: I also recently experienced this exact same issue with 6900 series phones after doing an upgrade to CUCM 10.5(2), fortunately updating the service URLs also fixed the problem.

Unity Connection Connection Failed to Record Name

I've experienced this in Unity Connection 10.5 & 9.1, but it may affect other versions as the Media Master applet is notoriously flakey.

Attempting to save an uploaded recording in the Media Master applet results in an error "Failed to record name", the workaround is to select "use phone" for playback (see screenshot below), upload the recording, play it & then saving works; not playing the recording or playing it through the computer fails.

For those of you having difficulty even launching the Media Master applet, edit the Java Runtime Environment (JRE) settings via Java in Control Panel (for Windows users). Add the IP address & FQDN of the Unity Connection server to the Exception Site List:

If you're using self-signed certificates, recent versions of the JRE don't like this so either import the certificate via Manage Certificates or generate a certificate from a trusted CA. As a general rule it's better to have certificates issued by a trusted CA, but this may not be feasible in all situations.

Unity Connection Port Status Monitor

There's a quick & dirty way to monitor port status in Unity Connection to confirm the called, calling, redirecting number, redirecting reason & mailbox reached.

First navigate to System Settings > Advanced > Conversations, enter your PC's IP address into "IP Addresses Allowed To Connect For Port Status Monitor Output" & then tick "Enable Remote Port Status Monitor Output":

Note that you have to wait a bit, sometimes even 15 minutes, for the change to become live.
Now you can Telnet to port 5000 on you CUC server & view port status information in real time, such as the example below:

CallData, 8, CallerId=7999, CalledId=9010, RedirectingId=9010, AltRedirectingId=, Origin=16, Reason=8, CallGuid=EEAA4D0D4BD944ABBA8E1860437598BC, CallerName=Cisco Trainer, LastRedirectingId=9010, AltLastRedirectingId=, LastRedirectingReason=8, PortDisplayName=PhoneSystem-1-001
Application, 8, 7999, AttemptForward
State, 8, 7999, State - AttemptForward.cde!Dummy
State, 8, 7999, Event is [NULL]
Application, 8, 7999, PHTransfer
State, 8, 7999, State - PHTransfer.cde!LoadInfo
State, 8, 7999, Event is [TrueEvent]
Application, 8, 7999, PHGreeting
State, 8, 7999, State - PHGreeting.cde!PlayGreeting
Display, 8, 7999, Call answered if needed
Display, 8, 7999, Playing greeting for Subscriber:  Voicemail Test House
State, 8, 7999, Event is [HangupEvent]
State, 8, 7999, State - PHGreeting.cde!DoHangup
State, 8, 7999, Event is [HangupEvent]
Display, 8, 7999, Idle

Handy for quickly troubleshooting calls not delivering to the mailbox you intended, however remember that in an HA cluster you'll need to Telnet to port 5000 on the server that the ports are active on.

CUCM Pattern or Number Reference Finder

For some unknown reason you can't run dependency records on hunt pilots or route patterns, which makes tracking down references to them a pain if you've deleted or changed the number. Fortunately you can run SQL queries from the CLI to get around this. Below is an SQL query for CUCM 8.X+ that checks for CFA, CFB, CFNA, CFNC, CFUR, AAR Destination Mask or Called Party Transformation that reference a specified number (in this example 119063):


SELECT n.DNOrPattern, n.AlertingName, n.Description, n.tkPatternUsage FROM NumPlan AS n LEFT JOIN CallForwardDynamic AS cfd ON cfd.fkNumPlan = n.pkid WHERE n.CFAptDestination LIKE '119063' OR n.CFBDestination LIKE '119063' OR n.CFBIntDestination LIKE '119063' OR n.CFNADestination LIKE '119063' OR n.CFNAIntDestination LIKE '119063' OR n.PFFDestination LIKE '119063' OR n.PFFIntDestination LIKE '119063' OR n.CFURDestination LIKE '119063' OR n.CFURIntDestination LIKE '119063' OR n.AARDestinationMask LIKE '119063' OR n.CalledPartyTransformationMask LIKE '119063' OR cfd.CFADestination LIKE '119063' ORDER BY n.DNOrPattern ASC

The output will include the pattern, description, alerting name & pattern type for any dial plan elements that reference the given number:

admin: run sql SELECT n.DNOrPattern, n.AlertingName, n.Description, n.tkPatternUsage FROM NumPlan AS n LEFT JOIN CallForwardDynamic AS cfd ON cfd.fkNumPlan=n.pkid WHERE n.CFAptDestination LIKE '119063' OR n.CFBDestination LIKE '119063' OR n.CFBIntDestination LIKE '119063' OR n.CFNADestination LIKE '119063' OR n.CFNAIntDestination LIKE '119063' OR n.PFFDestination LIKE '119063' OR n.PFFIntDestination LIKE '119063' OR n.CFURDestination LIKE '119063' OR n.CFURIntDestination LIKE '119063' OR n.AARDestinationMask LIKE '119063' OR n.CalledPartyTransformationMask LIKE '119063' OR cfd.CFADestination LIKE '119063' ORDER BY n.DNOrPattern ASC

dnorpattern alertingname   description    tkpatternusage
=========== ============== ============== ==============
9063        Catarina Vidal Catarina Vidal 2 

The tkPatternUsage field lists the type of dial plan element, use the table below to interpret it:

tkPatternUsage	Dial Plan Element
0	Call Park
1	Conference
2	Directory Number
3	Translation Pattern
4	Call Pick Up Group
5	Route Pattern
6	Message Waiting
7	Hunt Pilot
8	Voice Mail Port
9	Domain Routing
10	IP Address Routing
11	Device template
12	Directed Call Park
13	Device Intercom
14	Translation Intercom
15	Translation Calling Party Number
16	Mobility Handoff
17	Mobility Enterprise Feature Access
18	Mobility IVR
19	Device Intercom Template
20	Called Party Number Transformation
21	Call Control Discovery Learned Pattern
22	Uri Routing
23	ILS Learned Enterprise Number
24	ILS Learned E164 Number
25	ILS Learned Enterprise Numeric Pattern
26	ILS Learned E164 Numeric Pattern
27	Alternate Number
28	ILS Learned URI
29	ILS Learned PSTN Failover Rule
30	ILS Imported E164 Number

Rebuilding IM & Presence Servers Failing

Now that IM & Presence servers are configured as subscribers in CUCM, rebuilding them isn't quite so straight forward. If you just try reinstalling them then you'll get a nice unhelpful error message like the below:

To avoid this, within CUCM you must first delete the Presence Redundancy Group that contains the IM & P servers, then delete the IM & P servers. You may also need to bulk update all End Users to disable "Enable User for Unified CM IM and Presence". Then you'll be able to successfully reinstall the IM & Presence servers without it failing.

Also watch out for bug CSCus27634, which affects certain versions of IM & Presence 10.5, it results in the Cisco Presence Engine refusing to start due to it crashing. If you look the traces for the Cisco Presence Engine you'll see something a bit like this:

*** glibc detected *** /usr/local/pe/bin/pe: free(): invalid next size
(fast): 0x0ada0010 *** ======= Backtrace: =========
/lib/libc.so.6(+0x70b91)[0x787bb91]