3850 Switch Upgrade Failure

Every so often a software upgrade fails, be it a corrupted download from Cisco's website or occasional hiccup on the network whilst copying it to the device. Rather usefully the 3650 & 3850 switches have a USB port on the front which can be used with a USB memory stick to recover from a failed upgrade. This a walk through of the process & a couple of obstacles along the way.

To enter ROMMON to recovery from a failed upgrade either wait for 5 failed boot attempts or hold the Mode button down for 10 seconds whilst the switch boots. The command prompt should be "switch:". Plug in the USB drive & you can view the contents via dir usblflash0:. According the official documentation you can at this point either boot from the USB drive or copy the image to flash, however testing with IOS XE 3.7.x the flash was only ever read only:

switch: dir usbflash0:
Directory of usbflash0:/

     4  -rw-  328157104  cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin

7689236480 bytes available (328175616 bytes used)

switch: copy usbflash0:cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin flash:
flash:usbflash0:cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin: read only file system

If you boot off the USB drive via the boot command, you should be able to get to a point where you can remove the corrupt image & tidy up. However there's another hurdle if the IOS XE image includes a firmware update for the switch ASICs, as it will try to boot using the corrupt image on flash after completing the firmware update:

switch: boot usbflash0:cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin
Reading full image into memory............................................................................................................................................................................................................................................................................................................................done
Bundle Image
--------------------------------------
Kernel Address    : 0x5344e290
Kernel Size       : 0x3fb4ba/4175034
Initramfs Address : 0x5384974c
Initramfs Size    : 0xd47999/13924761
Compression Format: .mzip

Bootable image at @ ram:0x5344e290
Bootable image segment 0 address range [0x81100000, 0x820b0000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "usbflash0:cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin" uncompressed and installed, entry point: 0x816734a0
Loading Linux kernel with entry point 0x816734a0 ...
Bootloader: Done loading app on core_mask: 0x3f

### Launching Linux Kernel (flags = 0x5)

All packages are Digitally Signed
Starting System Services
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=4,mode=600,ptmxmode=000 0 0


FIPS(NGWC): Flash Key Check : Begin
FIPS(NGWC): Flash Key Check : End, Not Found, FIPS Mode Not Enabled


Front-end Microcode IMG MGR: found 4 microcode images for 1 device.
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_6_0
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_6_1
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_6_2
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_6_3

Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0]...594412 bytes.... Skipped[0].
Front-end Microcode IMG MGR: Preparing to program device[0]...392946 bytes.
Front-end Microcode IMG MGR: Programming device 0...rwRrrrrrrw..0%.........................................................................10%........................................................................20%..........................................................................30%........................................................................40%........................................................................50%..........................................................................60%.........................................................................70%........................................................................80%..........................................................................90%........................................................................100%
Front-end Microcode IMG MGR: Preparing to program device[0]...25166 bytes.
Front-end Microcode IMG MGR: Programming device 0...rrrrrrw..0%....10%....20%......30%...40%......50%....60%......70%...80%......90%....100%wRr!
Front-end Microcode IMG MGR: Microcode programming complete for device 0.
Front-end Microcode IMG MGR: Preparing to program device[0]...86370 bytes.
Front-end Microcode IMG MGR: Programming device 0...rwRrrrrrrw..0%................10%...............20%.................30%...............40%.................50%...............60%.................70%................80%...............90%.................100%w

Front-end Microcode IMG MGR: Microcode programming complete in 291 seconds

% Front-end Microcode IMG MGR: HW image is upgraded. MCU reset causes the switch to reload

If this happens use the boot  command to boot from the USB drive again & proceed with tidying up as normal.

7800 Series Phones Not Prompting For MRA Domain & User Credentials

I've seen this happen a few times after upgrading from 7800 series firmware 10.3.1 to the latest 11.0.1 firmware (which introduced official MRA support). Phones that could previously connect endlessly loop between "registering" & "not registered", they never prompt for the domain to resolve the _collab-edge._tls SRV record, nor do they attempt to resolve a cached domain either.
Correct behaviour is if DHCP option 150 isn't present or the TFTP server is unreachable, the phone tries to resolve the host name of the CUCM servers in its cached configuration file, if this also fails then the phone should determine that it should use MRA instead. Looking at the status messages on the phone confirms that the phone never gets beyond the lack of TFTP or CUCM connectivity:

[12:22:54,29/04/16] TFTP Error : SEPF09E636E5656.cnf.xml.sgn
[12:22:54,29/04/16] DNS Unknown IPv4 Host CUCM2.somewhere.com
[12:22:54,29/04/16] DNS Unknown IPv4 Host CUCM1.somewhere.com
[12:22:57,29/04/16] TFTP Error : SK93886772-e459-a46f-ff8d-e15df16797e8.xml.sgn
[12:22:58,29/04/16] Configuring IP
[12:23:03,29/04/16] No IPv4 TFTP Server
[12:23:03,29/04/16] Trust List Update Failed
[12:23:04,29/04/16] TFTP Error : SEPF09E636E5656.cnf.xml.sgn
[12:23:04,29/04/16] DNS Unknown IPv4 Host CUCM2.somewhere.com
[12:23:04,29/04/16] DNS Unknown IPv4 Host CUCM1.somewhere.com

Looking at the console logs on the phone also shows that the phone fails to connect to TFTP or resolve the server names but thinks it's not in Collaboration Edge mode:

1102 ERR Apr 29 12:23:27.529537 JAVA-CFG : config_process_ccm_properties : No valid IPv4 TFTP server configured
1103 NOT Apr 29 12:23:27.529953 JAVA-ccm0=CUCM2.somewhere.com ccm1=CUCM1.somewhere.com  ccm2= sip_port_0=5060 sip_port_1=5060 sip_port_2=5060 length=0
1104 NOT Apr 29 12:23:27.530006 JAVA-ccm0=CUCM2.somewhere.com ccm1=CUCM1.somewhere.com  ccm2= sec_sip_port_0=5061 sec_sip_port_1=5061 sec_sip_port_2=5061 length=0
1105 NOT Apr 29 12:23:27.530341 JAVA-ccm0_v6= ccm1_v6=  ccm2_v6= lenght=0
1106 NOT Apr 29 12:23:27.530382 JAVA-ip_mode is IPv4
1107 NOT Apr 29 12:23:27.530614 JAVA-config_process_ccm_properties: is in Edge mode = FALSE
1108 NOT Apr 29 12:23:27.597478 dgetserver(9651)-dgetserver.RESULT [_0_  [PASS] No error     ]
1109 NOT Apr 29 12:23:27.606755 downd-SOCKET accept errno=4 "Interrupted system call"
1110 INF Apr 29 12:23:27.630973 dnsmasq[2033]: query[A] CUCM1.somewhere.com from 127.0.0.1
1111 INF Apr 29 12:23:27.631110 dnsmasq[2033]: cached CUCM1.somewhere.com is NXDOMAIN-IPv4
1112 INF Apr 29 12:23:27.632905 dnsmasq[2033]: query[A] CUCM1.somewhere.com from 127.0.0.1
1113 INF Apr 29 12:23:27.633020 dnsmasq[2033]: cached CUCM1.somewhere.com is NXDOMAIN-IPv4
1114 ERR Apr 29 12:23:27.631788 JAVA-dnsGetHostByName :lookup CUCM1.somewhere.com failed.
1115 INF Apr 29 12:23:27.634020 dnsmasq[2033]: query[A] CUCM2.somewhere.com from 127.0.0.1
1116 INF Apr 29 12:23:27.634110 dnsmasq[2033]: cached CUCM2.somewhere.com is NXDOMAIN-IPv4
1117 INF Apr 29 12:23:27.634374 dnsmasq[2033]: query[A] CUCM2.somewhere.com from 127.0.0.1
1118 INF Apr 29 12:23:27.634565 dnsmasq[2033]: cached CUCM2.somewhere.com is NXDOMAIN-IPv4
1119 ERR Apr 29 12:23:27.634845 JAVA-dnsGetHostByName :lookup CUCM2.somewhere.com failed.

Eventually I stumbled across Admin Settings > Reset Settings > Service Name fixing the problem. Not exactly a clearly described setting, nor does it explain why the phone isn't getting as far as prompting for a domain, if it had cached an incorrect one I'd expect to see it trying to resolve the _collab-edge._tls SRV record for the incorrect domain.

ATA 186 Refusing to Register to New Server

Whilst working on a CUCM merger project, the process below worked for moving regular phones from the 2 existing CUCM clusters over to the new CUCM cluster:

1. Set DHCP scope option 150 to new CUCM server's IP address
2. Bulk erase ITL files on all phones using a 3rd party tool
3. Reset all phones

However all the ATA 186s flat out refused to register to the new CUCM despite multiple attempts. Looking at the ATA's built in web server showed the correct TFTP server had been picked up from DHCP. Eventually found that you can force the CUCM IP addresses to register to in the GUI:

Reset the ATA 186 after applying the change & voila it works!

Restoring Publisher Database From the Subscriber

I've been meaning to write about this feature for a while now, since it's actually pretty handy. Cisco introduced a feature whereby you can restore the publisher's database from a subscriber server in v8.6(1) of CUCM & CUC. You still have to rebuild a new publisher identical to the failed one, then get the licences migrated (if applicable) & the method differs between CUCM & CUC. Having seen several scenarios where customers have either only the original DRS backup done at go live or out of date backups available, this will help you get out of a sticky situation & back to a working cluster without losing any configuration data.
The Cisco documentation is pretty good for these features, so I'm going to cover some caveats & defer to the official instructions. Be aware that there are quite a few steps that must be followed correctly otherwise you run the risk of pushing data from the blank publisher to subscribers, it also includes multiple reboots, so this is a time consuming process.

CUCM
The DRS restore mechanism has been enhanced so that you can choose to restore the CCMDB component from a selected subscriber server. Now the Cisco guide says you can do a fresh DRS backup from the newly built publisher & do the restore from a subscriber, but whenever I've tried the option never appears unless the actual backup contains files from the intended subscriber server. So you'll need a DRS backup that contains both the publisher & subscriber server.

The main caveat is that your new publisher won't have files that aren't in the backup or have changed since then, such as certificates, device packs, TFTP server files (e.g. phone loads, music on hold, etc.) or logs/traces. This is because this information isn't held in the Informix database, which is what gets copied from the subscriber, so you'll need to deal with these out manually.

Cisco Guide to CUCM Restore From Subscriber

CUC
CUC's clustering is different from CUCM, in that you still have a publisher & subscriber, but there's also the concept of the primary & secondary server. The primary server, regardless of whether it's the publisher or subscriber, contains a writeable configuration database & message store; normally this is automatically replicated to the secondary server.
After rebuilding the publisher you're basically rebuilding the cluster, with the subscriber as primary & manually starting the synchronisation process.

The main caveat is that the publisher needs locales & any other patches installing manually, as these aren't pulled across from the subscriber. The Informix database with configuration, message store, spoken names, greetings & master certificates are what gets copied from the subscriber. You'll still need to deal with certificates that were specific to the publisher server, such as the Tomcat certificate.

Cisco Guide to CUC Restore From Subscriber

New Subscriber Installation Failing With 10.5 After DRS Restore

I recently encountered this bug in 10.5 which breaks the ability to install subscribers after a DRS restore. As Cisco's official P2V method is to do a DRS backup from the physical machine & then a DRS restore to an identically configured virtual machine, this largely applies to upgrade projects where you have to P2V CUCM as it was running on a physical MCS 7800 series server.

With the 10.5.2.12901 upgrade I was working on, the subscriber installation would get stuck after validating network connectivity to the publisher with a very helpful error message:

There is a bug report (CSCus35964) for this & a patch ciscocm.Fix_DRS_Recovery_v1.0.k3.cop.sgn that must be installed on the publisher, then the publisher rebooted.

4300 Series Routers Bugs

Having deployed a number of the new 4331 routers to replace 2800 series routers as voice gateways, there's been some quirks...

The first being that unicast music on hold stopped working. Identical configuration to the 2800 routers, but PSTN callers just hear silence. Eventually found that setting the CallManager service parameter Duplex Streaming Enabled to true fixes this problem:

I also hit a nasty bug in that 4000 series IOS XE releases 3.15.0, 3.15.1 & 3.16.0 aren't fit for purpose as voice gateways, whenever the router reloads the dial-peer configuration (along with NBAR & some ACLs) disappears. Why these versions aren't deferred releases yet I don't know?!

UPDATE: Now to cap it all off the 4300 series is undergoing a recall.

UCCX Editor Script Repository and Reactive Debugging Not Working

In UCCX 10.0 Cisco decided to change the way UCCX works so that hostnames not IP addresses are used for inter-server communication. The UCCX Editor was also changed to use hostnames, resulting in the odd situation where you can authenticate against the UCCX server by IP address, but browsing the script repository or using reactive debugging doesn't work. This is because the UCCX Editor will try to use the hostname of the UCCX servers, if the DNS A records don't exist or you're connected via a VPN without correct DNS configuration then the DNS resolution will fail & these features won't work. The error message provided is pretty vague on this, for example if you try to use reactive debugging it will get stuck with the message "Engine hostname list is empty. Check the ldap for enabled unified ccx engines".
The solution is to kludge it by editing your local hosts file to contain the hostname & FQDN of the UCCX server(s). On Windows edit the file C:\Windows\System32\drivers\etc\hosts, example entries below:

10.15.0.5      perry
10.15.0.5      perry.somewhere.com