3850 Switch Upgrade Failure

Every so often a software upgrade fails, be it a corrupted download from Cisco's website or occasional hiccup on the network whilst copying it to the device. Rather usefully the 3650 & 3850 switches have a USB port on the front which can be used with a USB memory stick to recover from a failed upgrade. This a walk through of the process & a couple of obstacles along the way.

To enter ROMMON to recovery from a failed upgrade either wait for 5 failed boot attempts or hold the Mode button down for 10 seconds whilst the switch boots. The command prompt should be "switch:". Plug in the USB drive & you can view the contents via dir usblflash0:. According the official documentation you can at this point either boot from the USB drive or copy the image to flash, however testing with IOS XE 3.7.x the flash was only ever read only:

switch: dir usbflash0:
Directory of usbflash0:/

     4  -rw-  328157104  cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin

7689236480 bytes available (328175616 bytes used)

switch: copy usbflash0:cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin flash:
flash:usbflash0:cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin: read only file system

If you boot off the USB drive via the boot command, you should be able to get to a point where you can remove the corrupt image & tidy up. However there's another hurdle if the IOS XE image includes a firmware update for the switch ASICs, as it will try to boot using the corrupt image on flash after completing the firmware update:

switch: boot usbflash0:cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin
Reading full image into memory............................................................................................................................................................................................................................................................................................................................done
Bundle Image
--------------------------------------
Kernel Address    : 0x5344e290
Kernel Size       : 0x3fb4ba/4175034
Initramfs Address : 0x5384974c
Initramfs Size    : 0xd47999/13924761
Compression Format: .mzip

Bootable image at @ ram:0x5344e290
Bootable image segment 0 address range [0x81100000, 0x820b0000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "usbflash0:cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin" uncompressed and installed, entry point: 0x816734a0
Loading Linux kernel with entry point 0x816734a0 ...
Bootloader: Done loading app on core_mask: 0x3f

### Launching Linux Kernel (flags = 0x5)

All packages are Digitally Signed
Starting System Services
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=4,mode=600,ptmxmode=000 0 0


FIPS(NGWC): Flash Key Check : Begin
FIPS(NGWC): Flash Key Check : End, Not Found, FIPS Mode Not Enabled


Front-end Microcode IMG MGR: found 4 microcode images for 1 device.
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_6_0
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_6_1
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_6_2
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_6_3

Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0]...594412 bytes.... Skipped[0].
Front-end Microcode IMG MGR: Preparing to program device[0]...392946 bytes.
Front-end Microcode IMG MGR: Programming device 0...rwRrrrrrrw..0%.........................................................................10%........................................................................20%..........................................................................30%........................................................................40%........................................................................50%..........................................................................60%.........................................................................70%........................................................................80%..........................................................................90%........................................................................100%
Front-end Microcode IMG MGR: Preparing to program device[0]...25166 bytes.
Front-end Microcode IMG MGR: Programming device 0...rrrrrrw..0%....10%....20%......30%...40%......50%....60%......70%...80%......90%....100%wRr!
Front-end Microcode IMG MGR: Microcode programming complete for device 0.
Front-end Microcode IMG MGR: Preparing to program device[0]...86370 bytes.
Front-end Microcode IMG MGR: Programming device 0...rwRrrrrrrw..0%................10%...............20%.................30%...............40%.................50%...............60%.................70%................80%...............90%.................100%w

Front-end Microcode IMG MGR: Microcode programming complete in 291 seconds

% Front-end Microcode IMG MGR: HW image is upgraded. MCU reset causes the switch to reload

If this happens use the boot  command to boot from the USB drive again & proceed with tidying up as normal.