Friday, 7 September 2018

4300 / 4400 Forwarding CPU Architecture and Utilisation

The 4300 & 4400 series routers run IOS XE & behave quite differently from previous branch router models (e.g. 2900 or 3900 series). They use dedicated CPU cores to handle forwarding traffic, the 4300 series uses specific cores on the Intel Atom C2000 series CPU & the 4400 series have a dedicated Cavium Octeon series CPU for this purpose. There's separation of the control plane via a dedicated CPU core, with packet forwarding spread across multiple cores. However the IOSd process, which handles the CLI, runs on a separate CPU from the control plane. As a result show process cpu only displays the CPU utilisation by the IOSd process, which will generally be very low. Instead show platform hardware qfp active datapath utilization must be used to see the CPU utilisation by the forwarding CPU(s).

Here is the show process cpu history output of a 4331 router that is handling so much traffic punted to the control plane that it isn't able to process OSPF hellos & keeps dropping OSPF adjacencies:
wanr2#sh proc cpu history
                                           11111
      444444444444666664444444444666666666600000777775555566666555
  100
   90
   80
   70
   60
   50
   40
   30
   20
   10             *****          *******************************
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)
      1 1  1    1     1
      080562778529765604564547466449777678786796886767577678698777
  100
   90
   80
   70
   60
   50
   40
   30
   20
   10 ###**#****##***** ** * * **  *****#*****#*##*#****##*##*##
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

For comparison, the forwarding CPU(s) are at 100%:
wanr2#sh plat hard qfp act datapath utilization
  CPP 0: Subdev 0            5 secs        1 min        5 min       60 min
Input:  Priority (pps)            2            2            2            2
                 (bps)         2744         2056         2072         2064
    Non-Priority (pps)        98504        98368        98543        77967
                 (bps)     48603624     48562944     48578680     38524136
           Total (pps)        98506        98370        98545        77969
                 (bps)     48606368     48565000     48580752     38526200
Output: Priority (pps)            2            3            2            3
                 (bps)         2704         3240         3152         3096
    Non-Priority (pps)          128          125          103          111
                 (bps)       243736       218816       174288       224744
           Total (pps)          130          128          105          114
                 (bps)       246440       222056       177440       227840
Processing: Load (pct)          100          100          100           79

If you know from which CPUs are assigned to which roles, you can also use show processes cpu platform sorted (0 is always IOSd):
wanr2#sh processes cpu platform sorted
CPU utilization for five seconds: 4%, one minute: 5%, five minutes: 4%
Core 0: CPU utilization for five seconds: 8%, one minute: 4%, five minutes: 3%
Core 1: CPU utilization for five seconds: 24%, one minute: 7%, five minutes: 4%
Core 2: CPU utilization for five seconds: 4%, one minute: 4%, five minutes: 3%
Core 3: CPU utilization for five seconds: 3%, one minute: 4%, five minutes: 4%
Core 4: CPU utilization for five seconds: 8%, one minute: 7%, five minutes: 7%
Core 5: CPU utilization for five seconds: 1%, one minute: 0%, five minutes: 0%
Core 6: CPU utilization for five seconds: 14%, one minute: 12%, five minutes: 11%
Core 7: CPU utilization for five seconds: 0%, one minute: 0%, five minutes: 0%

Note that these router platforms also have an optional higher throughput license, which unlocks more CPU cores for forwarding. If this feature has been licensed, it is enabled via the platform hardware throughput level command, which requires a reboot:
wanr2(config)#plat hardware throughput level ?
  100000  throughput in kbps
  300000  throughput in kbps
wanr2(config)#platform hardware throughput level 300000
         Feature Name:throughput
 
PLEASE  READ THE  FOLLOWING TERMS  CAREFULLY. INSTALLING THE LICENSE OR
LICENSE  KEY  PROVIDED FOR  ANY CISCO  PRODUCT  FEATURE  OR  USING SUCH
PRODUCT  FEATURE  CONSTITUTES  YOUR  FULL ACCEPTANCE  OF  THE FOLLOWING
TERMS. YOU MUST NOT PROCEED FURTHER IF YOU ARE NOT WILLING TO  BE BOUND
BY ALL THE TERMS SET FORTH HEREIN.
 
Use of this product feature requires  an additional license from Cisco,
together with an additional  payment.  You may use this product feature
on an evaluation basis, without payment to Cisco, for 60 days. Your use
of the  product,  including  during the 60 day  evaluation  period,  is
subject to the Cisco end user license agreement
http://www.cisco.com/en/US/docs/general/warranty/English/EU1KEN_.html
If you use the product feature beyond the 60 day evaluation period, you
must submit the appropriate payment to Cisco for the license. After the
60 day  evaluation  period,  your  use of the  product  feature will be
governed  solely by the Cisco  end user license agreement (link above),
together  with any supplements  relating to such product  feature.  The
above  applies  even if the evaluation  license  is  not  automatically
terminated  and you do  not receive any notice of the expiration of the
evaluation  period.  It is your  responsibility  to  determine when the
evaluation  period is complete and you are required to make  payment to
Cisco for your use of the product feature beyond the evaluation period.
 
Your  acceptance  of  this agreement  for the software  features on one
product  shall be deemed  your  acceptance  with  respect  to all  such
software  on all Cisco  products  you purchase  which includes the same
software.  (The foregoing  notwithstanding, you must purchase a license
for each software  feature you use past the 60 days evaluation  period,
so  that  if you enable a software  feature on  1000  devices, you must
purchase 1000 licenses for use past  the 60 day evaluation period.)
 
Activation  of the  software command line interface will be evidence of
your acceptance of this agreement.
 
 
ACCEPT? (yes/[no]): yes
% The config will take effect on next reboot

The current throughput level can be confirmed via show platform hardware throughput level:
wanr2#show plat hard throughput level
The current throughput level is 100000 kb/s

Further useful information can be found here: