Sunday 4 July 2021

Incident Models

An incident model is a means to streamline & standardise the troubleshooting of critical business systems. As a guide it should include the following information:

  • Overview of the system or application
  • Topology diagram(s) and/or list of devices
  • Flowchart for the incident handling process, with checkpoints and/or milestones
  • Template for notifications to the business or stakeholders
  • Basic troubleshooting commands
  • Links to support contracts & contacts


Additional information can be included, such as links to in depth troubleshooting guides, the original design documentation or configuraton backups. Sometimes I like to include keywords that may be mentioned in tickets related to this system, to help 1st line staff quickly triage an incident & apply the appropriate incident model.

Thursday 29 April 2021

Cisco Switch MAC Address Flapping Alerts

MAC address table instability can impact a switch's performance & on lower end switches cause high CPU utilisation that may impact other functions. Cisco switches can generate a syslog entry when they see a MAC address flap between ports, but it’s not enabled by default. Some NX-OS platforms actually temporarily disable MAC address table updates if a certain number of MAC address flaps occur within a set timeframe: https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/nx-os-software/213906-nexus-9000-mac-move-troubleshooting-and.html
The different switch platforms generate slightly different syslog messages, but the common factor is they all have MAC_MOVE in the text for NX-OS, or MACFLAP or HOSTFLAP for IOS / IOS XE. So I created an alert in Splunk to match these keywords in the last hour's log entries.

Commands

IOS / IOX XE:
mac address table notification mac-move

N3K:
mac address table notification mac-move
logging level fwm 6
logging monitor 6


N4K:
mac address table notification mac-move
logging level fwm 6
logging monitor 6


N5K / N6K:
mac address table notification mac-move
logging level fwm 6
logging monitor 6


N7K / N9K:
logging level l2fm 5

Monday 22 February 2021

Parsing Cisco Extended ACLs in Python

A little toy project to entertain myself, as kept being asked to document & explain the ACLs on some perimeter routers. Takes the output of show access-list & does an incomplete parse of the extended ACL, then outputs it in semi-human readable or plain English form. I say an incomplete parse as only implemented enough syntax to parse the ACLs I was being asked about (e.g. IPv4 only). Some tinkering with the core regex & if/elif statements would make it parse a more complete extended ACL syntax.

Usage: ACL_parser.py [filename] [translate]
Where 'translate' is optional argument to display the ACL lines in English also.


Example ACL:

Extended IP access list MyACL
    10 permit tcp host 21.35.80.22 eq telnet host 21.23.77.101
    20 permit tcp 21.35.80.0 0.0.0.255 eq 16100 21.23.77.0 0.0.0.255 range 8192 8921 (149407 matches)
    30 permit udp 21.35.80.0 0.0.0.3 lt 17600 host 21.23.77.101 eq www (80592 matches)
    40 permit tcp host 21.35.80.27 eq 10701 host 21.23.77.101 established (26008 matches)
    50 permit udp host 21.35.80.22 neq telnet 21.23.77.128 0.0.0.127 gt 1023
    60 permit tcp host 21.35.80.25 eq 16100 host 21.23.77.101 range 8192 8921 (149407 matches)
    70 permit udp 21.35.80.0 0.0.0.127 lt 17600 21.23.77.128 0.0.0.127 (80592 matches)
    75 permit icmp any 192.168.0.0 0.0.0.255 echo log
    80 deny ip any any log (1 match)


Example output from above ACL:

python ACL_parser.py test_acl.txt translate


10 permit tcp host 21.35.80.22 eq 23 host 21.23.77.101
line 10 permit tcp connections from IP address 21.35.80.22 where port equals 23, to IP address 21.23.77.101

20 permit tcp 21.35.80.0/24 eq 16100 21.23.77.0/24 range 8192 8921
line 20 permit tcp connections from IP addresses 21.35.80.0 - 21.35.80.255 where port equals 16100, to IP addresses 21.23.77.0 - 21.23.77.255 where port between 8192 - 8921

30 permit udp 21.35.80.0/30 lt 17600 host 21.23.77.101 eq 80
line 30 permit udp connections from IP addresses 21.35.80.0 - 21.35.80.3 where port less than 17600, to IP address 21.23.77.101 where port equals 80

40 permit tcp host 21.35.80.27 eq 10701 host 21.23.77.101 established
line 40 permit tcp connections from IP address 21.35.80.27 where port equals 10701, to IP address 21.23.77.101 if the connection is already established

50 permit udp host 21.35.80.22 neq 23 21.23.77.128/25 gt 1023
line 50 permit udp connections from IP address 21.35.80.22 where port doesn't equal 23, to IP addresses 21.23.77.128 - 21.23.77.255 where port greater than 1023

60 permit tcp host 21.35.80.25 eq 16100 host 21.23.77.101 range 8192 8921
line 60 permit tcp connections from IP address 21.35.80.25 where port equals 16100, to IP address 21.23.77.101 where port between 8192 - 8921

70 permit udp 21.35.80.0/25 lt 17600 21.23.77.128/25
line 70 permit udp connections from IP addresses 21.35.80.0 - 21.35.80.127 where port less than 17600, to IP addresses 21.23.77.128 - 21.23.77.255

75 permit icmp any 192.168.0.0/24 echo log
line 75 permit icmp connections from IP address any, to IP addresses 192.168.0.0 - 192.168.0.255 for ICMP echo, and log

80 deny ip any any log
line 80 deny ip connections from IP address any, to IP address any, and log

 

Source Code

#!/usr/bin/env python3

import re
import ipaddress
import sys

"""
Written by Chris Perkins in 2021
Licence: BSD 3-Clause

Parse Cisco extended ACL output from show access-list command & display in a human readable format
"""

# Subnet / wildcard mask to CIDR prefix length lookup table
SUBNET_MASKS = {
    "128.0.0.0": "1",
    "127.255.255.255": "1",
    "192.0.0.0": "2",
    "63.255.255.255": "2",
    "224.0.0.0": "3",
    "31.255.255.255": "3",
    "240.0.0.0": "4",
    "15.255.255.255": "4",
    "248.0.0.0": "5",
    "7.255.255.255": "5",
    "252.0.0.0": "6",
    "3.255.255.255": "6",
    "254.0.0.0": "7",
    "1.255.255.255": "7",
    "255.0.0.0": "8",
    "0.255.255.255": "8",
    "255.128.0.0": "9",
    "0.127.255.255": "9",
    "255.192.0.0": "10",
    "0.63.255.255": "10",
    "255.224.0.0": "11",
    "0.31.255.255": "11",
    "255.240.0.0": "12",
    "0.15.255.255": "12",
    "255.248.0.0": "13",
    "0.7.255.255": "13",
    "255.252.0.0": "14",
    "0.3.255.255": "14",
    "255.254.0.0": "15",
    "0.1.255.255": "15",
    "255.255.0.0": "16",
    "0.0.255.255": "16",
    "255.255.128.0": "17",
    "0.0.0.127.255": "17",
    "255.255.192.0": "18",
    "0.0.63.255": "18",
    "255.255.224.0": "19",
    "0.0.31.255": "19",
    "255.255.240.0": "20",
    "0.0.15.255": "20",
    "255.255.248.0": "21",
    "0.0.7.255": "21",
    "255.255.252.0": "22",
    "0.0.3.255": "22",
    "255.255.254.0": "23",
    "0.0.1.255": "23",
    "255.255.255.0": "24",
    "0.0.0.255": "24",
    "255.255.255.128": "25",
    "0.0.0.127": "25",
    "255.255.255.192": "26",
    "0.0.0.63": "26",
    "255.255.255.224": "27",
    "0.0.0.31": "27",
    "255.255.255.240": "28",
    "0.0.0.15": "28",
    "255.255.255.248": "29",
    "0.0.0.7": "29",
    "255.255.255.252": "30",
    "0.0.0.3": "30",
    "255.255.255.254": "31",
    "0.0.0.1": "31",
    "255.255.255.255": "32",
    "0.0.0.0": "32",
}

# Port names to port numbers lookup table
PORT_NAMES = {
    "aol": "5190",
    "bgp": "179",
    "biff": "512",
    "bootpc": "68",
    "bootps": "67",
    "chargen": "19",
    "cifs": "3020",
    "citrix-ica": "1494",
    "cmd": "514",
    "ctiqbe": "2748",
    "daytime": "13",
    "discard": "9",
    "dnsix": "195",
    "domain": "53",
    "echo": "7",
    "exec": "512",
    "finger": "79",
    "ftp": "21",
    "ftp-data": "20",
    "gopher": "70",
    "h323": "1720",
    "hostname": "101",
    "http": "80",
    "https": "443",
    "ident": "113",
    "imap4": "143",
    "irc": "194",
    "isakmp": "500",
    "kerberos": "750",
    "klogin": "543",
    "kshell": "544",
    "ldap": "389",
    "ldaps": "636",
    "login": "513",
    "lotusnotes": "1352",
    "lpd": "515",
    "mobile-ip": "434",
    "nameserver": "42",
    "netbios-dgm": "138",
    "netbios-ns": "137",
    "netbios-ssn": "139",
    "nfs": "2049",
    "nntp": "119",
    "ntp": "123",
    "pcanywhere-data": "5631",
    "pcanywhere-status": "5632",
    "pim-auto-rp": "496",
    "pop2": "109",
    "pop3": "110",
    "pptp": "1723",
    "radius": "1645",
    "radius-acct": "1646",
    "rip": "520",
    "rsh": "514",
    "rtsp": "554",
    "secureid-udp": "5510",
    "sip": "5060",
    "smtp": "25",
    "snmp": "161",
    "snmptrap": "162",
    "sqlnet": "1521",
    "ssh": "22",
    "sunrpc": "111",
    "syslog": "514",
    "tacacs": "49",
    "talk": "517",
    "telnet": "23",
    "tftp": "69",
    "time": "37",
    "uucp": "540",
    "vxlan": "4789",
    "who": "513",
    "whois": "43",
    "www": "80",
    "xdmcp": "177",
}

# ACL operator names lookup table
OPERATOR_NAMES = {
    "eq": "equals",
    "neq": "doesn't equal",
    "lt": "less than",
    "gt": "greater than",
    "range": "between",
}


def main():
    """Parse ACL from text file"""
    mansplain = False
    if len(sys.argv) < 2:
        print(f"Usage: {sys.argv[0]} [filename] [translate]")
        print(
            "Where 'translate' is optional argument to display the ACL lines in English also."
        )
        sys.exit(1)
    elif len(sys.argv) == 3:
        if sys.argv[2].lower() == "translate":
            mansplain = True

    try:
        with open(sys.argv[1]) as f:
            acl_string = f.read()
    except FileNotFoundError:
        print(f"Unable to open {sys.argv[1]}")
        sys.exit(1)

    for line in acl_string.splitlines():
        acl_parts = re.search(
            r"^\s*(\d+)\s+(permit|deny)\s(\w+)\s(\d+\.\d+\.\d+\.\d+|any|host)\s*(\d+\.\d+\.\d+\.\d+)?"
            r"\s*(eq|neq|lt|gt|range)?\s*([\w\-]+|[\w\-]+\s[\w\-]+)?\s*(established|echo|echo\-reply)?\s(\d+\.\d+\.\d+\.\d+|any|host)"
            r"\s*(\d+\.\d+\.\d+\.\d+)?\s*(eq|neq|lt|gt|range)?\s*([\w\-]+|[\w\-]+\s[\w\-]+)?\s*(established|echo|echo\-reply)?"
            r"\s*(log\-input|log)?\s*(\(\d+ match(es)?\))?$",
            line.lower(),
        )
        ace_dict = {
            "line_num": "",
            "action": "",
            "protocol": "",
            "source_network": "",
            "source_operator": "",
            "source_ports": "",
            "source_modifier": "",
            "destination_network": "",
            "destination_operator": "",
            "destination_ports": "",
            "destination_modifier": "",
            "optional_action": "",
        }

        if not acl_parts:
            continue

        # Parse the Access Control Entry items into a dictionary
        for item in acl_parts.groups():
            item = item if item is not None else ""
            if not ace_dict["line_num"] and re.search(r"^\d+", item):
                ace_dict["line_num"] = item
            elif not ace_dict["action"] and item in ["permit", "deny"]:
                ace_dict["action"] = item
            elif not ace_dict["protocol"] and item in [
                "ahp",
                "esp",
                "eigrp",
                "gre",
                "icmp",
                "igmp",
                "igrp",
                "ip",
                "ipv4",
                "ipinip",
                "nos",
                "ospf",
                "pim",
                "pcp",
                "tcp",
                "udp",
            ]:
                ace_dict["protocol"] = item
            elif not ace_dict["source_network"] and re.search(
                r"\d+\.\d+\.\d+\.\d+|any|host", item
            ):
                ace_dict["source_network"] = item
            elif (
                ace_dict["source_network"]
                and not ace_dict["destination_network"]
                and item in SUBNET_MASKS
            ):
                ace_dict["source_network"] += f"/{SUBNET_MASKS[item]}"
            elif (
                ace_dict["source_network"]
                and ace_dict["source_network"] == "host"
                and not ace_dict["destination_network"]
                and re.search(r"\d+\.\d+\.\d+\.\d+", item)
            ):
                ace_dict["source_network"] += f" {item}"
            elif (
                ace_dict["source_network"]
                and not ace_dict["destination_network"]
                and item in OPERATOR_NAMES
            ):
                ace_dict["source_operator"] = item
            elif (
                ace_dict["source_operator"]
                and not ace_dict["source_ports"]
                and re.search(r"\w+|\w+\s\w+", item)
            ):
                for port_number in item.split():
                    if port_number in PORT_NAMES:
                        ace_dict["source_ports"] += f" {PORT_NAMES[port_number]}"
                    else:
                        ace_dict["source_ports"] += f" {port_number}"
                ace_dict["source_ports"] = ace_dict["source_ports"].strip()
            elif (
                ace_dict["source_network"]
                and not ace_dict["destination_network"]
                and item in ["established", "echo", "echo-reply"]
            ):
                ace_dict["source_modifier"] = item
            elif not ace_dict["destination_network"] and re.search(
                r"\d+\.\d+\.\d+\.\d+|any|host", item
            ):
                ace_dict["destination_network"] = item
            elif ace_dict["destination_network"] and item in SUBNET_MASKS:
                ace_dict["destination_network"] += f"/{SUBNET_MASKS[item]}"
            elif (
                ace_dict["destination_network"]
                and ace_dict["destination_network"] == "host"
                and re.search(r"\d+\.\d+\.\d+\.\d+", item)
            ):
                ace_dict["destination_network"] += f" {item}"
            elif ace_dict["destination_network"] and item in OPERATOR_NAMES:
                ace_dict["destination_operator"] = item
            elif (
                ace_dict["destination_operator"]
                and not ace_dict["destination_ports"]
                and re.search(r"\w+|\w+\s\w+", item)
            ):
                for port_number in item.split():
                    if port_number in PORT_NAMES:
                        ace_dict["destination_ports"] += f" {PORT_NAMES[port_number]}"
                    else:
                        ace_dict["destination_ports"] += f" {port_number}"
                ace_dict["destination_ports"] = ace_dict["destination_ports"].strip()
            elif ace_dict["destination_network"] and item in [
                "established",
                "echo",
                "echo-reply",
            ]:
                ace_dict["destination_modifier"] = item
            elif (
                ace_dict["source_network"]
                and ace_dict["destination_network"]
                and item in ["log", "log-input"]
            ):
                ace_dict["optional_action"] = item

        parsed_ace = (
            f"{ace_dict['line_num']} "
            f"{ace_dict['action']} "
            f"{ace_dict['protocol']} "
            f"{ace_dict['source_network']} "
            f"{ace_dict['source_operator']} "
            f"{ace_dict['source_ports']} "
            f"{ace_dict['source_modifier']} "
            f"{ace_dict['destination_network']} "
            f"{ace_dict['destination_operator']} "
            f"{ace_dict['destination_ports']} "
            f"{ace_dict['destination_modifier']} "
            f"{ace_dict['optional_action']} "
        )
        print(re.sub(r" +", " ", parsed_ace))

        if mansplain:
            mansplained = (
                f"line {ace_dict['line_num']} {ace_dict['action']} {ace_dict['protocol']}"
                " connections"
            )
            if (
                "host" in ace_dict["source_network"]
                or "/" not in ace_dict["source_network"]
            ):
                mansplained += (
                    f" from IP address {ace_dict['source_network'].split()[-1]}"
                )
            else:
                ip_network = ipaddress.IPv4Network(ace_dict["source_network"])
                mansplained += (
                    f" from IP addresses {ip_network.network_address} - "
                    f"{ip_network.broadcast_address}"
                )
            if ace_dict["source_operator"]:
                if len(ace_dict["source_ports"].split()) == 2:
                    mansplained += (
                        f" where port {OPERATOR_NAMES[ace_dict['source_operator']]} "
                        f"{ace_dict['source_ports'].split()[0]} - {ace_dict['source_ports'].split()[1]}"
                    )
                else:
                    mansplained += (
                        f" where port {OPERATOR_NAMES[ace_dict['source_operator']]} "
                        f"{ace_dict['source_ports']}"
                    )
            if ace_dict["source_modifier"]:
                if ace_dict["source_modifier"] == "established":
                    mansplained += f" if the connection is already established"
                if ace_dict["source_modifier"] == "echo":
                    mansplained += f" for ICMP echo"
                if ace_dict["source_modifier"] == "echo-reply":
                    mansplained += f" for ICMP echo reply"
            if (
                "host" in ace_dict["destination_network"]
                or "/" not in ace_dict["destination_network"]
            ):
                mansplained += (
                    f", to IP address {ace_dict['destination_network'].split()[-1]}"
                )
            else:
                ip_network = ipaddress.IPv4Network(ace_dict["destination_network"])
                mansplained += f", to IP addresses {ip_network.network_address} - {ip_network.broadcast_address}"
            if ace_dict["destination_operator"]:
                if len(ace_dict["destination_ports"].split()) == 2:
                    mansplained += (
                        f" where port {OPERATOR_NAMES[ace_dict['destination_operator']]} "
                        f"{ace_dict['destination_ports'].split()[0]} - {ace_dict['destination_ports'].split()[1]}"
                    )
                else:
                    mansplained += (
                        f" where port {OPERATOR_NAMES[ace_dict['destination_operator']]} "
                        f"{ace_dict['destination_ports']}"
                    )
            if ace_dict["destination_modifier"]:
                if ace_dict["destination_modifier"] == "established":
                    mansplained += f" if the connection is already established"
                if ace_dict["destination_modifier"] == "echo":
                    mansplained += f" for ICMP echo"
                if ace_dict["destination_modifier"] == "echo-reply":
                    mansplained += f" for ICMP echo reply"
            if ace_dict["optional_action"]:
                mansplained += f", and {ace_dict['optional_action']}"
            print(f"{mansplained}\n")


if __name__ == "__main__":
    main()

Sunday 3 January 2021

Debugging .1x / RADIUS on 3850 Switches

It used to be you could simply enable the AAA and/or RADIUS debugs & review the outputs at your leisure via your favourite syslog analysis tool, such as Splunk or ELK stack. But on the 3650 & 3850 switches that's not the case, you have to use traces which aren't sent to syslog. Amended process below, though do note that the commands changed between IOS XE 3.x & 16.x releases.

 

First of all enable rotating the traces between files, so that you don't overwrite the outputs accidentally:

request platform software trace rotate all

Enable the traces that cover .1x, AAA & RADIUS:

set platform software trace smd R0 radius debug
set platform software trace smd R0 dot1x-all debug
set platform software trace smd R0 auth-mgr-all debug
set platform software trace smd R0 epm-all debug


Reproduce the issue & view the last 1,000 lines of traces:

show platform software trace message smd switch active R0

To view more traces requires exporting the traces for a set time period, then uploading off the switch:

request platform software trace archive last x days target flash:blah
copy flash:blah ftp:


Once you're done, return the traces to their usual state:

set platform software trace smd R0 radius notice
set platform software trace smd R0 dot1x-all notice
set platform software trace smd R0 auth-mgr-all notice
set platform software trace smd R0 epm-all notice



Which segways into why I'm writing about this...Had to diagnose wired .1x authentication failures that turned out to be a bug where a 3850 with equal cost uplinks (pretty common!) will chew up some of the RADIUS Access-Requests. Fun part is the RADIUS server never replies to the broken RADIUS message, so the switch then thinks the RADIUS server is timing out & marks it dead, which you'll see along with incrementing timeouts under show aaa servers. This was fixed in IOS XE 16.9.6.