I’m in the process of deploying a pair of Nexus 93180YC-EX switches for our new campus, and have learned that the APIC-EM EasyQoS module does not support the 9k platform. For those unfamiliar:
“The EasyQoS solution abstracts QoS policy by using a declarative model as opposed to an imperative model. A declarative model focuses on the intent or WHAT is to be accomplished, without describing HOW it is to be accomplished.” (http://www.cisco.com/c/en/us/td/docs/solutions/CVD/Dec2016/APIC-EM-EasyQoS-DesignGuide-Dec2016.html#_Toc469388744)
I’ve been playing with APIC-EM and like that it simplifies QoS design for the campus, especially because I can visually involve management in the QoS strategy. Without support for the 9k, we’re back to using the MQC and classifying traffic with ACLs. Not a big deal, until you realize that creating a comprehensive QoS policy on the 9k can quickly exhaust the TCAM with the default allocations.
A brief overview of this deployment:
Enterprise Services licenses were not purchased so the 9K switches will be operating in L2 mode. Layer 3 will be handled in the campus core, a pair of Catalyst 6880s running VSS. 2248TP-E FEXs will be hanging off the 9Ks in a straight-through configuration, with dual-homed servers connected via vPC. QoS marking, classification, and queuing will be configured based on the company’s defined business-important traffic and attached to host interfaces on the 9k or FEX.
Because of the potentially high number of Access Control Entries (ACEs) in the ACLs used for traffic classification, it’s important to understand how TCAM space will be consumed in order to prevent interfaces from being err-disabled due to TCAM exhaustion. This understanding is fundamentally the same as with the Catalyst switches, so this series of posts is being written with the following objectives:
- Where possible, identify the differences between the 93180YC-EX (and, presumably, other second-generation 9300 series Nexus switches) and the original product line. These observations are based on my experience configuring the 93180 versus the documentation available from Cisco.
- Demonstrate methods for determining TCAM utilization and estimating TCAM space requirements for QoS.
- Identify opportunities to optimize TCAM utilization in order to reduce the potential for resource exhaustion.
Before continuing, we should get an idea of our initial constraints. The default TCAM allocation on the 93180 may be displayed with the command show hardware access-list tcam region and is as follows:
NAT ACL[nat] size = 0 Ingress PACL [ing-ifacl] size = 0 VACL [vacl] size = 0 Ingress RACL [ing-racl] size = 1536 Ingress RBACL [ing-rbacl] size = 0 Ingress L2 QOS [ing-l2-qos] size = 512 Ingress L3/VLAN QOS [ing-l3-vlan-qos] size = 512 Ingress SUP [ing-sup] size = 512 Ingress L2 SPAN filter [ing-l2-span-filter] size = 256 Ingress L3 SPAN filter [ing-l3-span-filter] size = 256 Ingress FSTAT [ing-fstat] size = 0 span [span] size = 512 Egress RACL [egr-racl] size = 1536 Egress SUP [egr-sup] size = 256 Ingress Redirect [ing-redirect] size = 0
For this deployment, created policy maps will be applied to interfaces and not VLANs or SVIs, so we’re mainly concerned with the ing-l2-qos TCAM size. 512 entries – this is not very large compared to the 6880-X-LE which supports 64,000 entries for security and QoS!
To start this series off, let’s look at a simple ACL which defines some network management ports for classification and the resulting TCAM utilization:
ip access-list v4-NETWORK-MGMT 10 remark Microsoft Global Catalog 20 permit tcp any any eq 3268 30 permit tcp any eq 3268 any 40 permit udp any any eq 3268 50 permit udp any eq 3268 any 60 permit tcp any any eq 3269 70 permit tcp any eq 3269 any 80 permit udp any any eq 3269 90 permit udp any eq 3269 any 100 remark RADIUS 110 permit udp any any eq 1645 120 permit udp any eq 1645 any 130 permit udp any any eq 1646 140 permit udp any eq 1646 any 150 permit udp any any eq 1812 160 permit udp any eq 1812 any 170 permit udp any any eq 1813 180 permit udp any eq 1813 any 190 remark SNMP 200 permit tcp any any eq 161 210 permit tcp any eq 161 any 220 permit tcp any any eq 162 230 permit tcp any eq 162 any 240 permit tcp any any eq 1161 250 permit tcp any eq 1161 any 260 permit tcp any any eq 1162 270 permit tcp any eq 1162 any 280 permit udp any any eq snmp 290 permit udp any eq snmp any 300 permit udp any any eq snmptrap 310 permit udp any eq snmptrap any 320 remark SSH 330 permit tcp any any eq 22 340 permit tcp any eq 22 any 350 remark SYSLOG 360 permit udp any eq syslog any 370 permit udp any any eq syslog
We’ll also create a basic class-map and policy-map for marking, and apply it to interface Port-Channel 1101 (a FEX vPC port):
class-map type qos match-any INPUT-NETWORK-MGMT description Network Management Traffic match access-group name v4-NETWORK-MGMT ! policy-map type qos IN-UNTRUSTED description Inbound classification and marking for untrusted ports class INPUT-NETWORK-MGMT set dscp cs2 set qos-group 2 class class-default set qos-group 0 ! interface po1101 service-policy type qos input IN-UNTRUSTED
OK, let’s check our TCAM utilization. Some simplified explanations of TCAM state that one ACE will result in one TCAM entry. Is this true? Based on the “ACE = TCAM entry” logic, let’s see how many ‘permit’ ACEs are in place:
switch-9k-1# show ip access-list v4-NETWORK-MGMT | i permit | wc -l 32
So there should be 32 TCAM entries, right? Let’s verify:
switch-9k-1# show hardware access-list resource utilization | b 0x1
...
ACL Hardware Resource Utilization (Mod 1)
----------------------------------------------------------
Used Free Percent
Utilization
-------------------------------------------------------------------
Ingress L2 QOS 36 476 7.03
Ingress L2 QOS IPv4 33 6.45
Ingress L2 QOS IPv6 2 0.39
Ingress L2 QOS MAC 1 0.20
Wait, we didn’t create any IPv6 or MAC access lists, and there’s one more IPv4 entry than expected. Where are those coming from? Let’s check the entries applied to the Po1101 interface:
switch-9k-1# show system internal access-list input entries detail | b 1101 VDC-1 port-channel1101 : ==================== INSTANCE 0x1 --------------- Tcam 0 resource usage: ---------------------- LBL A = 0x2 Bank 1 ------ IPv4 Class Policies: QoS Netflow profile: 0 Netflow deny profile: 0 Entries: [Index] Entry [Stats] --------------------- [0x0010:0x0001:0x0601] permit tcp 0.0.0.0/0 0.0.0.0/0 eq 3268 [0] [0x001f:0x0002:0x0602] permit tcp 0.0.0.0/0 eq 3268 0.0.0.0/0 [0] [0x0038:0x0005:0x0605] permit udp 0.0.0.0/0 0.0.0.0/0 eq 3268 [0] [0x003b:0x0006:0x0606] permit udp 0.0.0.0/0 eq 3268 0.0.0.0/0 [0] [0x0001:0x0009:0x0609] permit tcp 0.0.0.0/0 0.0.0.0/0 eq 3269 [0] [0x0002:0x000a:0x060a] permit tcp 0.0.0.0/0 eq 3269 0.0.0.0/0 [0] [0x0005:0x000d:0x060d] permit udp 0.0.0.0/0 0.0.0.0/0 eq 3269 [0] [0x0006:0x000e:0x060e] permit udp 0.0.0.0/0 eq 3269 0.0.0.0/0 [0] [0x0009:0x0010:0x0610] permit udp 0.0.0.0/0 0.0.0.0/0 eq 1645 [0] [0x000a:0x0013:0x0613] permit udp 0.0.0.0/0 eq 1645 0.0.0.0/0 [0] [0x000d:0x0014:0x0614] permit udp 0.0.0.0/0 0.0.0.0/0 eq 1646 [0] [0x000e:0x0017:0x0617] permit udp 0.0.0.0/0 eq 1646 0.0.0.0/0 [0] [0x0013:0x0018:0x0618] permit udp 0.0.0.0/0 0.0.0.0/0 eq 1812 [0] [0x0014:0x001b:0x061b] permit udp 0.0.0.0/0 eq 1812 0.0.0.0/0 [0] [0x0017:0x001c:0x061c] permit udp 0.0.0.0/0 0.0.0.0/0 eq 1813 [0] [0x0018:0x001f:0x061f] permit udp 0.0.0.0/0 eq 1813 0.0.0.0/0 [0] [0x001b:0x0021:0x0621] permit tcp 0.0.0.0/0 0.0.0.0/0 eq 161 [0] [0x001c:0x0022:0x0622] permit tcp 0.0.0.0/0 eq 161 0.0.0.0/0 [0] [0x0021:0x0025:0x0625] permit tcp 0.0.0.0/0 0.0.0.0/0 eq 162 [0] [0x0022:0x0026:0x0626] permit tcp 0.0.0.0/0 eq 162 0.0.0.0/0 [0] [0x0025:0x0029:0x0629] permit tcp 0.0.0.0/0 0.0.0.0/0 eq 1161 [0] [0x0026:0x002a:0x062a] permit tcp 0.0.0.0/0 eq 1161 0.0.0.0/0 [0] [0x0029:0x002d:0x062d] permit tcp 0.0.0.0/0 0.0.0.0/0 eq 1162 [0] [0x002a:0x002e:0x062e] permit tcp 0.0.0.0/0 eq 1162 0.0.0.0/0 [0] [0x002d:0x0031:0x0631] permit udp 0.0.0.0/0 0.0.0.0/0 eq 161 [0] [0x002e:0x0032:0x0632] permit udp 0.0.0.0/0 eq 161 0.0.0.0/0 [0] [0x0031:0x0035:0x0635] permit udp 0.0.0.0/0 0.0.0.0/0 eq 162 [0] [0x0032:0x0036:0x0636] permit udp 0.0.0.0/0 eq 162 0.0.0.0/0 [0] [0x0035:0x0038:0x0638] permit tcp 0.0.0.0/0 0.0.0.0/0 eq 22 [0] [0x0036:0x003b:0x063b] permit tcp 0.0.0.0/0 eq 22 0.0.0.0/0 [0] [0x003d:0x003d:0x063d] permit udp 0.0.0.0/0 eq 514 0.0.0.0/0 [0] [0x003e:0x003e:0x063e] permit udp 0.0.0.0/0 0.0.0.0/0 eq 514 [0] [0x0044:0x0044:0x0644] permit ip 0.0.0.0/0 0.0.0.0/0 [0] IPv6 Class Policies: QoS Netflow profile: 0 Netflow deny profile: 0 Entries: [Index] Entry [Stats] --------------------- [0x0045:0x0046:0x0646] permit ipv6 0000:0000:0000:0000:0000:0000:0000:0000/0 0 000:0000:0000:0000:0000:0000:0000:0000/0 [0] MAC Class Policies: QoS Netflow profile: 0 Netflow deny profile: 0 Entries: [Index] Entry [Stats] --------------------- [0x0047:0x0045:0x0645] permit 0000.0000.0000 0000.0000.0000 0000.0000.0000 000 0.0000.0000 [0]
Got it. The last entry for each is a permit any – this must have been created by class class-default in the policy map. That would explain the additional IPv4 and MAC TCAM entries, but why are there two L2 QOS IPv6 entries?
Per Cisco, “The IPv4 TCAM regions are single wide. The IPv6, quality of service (QoS), MAC, control-plane policing (CoPP) , and system TCAM regions are double wide and consume double the physical TCAM entries. For example, a logical region size of 256 entries actually consumes 512 physical TCAM entries.”
In addition, “For the ACI leaf line card on Cisco Nexus 9300 Series switches, only the IPv6 TCAM regions consume double-wide entries. The rest of the TCAM regions consume single-wide entries.”
This is a Nexus 9300 series, but (to my knowledge) no ACI leaf line card is installed. It appears that there are some platform differences between the original 9300 series and the “version 2” 9300-EX series – this being one (you’ll also note that the Cisco 9000 TCAM Carving documentation shows many more TCAM regions). Thus, it appears that the 93180YC-EX uses a single wide TCAM for IPv4 and MAC regions, and a double wide TCAM for IPv6 (note – the ‘double wide TCAM for IPv6 may not be accurate in all cases; I’ve seen some odd behavior with this and will need to perform more testing). This explains our two entries above.
For ease of explanation and calculation, let’s remove the class-default definition from the IN-UNTRUSTED policy-map:
policy-map type qos IN-UNTRUSTED no class class-default
And verify that our TCAM count matches our ACEs:
switch-9k-1# show hardware access-list resource utilization | b 0x1
...
ACL Hardware Resource Utilization (Mod 1)
----------------------------------------------------------
Used Free Percent
Utilization
-------------------------------------------------------------------
Ingress L2 QOS 32 480 6.25
Ingress L2 QOS IPv4 32 6.25
Ingress L2 QOS IPv6 0 0.00
Ingress L2 QOS MAC 0 0.00
Good stuff. 32 ACEs, 32 IPv4 TCAM entries. No IPv6 or MAC entries. It looks like each ACE resulted in a TCAM entry, but will this continue to be the case?
In the next post, we’ll look at TCAM utilization in a bit more detail so we can begin to configure the 9k’s allocation in a manner which will fulfill the requirements of this network.