One issue I’ve encountered with the Nexus 9k (specifically, the 93180YC-EX) is that it has very limited TCAM space. I’m guessing this is because the vision for the 9k involves implementation of ACI and eventual migration of the control plane to APIC… but the 9k does make a very cost effective core/aggregation switch for smaller deployments if you don’t need all the features of the 7k. That being said, I’ve been working on implementing QoS on the 9k and have had to review the essentials of TCAM in order to properly carve out enough space to handle the number of Access Control Entries (ACEs) for classification and marking.
Simply counting the number of ACEs will not give an accurate approximation of the number of TCAM entries required. It’s very easy to err-disable ports when applying a policy map due to TCAM exhaustion.
Accurately determining TCAM requirements is beyond the scope of this post; an in-depth discussion written by Yogesh Ramdoss may be found in the Cisco Support Forums and will be discussed on this site in follow-up postings.
As a networking professional, precision in planning is important. However, there is an argument to be made for “close enough” when precision comes at the expense of a very limited time budget – and when “close enough” will give a slight over-estimation for budgetary concerns; in this case, the number of TCAM entries.
Now, about that TCAM utilization…
As already stated, the 93180 has a very limited TCAM. Here are the default allocations:
NAT ACL[nat] size = 0 Ingress PACL [ing-ifacl] size = 0 VACL [vacl] size = 0 Ingress RACL [ing-racl] size = 1536 Ingress RBACL [ing-rbacl] size = 0 Ingress L2 QOS [ing-l2-qos] size = 512 Ingress L3/VLAN QOS [ing-l3-vlan-qos] size = 512 Ingress SUP [ing-sup] size = 512 Ingress L2 SPAN filter [ing-l2-span-filter] size = 256 Ingress L3 SPAN filter [ing-l3-span-filter] size = 256 Ingress FSTAT [ing-fstat] size = 0 span [span] size = 512 Egress RACL [egr-racl] size = 1536 Egress SUP [egr-sup] size = 256 Ingress Redirect [ing-redirect] size = 0
We’ll be applying policy maps to interfaces, and the switches are running in L2 mode, so the region I’m concerned with is ‘ing-l2-qos.’ There are 512 entries available. According to Cisco, all QoS features are double-width, so classifying IPv4 and IPv6 packets with the following ACLs will result in the same number of entries:
ip access-list v4-NETWORK-MGMT 10 permit tcp any any eq 22 ipv6 access-list v6-NETWORK-MGMT 10 permit tcp any any eq 22
One TCAM entry will used for the IPv4 ACE, and one entry will be used for the IPv6 ACE. .4% of the available entries used for two ACEs. It won’t take too long to consume all 512 entries with a comprehensive set of ACLs for traffic classification. Now, let’s look at a more realistic ACL which includes some port ranges:
ip access-list v4-NETWORK-MGMT ! Mark to CS2 10 remark Ops and Admin Management 20 remark Microsoft Global Catalog 30 permit tcp any any range 3268 3269 40 permit tcp any range 3268 3269 any 50 permit udp any any range 3268 3269 60 permit udp any range 3268 3269 any 70 remark NTP 80 permit udp any any eq 123 90 permit udp any eq 123 any 100 remark RADIUS 110 permit udp any any range 1645 1646 120 permit udp any range 1645 1646 any 130 permit udp any any range 1812 1813 140 permit udp any range 1812 1813 any 150 remark SNMP 160 permit tcp any any range 161 162 170 permit tcp any range 161 162 any 180 permit udp any any range 161 162 190 permit udp any range 161 162 any 200 permit tcp any any range 1161 1162 210 permit tcp any range 1161 1162 any 220 remark SSH 230 permit tcp any any eq 22 240 permit tcp any eq 22 any 250 remark SYSLOG 260 permit udp any eq 514 any 270 permit udp any any eq 514 280 remark TFTP 290 permit udp any any eq 69 300 permit udp any eq 69 any
The actual ACL which will be applied has many more lines, but the above is a good starting point. How many TCAM entries will be consumed for this ACL? Will it be the number of ACEs present?
switch-9k-1# show ip access-list v4-NETWORK-MGMT | i permit | wc -l
22 !! 22 ACEs - what's our TCAM utilization?
switch-9k-1# show hardware access-list resource utilization | b 0x1
...
Used Free Percent
Utilization
----------------------------------------------
Ingress L2 QOS 30 482 5.86
Ingress L2 QOS IPv4 30 5.86
Why are there 30 entries for 22 ACEs? Let’s see what’s going on:
switch-9k-1# show system internal access-list input entries detail | b 1101 ... [0x0008:0x0001:0x0601] permit tcp 0.0.0.0/0 0.0.0.0/0 range 3268 3269 [0] [0x000b:0x0002:0x0602] permit tcp 0.0.0.0/0 range 3268 3269 0.0.0.0/0 [0] [0x0034:0x0005:0x0605] permit udp 0.0.0.0/0 0.0.0.0/0 range 3268 3269 [0] [0x0037:0x0006:0x0606] permit udp 0.0.0.0/0 range 3268 3269 0.0.0.0/0 [0] [0x0001:0x0008:0x0608] permit udp 0.0.0.0/0 0.0.0.0/0 eq 123 [0] [0x0002:0x000b:0x060b] permit udp 0.0.0.0/0 eq 123 0.0.0.0/0 [0] [0x0005:0x000e:0x060e] permit udp 0.0.0.0/0 0.0.0.0/0 eq 1646 [0] [0x0006:0x000f:0x060f] permit udp 0.0.0.0/0 0.0.0.0/0 eq 1645 [0] [0x000e:0x0010:0x0610] permit udp 0.0.0.0/0 eq 1646 0.0.0.0/0 [0] [0x000f:0x0011:0x0611] permit udp 0.0.0.0/0 eq 1645 0.0.0.0/0 [0] [0x0010:0x0015:0x0615] permit udp 0.0.0.0/0 0.0.0.0/0 range 1812 1813 [0] [0x0011:0x0016:0x0616] permit udp 0.0.0.0/0 range 1812 1813 0.0.0.0/0 [0] [0x0015:0x0018:0x0618] permit tcp 0.0.0.0/0 0.0.0.0/0 eq 162 [0] [0x0016:0x0019:0x0619] permit tcp 0.0.0.0/0 0.0.0.0/0 eq 161 [0] [0x0018:0x001e:0x061e] permit tcp 0.0.0.0/0 eq 162 0.0.0.0/0 [0] [0x0019:0x001f:0x061f] permit tcp 0.0.0.0/0 eq 161 0.0.0.0/0 [0] [0x001e:0x0020:0x0620] permit udp 0.0.0.0/0 0.0.0.0/0 eq 162 [0] [0x001f:0x0021:0x0621] permit udp 0.0.0.0/0 0.0.0.0/0 eq 161 [0] [0x0020:0x0026:0x0626] permit udp 0.0.0.0/0 eq 162 0.0.0.0/0 [0] [0x0021:0x0027:0x0627] permit udp 0.0.0.0/0 eq 161 0.0.0.0/0 [0] [0x0026:0x0028:0x0628] permit tcp 0.0.0.0/0 0.0.0.0/0 eq 1162 [0] [0x0027:0x0029:0x0629] permit tcp 0.0.0.0/0 0.0.0.0/0 eq 1161 [0] [0x0028:0x002e:0x062e] permit tcp 0.0.0.0/0 eq 1162 0.0.0.0/0 [0] [0x0029:0x002f:0x062f] permit tcp 0.0.0.0/0 eq 1161 0.0.0.0/0 [0] [0x002e:0x0031:0x0631] permit tcp 0.0.0.0/0 0.0.0.0/0 eq 22 [0] [0x002f:0x0032:0x0632] permit tcp 0.0.0.0/0 eq 22 0.0.0.0/0 [0] [0x0031:0x0034:0x0634] permit udp 0.0.0.0/0 eq 514 0.0.0.0/0 [0] [0x0032:0x0037:0x0637] permit udp 0.0.0.0/0 0.0.0.0/0 eq 514 [0] [0x0039:0x0039:0x0639] permit udp 0.0.0.0/0 0.0.0.0/0 eq 69 [0] [0x003a:0x003a:0x063a] permit udp 0.0.0.0/0 eq 69 0.0.0.0/0 [0]
If refer to the original ACL, we will see the bold lines correspond to:
110 permit udp any any range 1645 1646 120 permit udp any range 1645 1646 any ... 160 permit tcp any any range 161 162 170 permit tcp any range 161 162 any 180 permit udp any any range 161 162 190 permit udp any range 161 162 any 200 permit tcp any any range 1161 1162 210 permit tcp any range 1161 1162 any
What happened?
To expedite lookups, TCAM compiles entries with masks – similar to the subnet masks we deal with when allocating addresses. You can see above that the ranges for 3268-3269 and 1812-1813 consumed single entries in TCAM, but the highlighted ranges for 1645-1646, 161-162, 1161-1162 were expanded into one entry per port. There’s no binary mask possible which would include the port pairs. Let’s see:
0000 0110 0110 1101 = 1645 0000 0110 0110 1110 = 1646 0000 0110 0110 1100 = 1644 !! Smallest inclusive mask
The smallest mask which would include both 1645 and 1646 is shown above – this mask would include ports 1644 – 1647. As we didn’t include these in the ACL, there is no option but to create a separate TCAM entry for each port. The same with range 161-162:
0000 0000 1010 0001 = 161 0000 0000 1010 0010 = 162 0000 0000 1010 0000 = 160 !! Smallest inclusive mask
Note that the port numbers are represented as 16-bit integers; this is required to support the full range of 65,536 available ports.
If this seems familiar, it’s because these basics are taught at the CCNA level for determining if two addresses reside in the same network for VLSM! In the next post, we’ll look at a more complex example and figure out how to get “close enough” to effectively carve the limited TCAM space on the Nexus 9k.