In the last post, we examined TCAM utilization for the following access list:
ip access-list v4-NETWORK-MGMT 10 permit tcp any any range 721 1035
Expansion of this range gave us 8 TCAM entries:
switch-9k-1# show system internal access-list input entries detail ... [0x0000:0x0000:0x0600] permit tcp 0.0.0.0/0 0.0.0.0/0 range 1032 1035 [0] [0x0001:0x0001:0x0601] permit tcp 0.0.0.0/0 0.0.0.0/0 range 1024 1031 [0] [0x0002:0x0002:0x0602] permit tcp 0.0.0.0/0 0.0.0.0/0 eq 721 [0] [0x0003:0x0003:0x0603] permit tcp 0.0.0.0/0 0.0.0.0/0 range 722 723 [0] [0x0004:0x0004:0x0604] permit tcp 0.0.0.0/0 0.0.0.0/0 range 724 727 [0] [0x0005:0x0005:0x0605] permit tcp 0.0.0.0/0 0.0.0.0/0 range 728 735 [0] [0x0006:0x0006:0x0606] permit tcp 0.0.0.0/0 0.0.0.0/0 range 736 767 [0] [0x0007:0x0007:0x0607] permit tcp 0.0.0.0/0 0.0.0.0/0 range 768 1023 [0]
We know that TCAM operates on masks for rapid lookups, and we have seen how quickly our limited TCAM can be consumed. What happens if we add source or destination addresses into the mix?
no ip access-list v4-NETWORK-MGMT ! ip access-list v4-NETWORK-MGMT 10 permit tcp host 192.168.1.1 any range 721 1035
This example is easy – we have the same ACL as before, only with a source host. The number of TCAM entries will remain the same. This would also hold true if we included a destination host – the number of entries would not change. The same applies if we would change the ACL to use a source port range with a destination port of ‘any’
no ip access-list v4-NETWORK-MGMT ! ip access-list v4-NETWORK-MGMT 10 permit tcp host 192.168.1.1 range 721 1035 any
Still using 8 TCAM entries. What happens if we include a source AND destination port range?
no ip access-list v4-NETWORK-MGMT ! ip access-list v4-NETWORK-MGMT 10 permit tcp any range 721 1035 any range 721 1035
As stated in the support forum document by Yogesh Ramdoss, we should “… calculate the expansion factor for source and destination port ranges individually… and the final expansion factor is the product of the two (m = src_m * dest_m).” Since we used the same port ranges for the source and destination, we know the expansion factor is 8, so 8 * 8 = 64. This is because each expansion range must be compared to each other to obtain the whole lookup table. For example, for the range 768-1023, entries must be created which compare the range 768-1023 to port ranges 721, 722-723, 724-727, 728-735, 736-767, 768-1023, 1024-1031, and 1032-1035.
What about the following ACL?
no ip access-list v4-NETWORK-MGMT ! ip access-list v4-NETWORK-MGMT 10 permit tcp host 192.168.1.1 any range 721 1035 20 permit tcp host 192.168.1.2 any range 721 1035 30 permit tcp any range 721 1035 host 192.168.1.1 40 permit tcp any range 721 1035 host 192.168.1.2
Each ACE contains a single source or destination port and a single port range. We know this port range gives 8 TCAM entries, so the final TCAM usage will be (2 source IPs * 8 TCAM entries + 2 destination IPs * 8 TCAM entries) = 32 total TCAM entries. What if we include both source and destination port ranges with source and destination IP addresses?
no ip access-list v4-NETWORK-MGMT ! ip access-list v4-NETWORK-MGMT 10 permit tcp host 192.168.1.1 range 721 1035 any range 721 1035 20 permit tcp host 192.168.1.2 range 721 1035 any range 721 1035 30 permit tcp any range 721 1035 host 192.168.1.1 range 721 1035 40 permit tcp any range 721 1035 host 192.168.1.2 range 721 1035
Because we know that each ACE with a source and destination port range of 721-1035 may be calculated as (8 * 8 = 64) TCAM entries, there are 2 source IPs, and there are 2 destination IPs, we can calculate (2 * 2 * 64) = 256 total TCAM entries. To verify:
switch-9k-1# show hardware access-list resource utilization | b 0x1
...
----------------------------------------------
Used Free Percent
Utilization
-----------------------------------------------
Ingress L2 QOS 256 256 50.00
Ingress L2 QOS IPv4 256 50.00
Note the the above 4-line ACL now consumes 50% of our available TCAM! What happens if we include multiple source and destination hosts within subnets?
no ip access-list v4-NETWORK-MGMT ! ip access-list v4-NETWORK-MGMT 10 permit tcp 192.168.1.4/30 range 721 1035 any range 721 1035 20 permit tcp 192.168.1.16/28 range 721 1035 any range 721 1035 30 permit tcp any range 721 1035 192.168.1.4/30 range 721 1035 40 permit tcp any range 721 1035 192.168.1.16/28 range 721 1035
switch-9k-1# show hardware access-list resource utilization | b 0x1
...
----------------------------------------------
Used Free Percent
Utilization
-----------------------------------------------
Ingress L2 QOS 256 256 50.00
Ingress L2 QOS IPv4 256 50.00
Why has the utilization remained the same? Remember the TCAM operates on masks! Instead of a single host (/32), we are using masks to include multiple hosts, so the effective calculation is the same.
Let’s move to the next topic, Logical Operation Units or LOUs. In the real world, with the default hardware configuration, we will not see 256 TCAM entries for the above ACL. We will see many less:
switch-9k-1# show hardware access-list resource utilization | b 0x1
...
----------------------------------------------
Used Free Percent
Utilization
-----------------------------------------------
Ingress L2 QOS 4 508 0.78
Ingress L2 QOS IPv4 4 0.78
What happened?
To increase the speed at which port ranges with large expansion coefficients may be matched, switches allocate LOUs which “compress” a port range into a single lookup value. On the 9k, we can see the threshold at which an LOU is created for an expansion range:
switch-9k-1# show system internal access-list globals ... LOU Threshold Value : 5
Once a port range expands to greater than 5 TCAM entries, a single LOU is created for that port range. For the examples in the previous posts, I had increased this threshold to 15 for demonstration purposes using the command:
switch-9k-1# hardware access-list lou resource threshold 15
We can see LOU usage:
switch-9k-1# show hardware access-list resource utilization | b 0x1 ...
----------------------------------------------
Used Free Percent
Utilization
-----------------------------------------------
LOU 4 12 25.00 Both LOU Operands 4 Single LOU Operands 0 LOU L4 src port: 2 LOU L4 dst port: 2
This is the default LOU utilization on the 9k, based on the out-of-box CoPP policy. Once we apply the 4-line access list which would normally expand to 256 entries to the switch, we can see what happens with the LOU allocation and resulting TCAM entries:
switch-9k-1# show hardware access-list resource utilization | b 0x1
...
----------------------------------------------
Used Free Percent
Utilization
-----------------------------------------------
Ingress L2 QOS 4 508 0.78
Ingress L2 QOS IPv4 4 0.78
...
LOU 6 10 37.50
Both LOU Operands 6
Single LOU Operands 0
LOU L4 src port: 3
LOU L4 dst port: 3
switch-9k-1# show system internal access-list input entries detail
...
[0x0001:0x0001:0x0601] permit tcp 192.168.1.4/30 range 721 1035 0.0.0.0/0 range 721 1035 [0]
[0x0002:0x0002:0x0602] permit tcp 192.168.1.16/28 range 721 1035 0.0.0.0/0 range 721 1035 [0]
[0x0005:0x0005:0x0605] permit tcp 0.0.0.0/0 range 721 1035 192.168.1.4/30 range 721 1035 [0]
[0x0006:0x0006:0x0606] permit tcp 0.0.0.0/0 range 721 1035 192.168.1.16/28 range 721 1035 [0]
We can see from the above that one additional src port LOU was created for the source range 721-1035 and one dst port LOU created for the destination port range 721-1035. These LOUs will be used each time we need to match 721-1035, regardless of the number of IP source or destination ACEs. This reduces TCAM utilization from 256 entries to 4 – a critical feature when we’re dealing with such limited space! Note, however, that we have another potential issue – LOU exhaustion.
With proper planning and an understanding of TCAM utilization, we should now be able to move to the next step: carving the TCAM, which is covered in the next post.