This will (hopefully) be the last post of the Nexus 9k QoS series. At the end of the last post, policies have been applied:
- At the system level
- To FEX HIF ports
- To L3 Uplinks connected to the campus core
- To front-panel ports
Now we need to verify that the policies are implemented. Most of the policies were applied using the ‘no-stats’ option, which disables statistics for the policy and conserves a considerable amount of TCAM. For verification purposes, however, the policies were applied to some ports with statistics enabled.
Some problems were encountered during this process which will be explained as we go. First, we’ll see if traffic on the front-panel port connected to the NetApp is being classified and assigned to the proper qos-group for queueing:
n9k-1# show policy-map int e1/3
Global statistics status : enabled
Ethernet1/3
Service-policy (qos) input: POLICY-QOS-DEFAULT
SNMP Policy Index: 285227447
Class-map (qos): CLASS-QOS-EF (match-any)
Aggregate forwarded :
0 packets
Match: dscp 46
set qos-group 3
set cos 5
set dscp ef
Class-map (qos): CLASS-QOS-COS4 (match-any)
Slot 1
9831981894 packets
Aggregate forwarded :
9831981894 packets
Match: cos 4
set qos-group 2
set cos 4
(... output truncated ...)
Good, we were expecting iSCSI traffic to be marked by the NetApp as CoS 4, and it appears that this is the case. Time to see if the CoS 4 traffic is hitting the proper egress queue:
n9k-1# show queuing interface e1/3
slot 1
=======
Egress Queuing for Ethernet1/3 [System]
------------------------------------------------------------------------------
QoS-Group# Bandwidth% PrioLevel Shape QLimit
Min Max Units
------------------------------------------------------------------------------
3 - 1 - - - 9(D)
2 40 - - - - 9(D)
1 20 - - - - 9(D)
0 40 - - - - 9(D)
+-------------------------------------------------------------+
| QOS GROUP 0 |
+-------------------------------------------------------------+
| | Unicast |Multicast |
+-------------------------------------------------------------+
| Tx Pkts | 5424496382| 407436|
| Tx Byts | 16886447471756| 44509823|
| WRED/AFD & Tail Drop Pkts | 0| 0|
| WRED/AFD & Tail Drop Byts | 0| 0|
| Q Depth Byts | 0| 0|
| WD & Tail Drop Pkts | 0| 0|
+-------------------------------------------------------------+
| QOS GROUP 1 |
+-------------------------------------------------------------+
| | Unicast |Multicast |
+-------------------------------------------------------------+
| Tx Pkts | 0| 0|
| Tx Byts | 0| 0|
| WRED/AFD & Tail Drop Pkts | 0| 0|
| WRED/AFD & Tail Drop Byts | 0| 0|
| Q Depth Byts | 0| 0|
| WD & Tail Drop Pkts | 0| 0|
+-------------------------------------------------------------+
| QOS GROUP 2 |
+-------------------------------------------------------------+
| | Unicast |Multicast |
+-------------------------------------------------------------+
| Tx Pkts | 181054344| 34560|
| Tx Byts | 1307107319630| 4505960|
| WRED/AFD & Tail Drop Pkts | 0| 0|
| WRED/AFD & Tail Drop Byts | 0| 0|
| Q Depth Byts | 0| 0|
| WD & Tail Drop Pkts | 0| 0|
(... output truncated ...)
CoS 4 goes to qos-group 2 – looks good! Now, here’s where things get interesting. Our blades running ESXi are connected via B22HP FEX to the Nexus parents. The blades utilize virtual Distributed Switches (vDS) and are configured to classify and mark iSCSI traffic to CoS 4. Each blade is configured with an HP 530m 10Gb adapter (Broadcom 57810) with a dependent iSCSI adapter used for storage connectivity. Checking the policy statistics and queuing information returns an unexpected result:
n9k-1# show policy-map int e163/1/4
Global statistics status : enabled
Ethernet163/1/4
Service-policy (qos) input: POLICY-QOS-GLOBAL
SNMP Policy Index: 285222899
Class-map (qos): CLASS-QOS-COS5 (match-all)
Aggregate forwarded :
0 packets
Match: cos 5
set qos-group 3
set cos 5
Class-map (qos): CLASS-QOS-COS4 (match-any)
Aggregate forwarded :
0 packets
Match: cos 4
set qos-group 2
set cos 4
(... output truncated ...)
n9k-1# show queuing interface e163/1/4 slot 1 ======= Ethernet163/1/4 queuing information: ... Queueing: queue qos-group cos priority bandwidth mtu --------+------------+--------------+---------+---------+---- ctrl-hi n/a 7 PRI 0 2400 ctrl-lo n/a 7 PRI 0 2400 2 0 0 1 2 6 WRR 40 9280 3 1 3 WRR 20 9280 4 2 4 WRR 40 9280 5 3 5 PRI 0 9280 Queue limit: 66560 bytes Queue Statistics: queue rx tx flags ------+---------------+---------------+----- 0 0 132858 ctrl 1 92897 505 ctrl 2 2206248378 714035505 data 3 3965968809 4152889941 data 4 0 1293406937 data 5 0 291815 data
We are interested in CoS 4, which maps to queue 4. Looking at the statistics for queue 4, we see traffic being transmitted (i.e. outbound to the blade), but nothing received. It appears that the vDS marking policy is not being applied to iSCSI traffic.
Since starting this post, we have begun upgrading our vSphere hosts to 6.5. It turns out that the HP 530m dependent iSCSI adapter is not supported in this release, so we moved the vmk interfaces to the VMware software iSCSI initiator. Checking out the policy and queuing stats on a HIF which is connected to the software iSCSI initiator reveals the following:
n9k-1# show policy-map int e163/1/3
Global statistics status : enabled
Ethernet163/1/3
Service-policy (qos) input: POLICY-QOS-GLOBAL
SNMP Policy Index: 285222884
Class-map (qos): CLASS-QOS-COS5 (match-all)
Aggregate forwarded :
0 packets
Match: cos 5
set qos-group 3
set cos 5
Class-map (qos): CLASS-QOS-COS4 (match-any)
Slot 1
150296140 packets
Aggregate forwarded :
150296140 packets
Match: cos 4
set qos-group 2
set cos 4
(... output truncated ...)
n9k-1# show queuing interface e163/1/3 slot 1 ======= Ethernet163/1/3 queuing information: Input buffer allocation: Qos-group: ctrl frh: 0 drop-type: drop cos: 7 xon xoff buffer-size ---------+---------+----------- 2560 7680 10240 Qos-group: 0 1 2 3 (shared) frh: 8 drop-type: drop cos: 0 1 2 3 4 5 6 xon xoff buffer-size ---------+---------+----------- 0 126720 151040 Queueing: queue qos-group cos priority bandwidth mtu --------+------------+--------------+---------+---------+---- ctrl-hi n/a 7 PRI 0 2400 ctrl-lo n/a 7 PRI 0 2400 2 0 0 1 2 6 WRR 40 9280 3 1 3 WRR 20 9280 4 2 4 WRR 40 9280 5 3 5 PRI 0 9280 Queue limit: 66560 bytes Queue Statistics: queue rx tx flags ------+---------------+---------------+----- 0 0 957929 ctrl 1 174532 4364 ctrl 2 1941921266 5518939407 data 3 1935850666 16630206930 data 4 150299756 8947590636 data 5 0 2077494 data (... output truncated ...)
Well, that’s interesting – again, CoS 4 maps to queue 4, which now shows hits on the transmit and receive side. Now the vDS appears to be marking the iSCSI traffic as expected!
I’m still researching this, but it appears that iSCSI traffic bypasses the vDS Traffic Filtering and Marking policies when a dependent iSCSI adapter is utilized. I didn’t anticipate this behavior, as the dependent adapter requires a vmk interface to be created and bound to the hardware. The vmk is bound to a port group on the vDS; thus, the expected behavior would be for traffic entering or leaving the vmk would be affected by the vDS QoS policy. This appears to be the case for the same vmk interface when bound to the software iSCSI initiator but binding to the dependent adapter alters this behavior.
This post was delayed for some time due to the unexpected iSCSI behavior. I’ve decided that it’s time to move on from this issue and complete this article but will post an update once the cause of the missing marking is discovered. If you know the reason for this, please reach out and set me straight. In the meantime, keep networking!
-P