I admin a handful of blades in an HP BladeSystem. We have recently transitioned from using HP 6120xg switches for uplinks to a pair of Cisco B22HP Fabric Extenders connected to parent Nexus 93180YC-EX switches. Each blade is running VMware vSphere 6.0 Enterprise Plus, with virtual Distributed Switches (vDS). Most of the blades are HPE BL460c Gen8. We recently purchased a BL460c Gen9 and have run into problems adding it into the cluster.
Hardware note: the Gen9 is outfitted with onboard 536FLB (Broadcom 57840 based) and 534M Mezzanine (Broadcom 57810) Converged Network Adapters (CNAs). My problem was with the 534M, but it’s possible that the 536FLB would have the same issue detailed here.
The problem:
We run jumbo frames (9000-byte MTU) for our iSCSI SAN. MTU end-to-end on the parent Nexus 9k and the FEX ports is set at 9216. All blades except the Gen9 can successfully pass 9k frames to the NetApp. The Gen9 server is dropping ICMP frames larger than 2344 bytes with the DF bit set.