Broadcom Driver Woes

Shortly after deploying the Broadcom NetXtreme II v3.7.19 driver for the 5708 chipset to our Metaframe environment, the users were not happy campers. There ended up being two severe problems that we noticed in this particular revision of the drivers. Today I'm going to describe problem number one, the TCP Offload Engine feature (TOE).

Now we actually ran into this issue 3-4 months ago, so I don't remember what version of the driver we had before 3.7.19.

Once updated, everything appeared fine. Ping tests showed connectivity and expected latency. A traffic simulation tool PCATTCP showed that throughput was being handled as expected; the servers were stable.

The next day when users connected to theses Metaframe servers, they started complaining almost instantly of an obscure symptom, (they always are) slow response to operations in Outlook. For the record, we used Outlook 2003 clients connected to Exchange 2003. Cached Exchange Mode was off, and the driver was only loaded onto the client end.

Now I was baffled to see that the preview pane in Outlook was taking about 5 seconds to update per email, instead of perhaps 200ms. My previous test demonstrated that the NIC driver had achieved far greater throughput than what Outlook was demanding, and that ICMP latency had not increased at all. Surely the NIC doesn't know the difference between ICMP traffic and Outlook traffic! That would defy everything I know of the Networking models.

We ran up Outlook with the /RPCDIAG parameter, and sure enough, RPC latency had shot through the roof about 10-fold. It was also much greater than the standalone client I was using.

I had previously done some research into Microsoft's TCP Chimney Offload from the Scalable Networking Pack and Broadcom's TCP Offload Engine. Could the NIC actually be differentiating between TCP and ICMP traffic thus yeilding conflicting results.

I knew that TCP Chimney Offload (and consequently TOE) could be disabled on-the-fly with one NetSh command:

NetSh Int IP set Chimney Disable

Surely enough within a second, the TCP connections were brought back in host, and all symptoms went away.

The command for determining the offload status for a TCP connection is:

Netstat -t

I'm not 100% sure what broke in v3.7.19 of the driver. The driver just may have been build on a version of NDIS that finally supported TCP Chimney completely, and that TCP offload engine was actually never working previously in our environment. Unfortunately we never got a satisfactory answer as to what went wrong in this driver. If that was indeed expected behaviour for TOE, then there is definitely some tweaking required so that latency-sensitive protocols (RPC in this case) are excluded from being offloaded to the NIC. Some preliminary testing of v4.0.23b of the same driver suggests that TCP latency does increase significantly in an offload state, however Outlook doesn't appear to exhibit the same symptoms. Stay tuned for my investigations into Interrupt Coalescing in the 3.7.19 driver...