I'm currently studying the effects of various settings on a file servers performance. One of the TCP/IP registry options that is documented (scarcely), is the MaxFreeTcbs setting. Supposedly this setting is used to decide how many Transmission Control Blocks (TCB) to pre-allocate from non-paged pool. In theory this also has the effect of limiting the amount of concurrent TCP connections the server can sustain.
I loaded up poolmon, and noticed that each time a TCP connection was negotiated, a 456-byte allocation was made from the non-paged pool using the pooltag TCPT. There appeared to be a few TCBs pre-allocated, but only about 5. After that, they were allocated when the TCP connection was created.
What I noticed on a Windows Server 2003 R2 server is that this MaxFreeTcbs setting now appears to have no affect. I set the value to 10, and still able to create 500 simultaneous connections with a tool called PCATTCP. Each of them matched up to an allocation on the TCPT pool tag.
Interesting, nothing mentioned anywhere about this. Also, many of the TCP performance guides still make reference to this key.
Has anyone else run into this? Is this a change to the way TCP/IP works in the Scalable Networking Pack?
-M
Saturday, July 12, 2008
Saturday, May 3, 2008
Broadcom Driver Woes
Shortly after deploying the Broadcom NetXtreme II v3.7.19 driver for the 5708 chipset to our Metaframe environment, the users were not happy campers. There ended up being two severe problems that we noticed in this particular revision of the drivers. Today I'm going to describe problem number one, the TCP Offload Engine feature (TOE).
Now we actually ran into this issue 3-4 months ago, so I don't remember what version of the driver we had before 3.7.19.
Once updated, everything appeared fine. Ping tests showed connectivity and expected latency. A traffic simulation tool PCATTCP showed that throughput was being handled as expected; the servers were stable.
The next day when users connected to theses Metaframe servers, they started complaining almost instantly of an obscure symptom, (they always are) slow response to operations in Outlook. For the record, we used Outlook 2003 clients connected to Exchange 2003. Cached Exchange Mode was off, and the driver was only loaded onto the client end.
Now I was baffled to see that the preview pane in Outlook was taking about 5 seconds to update per email, instead of perhaps 200ms. My previous test demonstrated that the NIC driver had achieved far greater throughput than what Outlook was demanding, and that ICMP latency had not increased at all. Surely the NIC doesn't know the difference between ICMP traffic and Outlook traffic! That would defy everything I know of the Networking models.
We ran up Outlook with the /RPCDIAG parameter, and sure enough, RPC latency had shot through the roof about 10-fold. It was also much greater than the standalone client I was using.
I had previously done some research into Microsoft's TCP Chimney Offload from the Scalable Networking Pack and Broadcom's TCP Offload Engine. Could the NIC actually be differentiating between TCP and ICMP traffic thus yeilding conflicting results.
I knew that TCP Chimney Offload (and consequently TOE) could be disabled on-the-fly with one NetSh command:
NetSh Int IP set Chimney Disable
Surely enough within a second, the TCP connections were brought back in host, and all symptoms went away.
The command for determining the offload status for a TCP connection is:
Netstat -t
I'm not 100% sure what broke in v3.7.19 of the driver. The driver just may have been build on a version of NDIS that finally supported TCP Chimney completely, and that TCP offload engine was actually never working previously in our environment. Unfortunately we never got a satisfactory answer as to what went wrong in this driver. If that was indeed expected behaviour for TOE, then there is definitely some tweaking required so that latency-sensitive protocols (RPC in this case) are excluded from being offloaded to the NIC. Some preliminary testing of v4.0.23b of the same driver suggests that TCP latency does increase significantly in an offload state, however Outlook doesn't appear to exhibit the same symptoms. Stay tuned for my investigations into Interrupt Coalescing in the 3.7.19 driver...
Now we actually ran into this issue 3-4 months ago, so I don't remember what version of the driver we had before 3.7.19.
Once updated, everything appeared fine. Ping tests showed connectivity and expected latency. A traffic simulation tool PCATTCP showed that throughput was being handled as expected; the servers were stable.
The next day when users connected to theses Metaframe servers, they started complaining almost instantly of an obscure symptom, (they always are) slow response to operations in Outlook. For the record, we used Outlook 2003 clients connected to Exchange 2003. Cached Exchange Mode was off, and the driver was only loaded onto the client end.
Now I was baffled to see that the preview pane in Outlook was taking about 5 seconds to update per email, instead of perhaps 200ms. My previous test demonstrated that the NIC driver had achieved far greater throughput than what Outlook was demanding, and that ICMP latency had not increased at all. Surely the NIC doesn't know the difference between ICMP traffic and Outlook traffic! That would defy everything I know of the Networking models.
We ran up Outlook with the /RPCDIAG parameter, and sure enough, RPC latency had shot through the roof about 10-fold. It was also much greater than the standalone client I was using.
I had previously done some research into Microsoft's TCP Chimney Offload from the Scalable Networking Pack and Broadcom's TCP Offload Engine. Could the NIC actually be differentiating between TCP and ICMP traffic thus yeilding conflicting results.
I knew that TCP Chimney Offload (and consequently TOE) could be disabled on-the-fly with one NetSh command:
NetSh Int IP set Chimney Disable
Surely enough within a second, the TCP connections were brought back in host, and all symptoms went away.
The command for determining the offload status for a TCP connection is:
Netstat -t
I'm not 100% sure what broke in v3.7.19 of the driver. The driver just may have been build on a version of NDIS that finally supported TCP Chimney completely, and that TCP offload engine was actually never working previously in our environment. Unfortunately we never got a satisfactory answer as to what went wrong in this driver. If that was indeed expected behaviour for TOE, then there is definitely some tweaking required so that latency-sensitive protocols (RPC in this case) are excluded from being offloaded to the NIC. Some preliminary testing of v4.0.23b of the same driver suggests that TCP latency does increase significantly in an offload state, however Outlook doesn't appear to exhibit the same symptoms. Stay tuned for my investigations into Interrupt Coalescing in the 3.7.19 driver...
Subscribe to:
Posts (Atom)