TIME_WAIT in netstat

So, ever wonder what all those TIME_WAITs are doing in your netstat listing?

Okay, for those of you who don't spend all your waking hours fooling around with Web servers, let me back up a little and explain what that sentence meant.

Netstat is a little utility that many administrators use to monitor the network connections on their servers. It is quite useful for tracking down that small subset of performance bottlenecks that aren't attributable to yet another piece of convoluted application code that some careless programmer wrote and now you have to take care of. But I digress.

When you run netstat on your busy IIS box, you might get something that looks like this:

C:\>netstat -np tcp

Active Connections

Proto Local Address Foreign Address State

TCP 192.168.0.1:80 192.168.0.12:1217 ESTABLISHED
TCP 192.168.0.1:80 192.168.0.5:1218 TIME_WAIT
TCP 192.168.0.1:80 192.168.0.234:1252 TIME_WAIT
TCP 192.168.0.1:80 192.168.0.37:1267 ESTABLISHED
TCP 192.168.0.1:80 192.168.0.23:1298 TIME_WAIT
TCP 192.168.0.1:80 192.168.0.32:1345 TIME_WAIT

And so on and on, for many, many lines. Each line here represents a connection between a TCP socket your server and a matching one on some other machine--usually an HTTP client such as a browser or proxy server, but depending on your architecture you might also see connections to other kinds of servers (database, application, directory, etc.). Each connection has a unique combination of IP addresses and port numbers that identify the endpoints to which the sockets are bound. More to the point, each one also has a state indicator. As connections are set up used and torn down, they pass through a variety of these states, most of which aren't shown here, because they come and go quite quickly).

The connections in the ESTABLISHED state are, well, established--they are neither being set up nor torn down but just used. This is what you will often see the most of. But what about the others? On a busy HTTP server, the number of sockets in this TIME_WAIT state can far exceed those in the ESTABLISHED state. For instance, I checked an IIS 6.0 box that serves a fairly busy corporate site earlier today and got 124 ESTABLISHED connections versus 431 in TIME_WAIT.

What does this all mean? More importantly, is it something you should be worried about?

The answers are:

1. It's complicated.

2. Maybe.

To understand what all those TIME_WAITs are doing there, it's useful to review (or learn) a little TCP. I'll wait here while you brush up on RFC793.

That was fast. Just kidding. The bit you need to know is so simple, even I can explain it.

As you know, TCP provides a reliable connection between two endpoints, across which data can be sent in segmented form. As part of this, TCP also provides a mechanism for gracefully shutting down such connections. This is accomplished with a full duplex handshake, which can be diagrammed like so:

Server Client

-------------- FIN -------------->

<------------- ACK ---------------

<------------- FIN ---------------

-------------- ACK ------------->

As you can see by this very sophisticated diagram, a graceful shutdown requires the two endpoints to exchange some TCP/IP packets with the FIN and ACK bits set, in a certain sequence. This exchange of packets in turn corresponds to certain state changes on each side of the connection. In the diagram, I've labeled the two sides "Server" and "Client" such that the sequence of events mirrors what usually happens when connections are closed by HTTP.

Here is what happens, step-by-step:

1. First the application at one endpoint--in this example, that would be the Web server--initiates what is called an "active close." The Web server itself is now done with the connection, but the TCP implementation that supplied the socket it was using still has some work to do. It sends a FIN to the other endpoint and goes into a state called FIN_WAIT_1.

2. Next the TCP endpoint on the browser's side of the connection acknowledges the server's FIN by sending back an ACK, and goes into a state called CLOSE_WAIT. When the server side receives this ACK, it switches to a state called FIN_WAIT_2. The connection is now half-closed.

3. At this point, the socket on the client side is in a "passive close," meaning it waits for the application that was using it (the browser) to close. When this happens, the client sends its own FIN to the server, and deallocates the socket on the client side. It's done.

4. When the server gets that last FIN, it of course sends back on ACK to acknowledge it, and then goes into the infamous TIME_WAIT state. For how long? Ah, there's the rub.

The socket that initiated the close is supposed to stay in this state for twice the Maximum Segment Lifetime--2MLS in geek speak. The MLS is supposed to be the length of time a TCP segment can stay alive in the network. So, 2MLS makes sure that any segments still out there when the close starts have time to arrive and be discarded. Why bother with this, you ask?

Because of delayed duplicates, that's why. Given the nature of TCP/IP, it's possible that, after an active close has commenced, there are still duplicate packets running around, trying desperately to make their way to their destination sockets. If a new socket binds to the same IP/port combination before these old packets have had time to get flushed out of the network, old and new data could become intermixed. Imagine the havoc this could cause around the office: "You got JavaScript in my JPEG!"

So, TIME_WAIT was invented to keep new connections from being haunted by the ghosts of connections past. That seems like a good thing. So what's the problem?

The problem is that 2MLS happens to be a rather long time--240 seconds, by default. There are several costs associated with this. The state for each socket is maintained in a data structure called a TCP Control Block (TCB). When IP packets come in they have to be associated with the right TCB and the more TCBs there are, the longer that search takes. Modern implementations of TCP combat this by using a hash table instead of a linear search. Also, since each TIME_WAIT ties up an IP/port combination, too many of them can lead to exhaustion of the default number of ephemeral ports available for handling new requests. And even if the TCB search is relatively fast, and even if there are plenty of ports to bind to, the extra TCBs still take up memory on the server side. In short, the need to limit the costs of TIME_WAIT turns out to be a long-standing problem. In fact, this was part of the original case for persistent connections in HTTP 1.1.

The good news is that you can address this problem by shortening the TIME_WAIT interval. This article by Brett Hill explains how to do so for IIS. As Brett explains, four minutes is probably longer than needed for duplicate packets to flush out of the network, given that modern network latencies tend to be much shorter than that. The bad news is that, while shortening the interval is quite common, it still entails risks. As Faber, Touch and Yue (who are the real experts on this) explain: "The size of the MSL to maintain a given memory usage level is inversely proportional to the connection rate." In other words, the more you find yourself needing to reduce the length of TIME_WAIT, the more likely doing so will cause problems.