macOS 26 shipped with TIME_WAIT sockets that never expire

This bug was identified in macOS 26.0 and was patched somewhere between then and 26.5.2. This is simply a documentation of my experience with it. If you are still running an early build of macOS 26 and you find your machine randomly hanging for minutes at a time when connecting to localhost, then this documentation is for you (update your OS).

In mid July I was working on creating a training corpus using a local inference server on my Mac Studio. I had three worker threads using a 120B model to stream millions of tokens into jsonl shards. This is a slow process, but I noticed every 20 minutes or so the entire system would hang. No errors, no temp issues, no log lines, and then three or four minutes later it would pick back up and chug away happily like nothing ever happened.

This was more annoyance than anything destructive, but I got curious about what was happening and started poking around.

The server

The obvious suspect was the inference server (oMLX) running on localhost. oMLX is typically very solid, but I have a few commits in it and my copy tends to a slightly customized branch so it seemed obvious that any problem would likely originate there. Long sustained runs from big (for me at least) models have lots of places to stall: memory pressure, cache spills, wedged schedulers, thermal throttling, etc. But when I started poking into logs during the stalls I found none of this. The event loop was idle and the scheduler queue was empty. CPU usage was basically 0, no model load or unload events, and the smoking gun - all three workers stalled at the same time and then all cleared together.

Time to start probing. I waited for a stall and then started hitting localhost. Some went through and some stalled even though it was the same endpoint at the same time. Interesting. Take a look:

23:02:19 probe[get-models]      code=200 took=0s
23:02:19 probe[get-admin-stats] code=200 took=0s
23:02:29 probe[post-chat]       code=000 took=10s   <- curl gave up at its own timeout
23:02:29 probe[post-login]      code=200 took=0s
23:02:39 probe[get-chat-noauth] code=000 took=10s

If this was a server issue then you'd expect ALL the connections to have hung. Instead, some were getting replies instantly while others languished - something was discriminating on a per connection basis preventing some requests from ever even reaching the server. All these hung connections were sitting in SYN_SENT over loopback which meant SYN was being sent, but nothing was replying. Why was it inconsistent?

The stuck

Another clue: all the connections that closed left their four-tuple in TIME_WAIT, kept for 2xMSL (30 seconds on macOS as net.inet.tcp.msl is 15000). On a healthy machine, TIME_WAIT should ebb and flow as connections churn, but this machine was not healthy. The TIME_WAIT count kept increasing and by late evening there were nearly 18,000 stuck TIME_WAIT connections, most of them pointed at my inference server port. The oldest were hours old, well past their expiry - the kernel was failing to reap these connections.

The repro, more or less:

import socket
for _ in range(10):
    s = socket.create_connection(("127.0.0.1", 9999))  # any listening port
    s.close()
# netstat -an | grep TIME_WAIT
# waited 30 seconds... waited an hour... still there

Ten connections create ten TIME_WAIT entries that just get stuck and accumulate. Thirty seconds later (when they should be reaped), they were still there. An hour later, still there.

The failure chain

So we had an answer. The workers were slowly drowning the studio's address space, filling it with more and more TIME_WAIT entries that never expired. The processes were using urllib which opens a new TCP connection for every request (not the most efficient system, granted) and these workers were making thousands of requests an hour. Every single close blocked that tuple until a full machine restart and of course there are only so many ephemeral ports.

Eventually a worker gets handed a port it had already used while churning through connections over the last few hours. The kernel then sees the SYN, notes that that tuple was still in TIME_WAIT, and then drops the connection. On a working kernel, those TIME_WAIT holds would be reaped, but something was broken and the client just kept retransmitting SYNs at nothing until the connection timeout trigger forced it to finally give up.

These stalls tended to be 3.5-4.9 minutes long. If you add the default keepinit retry ladder, well no surprise that that's roughly the number you get. The bad ports were essentially random luck which is why we saw some connections get through while others would just hang.

The workaround

The solution, as stupid as it sounds, was just to stop opening connections. I rewrote the workers' HTTP layer to just use one connection per thread and to keep it alive (plus a second silent retry if needed since uvicorn hangs on idle keep-alive after five seconds or so). Now that connection churn was essentially eliminated, the rest of the corpus run ran without any stalls or issues through the night. This, however, did not clear the pile of dead tuples - only a complete reboot did that.

It's unsurprising that this wasn't a widely reported issue - who else is stupid (or lazy) enough to open and close thousands of loopback connections an hour? It's certainly not the behavior of a well written client. I just happened to be running exactly such a hastily hacked script so as to encounter this bug.

The patch

On July 27th, I went to rerun the repro to write up a bug report, but my machine had updated to 26.5.2 the day before, and lo and behold the bug was fixed. I'm not sure when it happened (since I was behind in updates) and I couldn't find any notes or mentions of TIME_WAIT or TCP related to this, so I semi regret not filing Apple Feedback to immortalize the bug in some database, but I'm glad it was fixed. If you know which build fixed the issue (or if you are still able to reproduce it), I'd love to hear about it!