Six drivers crashed the API: when a streaming endpoint pins a worker per connection
A live demo died with six active users, and the cause wasn't slow code, it was arithmetic: long-lived streams each ate a worker until the pool ran dry.
Six drivers logged in and the entire backend fell over. Not slow, not flaky: dead, throwing timeouts to everyone, including the people just trying to load a list. Six users. The server had handled load tests with far more traffic than that, so my first instinct was that something exotic had broken. It hadn’t. The math had simply run out, and six was the number where it noticed.
A stream is not a request
Most endpoints are polite. A request comes in, the worker does its job in a few milliseconds, and the worker goes back in the pool for the next caller. The whole model assumes the worker is yours for an instant and then it isn’t.
A location stream breaks that assumption completely. To push a driver’s position to the dispatcher in something close to real time, I’d opened a long-lived connection that stayed open for the entire duration of the trip. Minutes. Sometimes the better part of an hour. And for every second of that, the connection held onto its worker process and would not let go.
So the pool wasn’t being borrowed and returned. It was being consumed and kept.
- The pool had twenty workers. That sounds like room for plenty of users when each one needs a worker for fifteen milliseconds.
- Each active stream took one worker and pinned it. Not for fifteen milliseconds. For the whole trip.
- A handful of drivers on the road meant a handful of workers gone, never returning to serve anyone else.
By six concurrent streams I’d already burned a third of the capacity on connections that were doing almost nothing but sitting there. The rest of the pool filled with the next few users, and then there was nothing left to answer the customer trying to refresh their order. The API wasn’t crashing. It was full.
The lock that turned a third into all of it
There was a second trap underneath, and it’s the one that turned a degraded service into a total outage. Each stream ran inside a per-request session lock, the kind of thing that quietly serializes a single user’s calls so they don’t stomp on each other. Reasonable in isolation. Lethal here.
Because the stream sat inside that lock and never exited, every other request from that same driver queued up behind a connection that was never going to finish. The driver’s app would try to fetch their next job, and that fetch would block, waiting for a lock held by a stream that had no end in sight. So it wasn’t just twenty workers draining. It was each streaming user also jamming their own other requests behind the one call that never returned.
Capacity isn’t workers, it’s workers divided by how long each one is held. A connection measured in minutes makes a twenty-worker pool a six-user pool.
The fix was arithmetic, then architecture
I didn’t make anything faster. There was nothing slow to speed up. The fix was admitting what the numbers actually said.
First, the arithmetic. Your real ceiling is the worker count divided by the number of long-lived connections you expect at once, and once I wrote that down the failure stopped being mysterious and started being obvious. Twenty over a per-trip stream is a ceiling you can hit on a slow afternoon.
Second, the architecture. Long-lived connections do not belong on the same pool that serves quick requests. Move them off, onto something built to hold many idle connections cheaply, so a parked stream costs you a socket and not a whole worker. And take the stream out of the per-request lock so a user’s other calls aren’t hostages to it.
But the honest answer, the one I reached for last and should have reached for first, is that I didn’t need streaming at all. The dispatcher did not require sub-second location updates. A five-second poll of a snapshot endpoint would have shown the same dots moving across the same map, served by the same polite, instantly-returned workers as everything else, and six drivers would have been a rounding error instead of an outage. I’d reached for real time because it sounded right, and paid for it in the one currency the pool actually had: workers held too long.