Ten thousand threads, twenty connections
Virtual threads made threads cheap; connections are not. Requests pour into a small PostgreSQL pool under three policies, and the ceiling stays where Little’s law puts it.
What you’re seeing
A pool of 10 connections, each held for 50 ms, can serve at most 200 requests a second: Little’s law, with ten requests in the pool and a twentieth of a second each. In the recorded runs 500 arrive every second, and the throughput is about 197 a second whatever the policy. What the policy decides is who pays for the other 300.
Unbounded, every request queues inside the pool, and 1,315 of the 2,500 time out after waiting a full second; the ones served waited almost as long (p99 1,050 ms). Throttled, nobody fails, but the queue grows for five seconds: the median request takes 3.8 s and the slowest 7.6 s. Failing fast turns 1,488 away within 100 ms and serves the rest in 150 ms.
Where a request’s time went
After a live run the page draws about thirty of the requests as the server traced them, evenly spread over the run: the wait at the gate, the wait in the pool, the query. Throttled, the wait at the gate grows request by request while the query stays at the time set. Unbounded, the waiting moves into the pool: a request that waits there a full second leaves with a timeout, and the ones served had waited almost as long.
Why it matters now
Virtual threads made threads cheap, so a server no longer runs out of them before anything else. The next limit is the pool. InfoQ’s review of virtual threads in production found exactly this: moving to them lifted throughput from 624 to 1,539 requests a second, and the error rate stayed at 37%, because both runs hit the connection pool’s ceiling.
The policy is the design decision. Throttling, as Spring’s @ConcurrencyLimit does, hides the overload as latency; failing fast makes it visible at once and keeps the requests it serves fast; leaving it unbounded combines the worst of both.
Where it meets my work
Tuning p99 latency and throughput is part of my work at Picsart, on services that run on Kubernetes. A pool is one of the first places where that latency queues: once threads are cheap, the wait for a connection is what sets the p99, which is why a pool is sized against what the database allows rather than left at its default.
A real pool on real PostgreSQL, with a stand-in for work: each request holds its connection in pg_sleep for the time set, and does nothing else. Each run builds its own HikariCP pool and fills it before the first request, so opening connections is not part of the measurement. The recordings ran PostgreSQL in a container beside the API; a managed database adds network time to every query.
The code behind it
The files that do the work, by their place in this site’s repository.
- api/src/main/java/com/aghazaryan/api/lab/pool/
PoolRun.javaBuilds and fills a pool, then sends requests on schedule, each on a virtual threadPolicy.javaThe three policies, each a gate in front of the poolGate.javaNo gate, a fair semaphore, or the same with a 100 ms limitTimedDataSource.javaA decorator that times every wait for a connection
- api/src/main/java/com/aghazaryan/api/database/
Database.javaThe port that gives the experiment its database, or a 503
- api/src/main/java/com/aghazaryan/api/tracing/
TraceStore.javaKeeps the run’s trace for the waterfall: a span processor, bounded, nothing exported