Ten thousand blocking calls

A fixed pool can only have as many tasks in flight as it has threads; everything else waits in the queue, so the orange line climbs one wave of tasks per blocking interval. A virtual thread that blocks is unmounted from its carrier, so a handful of OS threads keep thousands of blocked tasks in flight at once.

The gain is not free speed: each task still takes as long as its I/O. What disappears is the queueing in front of the pool.

CPU-bound work is limited by cores, not threads: both executors finish together.

Pinning. Until JDK 24, a virtual thread that blocked inside a synchronized block could not leave its carrier, so throughput collapsed to one task per carrier. JEP 491 removed that. The synchronized preset runs it on this server’s Java 25 and sets it beside a Java 21 recording of the same tasks: 1,000 done in about a tenth of a second, against 199 in ten.

What still pins is a native frame on the stack. In the native frame preset the C library’s qsort, called through the Foreign Function & Memory API, calls back into a Java comparator that blocks, as a native driver calling back into Java would. Throughput collapses again, and Flight Recorder reports each blocked task as a jdk.VirtualThreadPinned event: the page counts them live, and in production they are the events to alert on.

With virtual threads the bottleneck moves downstream: the database connection pool, a rate-limited partner API. Concurrency limits belong there — a semaphore, a sized pool — rather than in the thread count.

SimpleDateFormat keeps its working state in fields, so threads must not share one. The classic fix is a ThreadLocal: one formatter per thread, reused by every task that thread runs. In figure 2, on a pool of 200 threads, 10,000 tasks build 200 formatters.

A virtual thread runs one task and ends, so the same code builds a formatter for every task: 10,000 of them, and about 2.8 KB of heap per task in the recorded run, slightly more than building a new one on every call (2.6 KB), because each virtual thread also gets its own thread-local map. Nothing fails and nothing leaks: the cache just stops caching, which no error or log line will say; a profiler will.

The fix depends on what the thread-local held. An object that is safe to share should be shared: DateTimeFormatter is immutable, and one serves every thread. One that is not goes into an explicit, bounded pool. Request context, such as the user or a trace id, belongs in a ScopedValue (final in Java 25), bound for the duration of a call rather than for the life of a thread.

Virtual threads are among the Java 21 features on my CV, and Java 21 is what my services at Picsart run on. Cheap threads move the bottleneck downstream, to a pool or a database limit: the connection-pool experiment shows exactly that.

The API behind this page runs on virtual threads, including the stream you are watching.