Armen Ghazaryan

The leak runs in a JVM of its own

The memory-leak experiment runs its workload in a second JVM inside the API's container, one at a time: a leak meant to end in OutOfMemoryError cannot run in the server's own heap.

Status
Accepted 3 October 2026
The record
0011-a-jvm-of-its-own-for-the-leak.md

Context

The lab's memory-leak experiment is a model in the browser. To show that the same leak ends the same way on real collectors, the workload has to run until it exhausts a heap, under Serial, G1 and ZGC in turn. The server runs one JVM, with one collector, whose heap serves every visitor: exhausting it would take the site down, and its collector cannot be switched per request.

Decision

LeakLab starts a child JVM with ProcessBuilder: the server's own java, the workload's own code source as the class path, -Xms64m -Xmx64m, the collector the visitor picks, one CPU, -XX:+ExitOnOutOfMemoryError and -Xlog:gc on standard output, which the server reads line by line and streams as events. Nothing in the command comes from the request but two enums, and JAVA_TOOL_OPTIONS and its kin are removed from the child's environment. The run holds the server's benchmark gate (ADR 0006), so only one child exists at a time; it is killed when the client goes away, when it outlives its 35-second deadline, and when the server stops.

Rejected: running the workload in the server's JVM (it would take the server with it), a Cloud Run job per run (seconds of cold start and a second deployable for a 20-second experiment), and the model alone (it shows the trend, not a real collector's behaviour).

Consequences

  • Memory: the server and a child peaked at 321 MiB of the container's 512 on the production image, after the other experiments had run; the child itself took about 100 MiB.
  • CPU: the child takes the one vCPU for up to 20 seconds; the gate keeps the other benchmarks out meanwhile, and the rest of the API answers more slowly.
  • The API service runs on Cloud Run's second-generation execution environment, set in Terraform (infra/gcp/terraform/run.tf, ADR 0022): a full Linux kernel, so process creation, cgroup limits and ZGC's memory mapping behave as in the container that was measured.
  • Where the server runs from a jar inside a jar, the workload cannot be put on a child's class path, and the endpoint answers 503 LEAK_UNAVAILABLE; the page then offers the recorded runs.