How We Cut p99 by 85% With Adaptive Batching
A fixed batch size is a guess about your traffic. We replaced ours with a feedback loop and the tail collapsed.
The default in most vector engines is a fixed batch: collect 64 queries, walk the graph, return. It is simple, predictable, and wrong at every traffic level except the one it was tuned for.
“A constant in your query path is a guess about the future. The future does not care.”
The diagnosis
Our p99 was sawtoothing between 80 ms and 500 ms on a customer with spiky traffic. The cause was trivial in hindsight.
- Under low traffic, a batch never filled, so a query waited out the full timer before anything happened.
- Under high traffic, the batch overflowed and queries queued behind the graph walk.
- In the transition, both pathologies fought each other, which is where the sawtooth came from.
A fixed batch is two bad regimes glued together.
The fix
Batch size is now a function of the rolling five-second query rate, with three modes.
- Cold (under 50 queries a second): execute immediately, no batching. A single query should not pay for a crowd that is not there.
- Warm (50 to 2,000): interpolate between time-based and size-based dispatch. The mode itself is the gradient.
- Hot (2,000 and above): fill to 256 queries before dispatch. At that rate the shared graph traversal genuinely amortises.
We also added a relief valve. If queue depth crosses 80 percent we dispatch immediately at whatever size we have. An undersized batch beats a spiked tail.
The graph
- p99: 240 ms to 36 ms
- p99.9: 1.4 s to 110 ms
- mean: barely moved, within the margin of error
That last line is the right shape. We did not get faster. We got less variable. Means are noisy; tails tell the truth, and the tail is what your user is waiting on.
