What Changes Between 100 Million Vectors and 12 Billion

At a hundred million vectors a single well-tuned HNSW graph on a large machine will serve you, and the honest advice is to stop reading and go build something else. The problems start when the collection outgrows one machine’s memory, because every answer after that is about placement.

What stays the same

The query. A nearest-neighbour search is still a graph walk with a candidate list, and the client code does not change. Teams expect the API to get more complicated at scale. It does not. The operations underneath it do.

What breaks first

Memory, and not in the way people plan for.

• Graph degree, not vector count. A 768-dimension vector is 3KB. The HNSW neighbour lists around it are often larger. Teams size RAM off the raw embeddings and are surprised by a factor of two.
• The rebuild, not the steady state. A shard being rebuilt needs headroom for the old graph and the new one at once. Size for the rebuild or accept that you cannot run one during business hours.
• The tail shard. Traffic is never uniform. One shard always holds the popular region, the recent documents, the English-language content. Hashing by id spreads bytes evenly and latency unevenly.

How we place shards

We target eighty million vectors per shard and two replicas, then let the planner move shards between nodes based on measured query load rather than size. A shard that answers six times the median gets its own node. This is unglamorous and it is most of what the platform does.

Above two billion vectors we switch the collection to DiskANN. The recall cost is real, about two points at the same latency budget, and it is still cheaper than the RAM. We say so on the plan page rather than in a footnote.

The number that actually matters

Not vectors indexed. Queries per second per gigabyte of RAM. It is the only figure that tells you whether the next billion vectors is a purchase order or a re-architecture. Ours is 41 today. Two years ago it was 9.