How llm-d's Endpoint Picker knows which replica holds your prefix

Sat, 26 Sep 2026 00:00:00 +0000

The first thing that struck me comparing typical HTTP request routing with routing for LLM inference was the importance of KV-cache-aware routing. A model has to build a KV cache entry for every token of the prompt before emitting the first token. On any prompt with non-trivial token count, prefill will dominate time-to-first-token.

In a typical conversation, each request builds on the previous request; the new prompt starts with the old one. If we can go back to the same replica that served the previous request, we can re-use the existing KV cache blocks and generate only the new ones. If we end up on a new replica, then the whole KV cache has to be generated again.

Llm-Inference on Nawinder Pawar

How llm-d's Endpoint Picker knows which replica holds your prefix