NVIDIA Dynamo for distributed LLM inference

An inference engine executes and schedules model work. Depending on the engine, one replica may use one GPU, several GPUs, or several nodes. NVIDIA Dynamo is a distributed inference framework that coordinates engines and serving components across a deployment: routing requests, constructing aggregated or disaggregated topologies, applying scaling policies, and integrating cache, transfer, deployment, and cluster services.

The boundary is responsibility, not a strict node count. A backend can span nodes without Dynamo; Dynamo can also coordinate a smaller topology. The architecture question is which distributed capability improves a measured workload objective enough to justify its operational cost.

What you will understand by the end

• How engine execution differs from distributed serving coordination.
• Three Dynamo capabilities covered here: KV-aware routing, prefill/decode disaggregation, and orchestration of distributed backends.
• How Planner modes use traffic, queues, cache state, performance models, TTFT, and ITL.
• Three distinct reasons to distribute: model fit, aggregate throughput, and phase specialization.
• How to test whether routing, disaggregation, or LLM-aware scaling earns its complexity.

Engine execution versus distributed coordination

Distributed serving framework (Dynamo)
  request graph · routing · planning · deployment · cache/transfer services · operations
                         │
          ┌──────────────┴──────────────┐
          ▼                             ▼
  backend engine replica        backend engine replica
  scheduling · batching         scheduling · batching
  KV management · kernels       KV management · kernels
  may span GPUs or nodes        may span GPUs or nodes

Backend engines implement tensor, pipeline, data, context, or expert parallel execution. For example, vLLM and TensorRT-LLM support distributed model execution. Dynamo can deploy and coordinate those workers, apply topology-aware placement, route to the resulting replicas, and connect them into a wider scaling or disaggregation graph. It does not replace the backend's collective communication or parallel execution algorithms.

Three distributed-serving capabilities covered here

This lesson focuses on three useful capabilities, not the complete product:

1. KV-aware routing. The backend retains reusable KV blocks. Dynamo's router consumes or approximates cache-state information and selects a worker using estimated prefix overlap and current load. A route produces a hit only if the tokenized prefix matches, the relevant blocks still exist, and the backend supports reuse.
2. Prefill/decode disaggregation. Separate worker pools process prompts and generate output tokens. KV state crosses the phase boundary through NIXL or a backend connector over the available GPU, RDMA, or network fabric.
3. Distributed-backend orchestration. Dynamo deploys, places, connects, routes to, and scales backend workers or replicas that may themselves span GPUs or nodes.

Dynamo's current platform also includes additional planner modes, cache management and offload, loading and deployment, topology, resilience, observability, profiling, and other operations capabilities. Backend support and maturity vary by release; verify the current feature matrix before designing around one.

KV-aware routing is cache affinity plus load

Prefix reuse can save material prefill work when long tokenized prefixes repeat and remain resident. Cache affinity alone can create hot replicas, so Dynamo's KV routing cost considers overlap and worker load. Its view can come from backend cache events or, in an approximate mode, predictions based on routing decisions with expiry.

The operational signals belong together: cache hit or overlap rate, cache-event freshness, eviction, per-worker queue and load, routing skew, and request latency. High theoretical overlap does not help if blocks were evicted or stale state routes traffic to the wrong worker.

Disaggregation separates phases and adds a transfer seam

In many common dense-model regimes, prefill is relatively compute-intensive, while low-batch autoregressive decode is relatively memory-bandwidth-intensive. Treat this as a starting hypothesis, not a diagnosis. Model architecture, sequence lengths, batching, precision, speculative decoding, expert or tensor parallelism, communication, attention kernels, and hardware can change the limiting resource.

request → prefill router → PREFILL workers
                              │
                              │ KV transfer through NIXL / backend connector
                              │ over the available GPU, RDMA, or network fabric
                              ▼
                         DECODE workers → streamed output

Independent pools can isolate long prefills from ongoing decode and adjust phase capacity separately. They also introduce transfer latency and bandwidth, topology constraints, additional queues, and another failure boundary. Measure the actual connector and fabric; do not label transfer free or infer its performance from “NVLink” or “InfiniBand” alone.

Planner: TTFT, ITL, traffic shape, and scaling modes

Time to first token (TTFT) and inter-token latency (ITL) are user-facing latency metrics. TTFT is not prefill compute alone: routing, queueing, scheduling, prefix reuse, and KV transfer can contribute. ITL is not interchangeable with tokens per second: throughput is a capacity metric, while ITL describes a user's token spacing.

traffic shape + queue/cache signals + engine performance model
                           │
                           ▼
               Planner optimization policy
            throughput · latency · load · SLA
                           │
                TTFT / ITL targets in SLA mode
                           │
               ┌───────────┴───────────┐
               ▼                       ▼
        prefill replicas          decode replicas

Current Dynamo documentation describes four optimization targets:

• throughput: queue-depth and KV-utilization thresholds;
• latency: earlier, more aggressive scaling to keep queues short;
• load: user-defined prefill queue-token and decode KV-utilization thresholds;
• sla: engine performance estimates and traffic signals targeting configured TTFT and ITL.

Predictive throughput-based and reactive load-based control loops can be used separately or together depending on mode and configuration. Name the policy when describing behavior; there is no single fixed rule that always scales prefill on TTFT and decode on tokens/second.

Three different reasons to distribute

Do not collapse every distributed design into “the model is large”:

Reason	Problem	Possible mechanism	Evidence required
Capacity distribution	Model, active weights, or KV state does not fit efficiently	Backend tensor/pipeline/expert parallelism; cache offload	Per-device memory, communication, latency, model fit
Throughput distribution	One replica cannot serve offered load within SLO	More replicas, routing, autoscaling	Throughput-at-SLO, queue growth, scaling delay, cost
Phase specialization	Prefill and decode demand or interference diverges	Disaggregated pools and KV transfer	TTFT/ITL, phase utilization, transfer time/fabric load

Parameter count alone cannot choose among them. Precision, quantization, active MoE parameters, GPU memory, KV demand, topology, kernels, and workload shape determine the efficient footprint. Likewise, high request volume does not guarantee cache-aware routing or disaggregation will help.

A benchmark-driven adoption ladder

Distributed components are independently adoptable rather than one all-or-nothing gate:

1. Start with an engine on the smallest topology that fits and meets the SLO.
2. Use engine-native multi-GPU or multi-node parallelism when model fit or execution demands it.
3. Add replicas and ordinary routing when aggregate capacity is the problem.
4. Test KV-aware routing when repeated prefixes and retained blocks can save material work.
5. Test prefill/decode disaggregation when phase imbalance or interference is measured and the KV-transfer path can sustain it.
6. Evaluate LLM-aware planning when fixed capacity or conventional scaling cannot meet the workload's latency/cost behavior.
7. Add other deployment, cache, or resilience services only for a demonstrated need.

Use a controlled evaluation:

1. Define input/output-length distributions, concurrency, arrival bursts, and tokenized prefix overlap.
2. Define p50/p95/p99 TTFT, ITL, end-to-end latency, throughput-at-SLO, and cost objectives.
3. Benchmark the simplest aggregated engine topology.
4. Test replica scaling and routing; include startup and scaling lag.
5. Test KV routing only when prefixes repeat; measure realized hits, eviction, and skew.
6. Test disaggregation while measuring KV-transfer time, topology, and fabric utilization.
7. Compare GPU-hours, tail latency, goodput, failure behavior, and operational complexity.

Failure modes and observability

• Cache-state lag or event loss: router estimates diverge from backend state.
• Affinity hot spots: reusable prefixes overwhelm one worker despite spare fleet capacity.
• Cache eviction: expected hits disappear before the follow-up request arrives.
• Pool imbalance: prefill queues while decode idles, or the reverse.
• KV-transfer saturation or timeout: the seam erases the phase-isolation benefit.
• Planner oscillation or cold-start lag: replica decisions arrive after the burst or repeatedly over-correct.
• Backend/version mismatch: a topology or cache feature is unsupported or immature for the chosen engine release.
• Placement or collective failure: a multi-node backend loses workers or communication performance.

Dashboard observed TTFT and ITL distributions, queue tokens, decode load/KV utilization, input/output lengths, worker counts, scaling decisions, cache overlap/hit/eviction, routing skew, transfer latency and throughput, errors, and per-worker health.

What the project evidence can establish

Common mistakes

• Calling engines one-node systems. Backend-native distributed execution and Dynamo coordination solve different responsibilities.
• Using parameter count as an adoption threshold. Measure fit, topology, workload, and SLO behavior.
• Treating cache state as router-owned truth. Backends retain blocks; events, expiry, and eviction determine whether routing estimates remain useful.
• Maximizing cache affinity without load control. Reuse can create hot workers.
• Treating KV transfer as free. Benchmark the connector and actual fabric under load.
• Equating ITL with aggregate token throughput. One is user-visible spacing; the other is system capacity.
• Assuming a conventional autoscaler is always sufficient—or always insufficient. Start with the simplest measured policy and change it when workload evidence demands more.

Practical guidance

• Start with a fixed replica count or the simplest autoscaler that meets the SLO. Conventional HPA/KEDA or custom policies can work when driven by appropriate inference metrics.
• When request content, cache affinity, phase imbalance, or TTFT/ITL behavior defeats those policies, evaluate an LLM-aware planner against the same workload and cost objective.
• Pin versions and verify the current backend feature matrix. Product behavior and maturity change across releases.
• Treat topology, router, planner, backend, and connector configuration as one versioned system; benchmark failure and recovery, not only steady-state throughput.

Summary

• An inference engine may span GPUs or nodes; Dynamo coordinates engines and distributed serving services. The boundary is responsibility, not node count.
• KV-aware routing combines estimated cache overlap with load and depends on backend-retained blocks, fresh state, and reuse support.
• Disaggregation can isolate and scale prefill/decode work, but KV transfer creates a measured topology-dependent seam.
• Planner SLA mode targets TTFT and ITL; other optimization targets include throughput, latency, and explicit load thresholds.
• Capacity distribution, throughput distribution, and phase specialization are distinct reasons to distribute.
• Adopt each capability only after it beats a simpler baseline on SLO, cost, utilization, resilience, or another named objective.

Knowledge check

What does an inference engine do versus NVIDIA Dynamo, and can either span nodes?

An inference engine executes and schedules model work, manages KV state, and runs kernels and parallelism. Depending on the backend, one replica can span GPUs or nodes. Dynamo coordinates engines and serving services: routing, deployment graphs, disaggregated pools, planning, cache/transfer integration, and cluster operations. Dynamo often operates across nodes but node count is not the defining boundary.

In SLA mode, which user-facing latency metrics does the Planner target, and why are they not simply “prefill time” and “tokens per second”?

The targets are TTFT and ITL. TTFT can include routing, queueing, scheduling, prefix reuse, prefill execution, and transfer—not only prefill compute. ITL measures token spacing for a request, while tokens per second measures aggregate capacity. The Planner also has throughput, latency, and load optimization targets, so state the configured policy.

A dense model on one GPU becomes slow under load. Is Dynamo the fix?

There is not enough information. First identify queueing, compute, memory bandwidth, KV capacity, model fit, kernels, and SLO configuration. Tune and benchmark the engine, then test replica scaling if capacity is the issue. Dynamo becomes relevant only if distributed routing, cache affinity, disaggregation, multi-worker orchestration, or LLM-aware planning solves a measured problem better than the simpler design.

Primary sources and version boundary

Product capabilities were reviewed against NVIDIA Dynamo documentation on 2026-08-02. Backend support and maturity vary by release; consult the current feature matrix.

• Introduction to NVIDIA Dynamo — platform components and composable capabilities.
• Dynamo Planner and Planner Guide — TTFT/ITL, optimization targets, and control modes.
• Dynamo Router Guide — cache-state inputs, overlap/load routing, and deployment modes.
• Dynamo disaggregated serving — prefill/decode routing and topology constraints.
• Dynamo disaggregation design — NIXL KV transfer and backend metadata.
• vLLM parallelism and scaling — backend multi-GPU and multi-node execution.
• TensorRT-LLM parallelism — backend tensor, pipeline, and expert parallelism.

Related chapters

• Inference runtimes — execution, scheduling, and backend capabilities
• Batching and concurrency — queues, running work, and token budgets
• The KV cache and context growth — retained state and capacity
• Latency, throughput and token rates — TTFT, ITL, throughput, and goodput
• Prefill and decode — phase behavior and regime-dependent bottlenecks