Model parallelism and distributed inference

More than one GPU can solve different problems. A model may not fit on one device. One replica may miss a latency or concurrency target. Or aggregate traffic may simply require more independent workers.

Those problems do not all imply “split the model.” The durable decision is: identify what must be distributed, then place its communication on a topology that can support it.

What you will understand by the end

• The difference between distributing requests and sharding one model instance.
• What replica, tensor, pipeline, and expert parallelism distribute.
• Why tensor parallelism benefits strongly from fast interconnect but does not require NVLink as a matter of correctness.
• Why scale-up describes a tightly coupled accelerator domain—not necessarily one server.
• How to compare link bandwidth without mixing per-direction, bidirectional, per-port, and aggregated figures.
• Why the best TP/PP degree is a benchmark result rather than a fixed rule.

First name the constraint

Several distinct constraints can move a deployment beyond one GPU:

• Model fit. At the selected format and residency strategy, weights plus runtime state exceed usable device memory. The options include quantization, pruning, CPU/NVMe offload, or sharding the model.
• Single-request latency. More devices may reduce per-device computation, but only if communication does not erase the gain.
• Aggregate throughput. Independent replicas can distribute requests when one copy already meets per-request requirements.
• KV capacity and concurrency. Sharding or replicating can change how much KV state the service can hold and how requests are assigned.

A fully resident 140 GB representation cannot load on one 80 GB GPU. That is a capacity fact for that representation—not proof that one particular sharding strategy or GPU family is required.

Four common distribution strategies

Strategy	What is distributed	Typical reason
Replica / data parallel	Requests across complete model copies	Aggregate throughput and availability
Tensor parallel (TP)	Tensors within transformer blocks	Model fit, per-rank compute/memory, sometimes latency
Pipeline parallel (PP)	Layers or blocks across stages	Model fit with less frequent communication
Expert parallel (EP)	Experts in a mixture-of-experts model	Sparse-model capacity and throughput

  REPLICAS                 TENSOR SHARDS              PIPELINE STAGES
  ┌────┐  ┌────┐           ┌──────────────────┐       ┌────┐→┌────┐→┌────┐
  │full│  │full│           │ one block split  │       │L1-8│ │L9- │ │L17-│
  │model│ │model│          │ GPU0 | GPU1      │       │    │ │16  │ │24  │
  └────┘  └────┘           │ frequent collectives│    └────┘ └────┘ └────┘
  separate requests        └──────────────────┘       activations cross stages

Replica or data parallelism

Each GPU holds a complete model copy and processes different requests. This distributes traffic; it does not split one model instance. Cross-replica coordination may exist for routing, cache affinity, or load balancing, but the token computation does not need a collective across replicas.

Tensor parallelism

TP divides matrices within transformer blocks. Ranks process the same token positions and exchange partial results through frequent collectives. Depending on the partition and runtime, those may include all-reduce, all-gather, reduce-scatter, or combinations—not one universal “all-reduce every layer.”

Fast, low-latency links often make TP much more attractive. It is commonly placed inside the fastest communication domain, but it can execute over PCIe or a high-speed multi-node fabric at potentially substantial cost.

Pipeline parallelism

PP places different layers on different stages. Activations move between adjacent stages, so communication is less frequent than TP. It can therefore be a better starting point on slower links. Pipeline balance, microbatching, and bubbles still determine utilization and latency; “less communication” does not mean “free.”

Official vLLM scaling guidance specifically recommends considering pipeline rather than tensor parallelism on GPUs without NVLink, using L40S as its example.

Expert parallelism

EP places different experts on different ranks and routes tokens to the selected experts. This is not a loose communication pattern: token dispatch and return can create substantial all-to-all traffic, while uneven expert popularity creates load imbalance. EP is highly topology- and workload-sensitive.

Production deployments often combine strategies—for example, TP inside a fast domain, PP across domains, EP for MoE layers, and replicas for traffic.

Interconnect changes the viable operating envelope

NVLink and NVSwitch lower the cost of tightly coupled execution. They do not turn several GPUs into one transparent device. The runtime still shards weights, places KV state, performs collectives, synchronizes ranks, and handles imbalance and failures.

Bandwidth comparisons also need a common basis:

Link example	Advertised bandwidth basis
H100 NVLink	900 GB/s aggregate per GPU
PCIe Gen5 x16	64 GB/s each direction; 128 GB/s bidirectional
NDR 400 Gb/s NIC	50 GB/s raw line rate per port
Multi-rail node fabric	Aggregate depends on NIC count, rail layout, and topology

The H100 specifications provide the NVLink and PCIe figures. The DGX H100/H200 system guide shows eight 400 Gb/s cluster-network interfaces—illustrating why a node aggregate must not be compared with a single-port number.

Advertised link bandwidth is not application collective throughput. Direction, message size, topology, protocol overhead, contention, and collective implementation determine what the runtime sustains.

Scale-up and scale-out are communication domains

• Scale-up is a tightly coupled, high-bandwidth accelerator domain. Historically this often meant one NVSwitch node. Current NVLink domains can span a rack: NVIDIA documents NVLink domains of up to 72 GPUs.
• Scale-out connects accelerator domains through a cluster fabric such as InfiniBand or Ethernet. It reaches more devices but usually changes latency, bandwidth, failure, and placement characteristics.

Therefore, “TP within one node; PP between nodes” is a useful historical starting point, not a law. TensorRT-LLM notes an explicit exception for rack-scale multi-node NVLink systems, and NVIDIA's inference reference architecture includes multi-node tensor, pipeline, and expert execution.

Scaling behavior depends on what you scale

Sharding one model across more GPUs usually yields sublinear speedup because it adds communication and synchronization:

• TP trades smaller per-rank matrix operations for frequent collectives.
• PP trades less frequent transfers for stage imbalance and pipeline bubbles.
• EP adds routing collectives and may suffer expert imbalance.

Replicas are different. Aggregate replica throughput can scale closer to linearly while the router, CPU, network, cache placement, and request distribution remain controlled. That is why “two GPUs never give 2×” is too broad.

Starting with the smallest shard degree that fits often reduces communication, but it may miss latency, throughput, or KV-capacity goals. A higher degree may help enough to justify its communication. The TensorRT-LLM sharding guide treats TP, PP, and their combination as configurations to compare.

Practical decision sequence

1. State the objective: model fit, single-request latency, aggregate throughput, KV capacity, availability, or some combination.
2. Test whether format changes, quantization, or offload solve the capacity problem more simply than sharding.
3. Start with replicas when the model fits and per-replica latency and concurrency are acceptable.
4. If one instance must span devices, compare supported TP, PP, EP, and hybrid layouts.
5. Place communication-heavy groups on the fastest available domain; label every bandwidth figure by direction and aggregation basis.
6. Benchmark nearby TP/PP degrees with representative prompt lengths, output lengths, batch policy, and concurrency. Measure TTFT, inter-token latency, throughput, KV capacity, communication time, and cost.

Common mistakes

• Calling replicas a model split. They distribute requests across complete copies.
• Treating NVLink as a correctness requirement. It improves the operating envelope; topology-aware alternatives still exist.
• Assuming PP is always light or EP is loose. Bubbles, activation transfers, all-to-all routing, and imbalance can dominate.
• Comparing unlike bandwidth figures. Per-direction, bidirectional, per-GPU, per-port, and multi-rail aggregates are different quantities.
• Assuming all multi-GPU scaling is sublinear. That is a safe expectation for a single sharded request, not necessarily for independent replica throughput.
• Stopping at “the smallest degree that fits.” It is a benchmark starting point, not the objective.

Summary

• Replicas distribute requests; TP, PP, and EP distribute parts of one model instance.
• Fast interconnect strongly benefits communication-heavy parallelism but is not a universal prerequisite for serving a model larger than one GPU.
• Scale-up means a tightly coupled accelerator domain, which may span a rack; it no longer means exactly one eight-GPU server.
• Sharded speedup is usually sublinear. Replica throughput can scale closer to linearly until a shared component binds.
• Choose from fit, latency, throughput, KV capacity, topology, and cost—then benchmark the exact runtime configuration.

Knowledge check

A 140 GB weight representation will not load on one GPU. You have eight 24 GB L4s without NVLink. Is serving it possible?

Possibly. The cards provide 192 GB of aggregate raw device memory, but the 140 GB weights are not the entire requirement: runtime workspaces, quantization metadata, KV state, and per-stage imbalance also consume memory. Pipeline parallelism is generally a better starting point than communication-heavy TP on PCIe-only GPUs. Quantization or offload may also help. Whether the deployment serves well depends on framework support, balance, TTFT, inter-token latency, throughput, and cost measured under the intended workload.

The model fits on one GPU, but traffic is expected to grow 5×. What should you try first?

Start with replicas if one copy already meets latency and concurrency requirements. Their independent request processing can scale aggregate throughput without per-token sharding collectives. Consider TP or PP if one replica cannot meet latency, KV capacity, or other per-instance requirements—even when the weights technically fit.

Why can you not compare “900 GB/s NVLink” directly with “400 Gb/s networking”?

The units and aggregation bases differ. 400 Gb/s is 50 GB/s of raw line rate for one port; the NVLink figure is an aggregate per-GPU number. You must also label direction, port or rail count, topology, and whether the value is advertised bandwidth or measured collective throughput.

Primary sources

• vLLM parallelism and scaling — TP/PP combinations and PP guidance for GPUs without NVLink.
• TensorRT-LLM parallelism strategies — replica, tensor, pipeline, expert, and hybrid strategies.
• TensorRT-LLM model-sharding guide — topology-aware TP/PP selection.
• NVIDIA H100 specifications and DGX H100/H200 system guide — link and multi-rail bandwidth bases.
• NVIDIA NVLink and inference reference architecture — rack-scale domains and multi-node execution.

Related chapters

• Inside an NVIDIA GPU — the device components being distributed
• GPU architecture and inference hardware — topology as a bottleneck hypothesis
• The KV cache and context growth — runtime state that must fit alongside weights
• Orchestration and disaggregation — separating prefill and decode pools