Model parallelism and distributed inference

Up to now, one GPU held the whole model. But frontier models are far too big for a single card's HBM, and even models that fit sometimes need more aggregate bandwidth than one card has. Then you must split the model across GPUs — and the moment you do, the wires between those GPUs become part of your performance model. This chapter covers the ways to split (tensor, pipeline, expert, replica) and the interconnect (NVLink, NVSwitch, PCIe, RDMA) that decides whether splitting buys you speed or just coordination overhead.

What you will understand by the end

• Why one GPU stops being enough — capacity or bandwidth.
• The main ways to distribute a model: replica, tensor, pipeline, expert parallelism.
• Scale-up vs scale-out, and why the interconnect (NVLink/NVSwitch vs PCIe/RDMA) determines what is viable.
• Why more GPUs is never linear speedup — communication is the tax.

Why one GPU stops being enough

Two independent reasons push you off a single card:

• Capacity. A model's weights (plus KV cache) exceed one GPU's HBM. A 140 GB model simply cannot load on an 80 GB card — it must be split, whatever the speed.
• Throughput/latency. Even a model that fits may need more aggregate memory bandwidth or compute than one card provides to hit your token-rate SLO.

The first is a hard wall; the second is an optimisation. Both lead to the same place: multiple GPUs cooperating on one model.

The four ways to split

  REPLICA (data)         TENSOR (within a layer)      PIPELINE (across layers)
  ┌────┐ ┌────┐          ┌──────────────────┐         ┌────┐→┌────┐→┌────┐→┌────┐
  │full│ │full│          │  layer split      │        │L1-8│ │L9- │ │L17-│ │L25-│
  │mdl │ │mdl │          │ GPU0 | GPU1 halves │        │    │ │16  │ │24  │ │32  │
  └────┘ └────┘          │  add results each  │        └────┘ └────┘ └────┘ └────┘
  separate requests      │  step (needs NVLink)│       one request flows through
  → more throughput      └──────────────────┘         → fits a huge model

• Replica / data parallelism. Put a full copy of the model on each GPU and send different requests to each. No model splitting, no cross-GPU chatter per token — this is how you add throughput once the model already fits. It does nothing for a model that is too big.
• Tensor parallelism (TP). Split each layer's big matrices across GPUs — each holds a slice, every GPU does part of every layer, and they combine results every step. This fits a bigger model and adds bandwidth, but it is extremely communication-heavy: GPUs exchange data on every layer, so it demands a fast link (NVLink) and is normally kept within a single node.
• Pipeline parallelism (PP). Put different layers on different GPUs — GPU 0 runs layers 1–8, GPU 1 runs 9–16, and so on. A request flows through the stages like an assembly line. Communication is light (only activations hand off between stages), so it tolerates slower links and spans nodes — but naive pipelines can leave GPUs idle ("bubbles") waiting for work.
• Expert parallelism (EP). For mixture-of-experts models, place different experts on different GPUs and route each token to the ones it needs. It scales sparse models but adds routing traffic across the fabric.

Real deployments combine these — e.g. tensor-parallel within a node, pipeline-parallel across nodes, replicated for throughput.

The interconnect decides what is viable

Splitting a model means GPUs constantly shipping data to each other, so the link between them is now on the critical path. The hierarchy, fastest to slowest:

  ┌──────────────────── ONE DGX-style node (8 GPUs) ────────────────────┐
  │   GPU0 ═╗   GPU1 ═╗   GPU2 ═╗   GPU3 ═╗   ...   GPU7 ═╗              │
  │         ║         ║         ║         ║              ║   NVLink      │
  │      ┌──╨─────────╨─────────╨─────────╨──────────────╨──┐  (fast,   │
  │      │            N V S W I T C H                       │   ~TB/s,  │
  │      └───────────────────────────────────────────────── ┘  all-to- │
  │         every GPU talks to every other at full NVLink speed  all)   │
  └────────────────────────────┬────────────────────────────────────────┘
                               │ RDMA / InfiniBand between nodes (~100s GB/s)
             ┌─────────────────┴─────────────────┐
        ┌────────┐                           ┌────────┐
        │ node 2 │   ... a pod/SuperPOD of    │ node N │
        └────────┘       many nodes           └────────┘

• NVLink — a direct, very fast GPU-to-GPU link (roughly 0.9 TB/s on Hopper, ~1.8 TB/s on Blackwell), far quicker than PCIe. It is what lets 8 GPUs in a node behave almost like one big GPU.
• NVSwitch — a switch that connects all the GPUs in a node so every GPU reaches every other at full NVLink speed (all-to-all), not just neighbours. This is what makes tensor parallelism across 8 cards practical.
• PCIe — the general bus to the CPU and to GPUs without NVLink (~64 GB/s on Gen5). An order of magnitude slower than NVLink; a tensor-parallel job over PCIe spends most of its time waiting.
• RDMA / InfiniBand — fast networking that moves data between nodes, GPU-to-GPU, bypassing the CPU. This is the fabric of multi-node clusters (pods, SuperPODs).

Scale-up versus scale-out

Two directions of growth, and the interconnect is the dividing line:

• Scale-up — a bigger single node: more GPUs joined by NVLink/NVSwitch so they act as one large accelerator. This is where tight, communication-heavy splits (tensor parallelism) live. Limit: a node holds a fixed number of GPUs.
• Scale-out — more nodes joined by RDMA networking. This is where you add capacity beyond one node (pipeline across nodes, more replicas) — but cross-node links are slower, so you keep the chatty splits inside a node and the loose ones across nodes.

The tax: it is never linear

Two GPUs never give exactly 2× — every split adds coordination:

• Tensor parallelism adds an all-reduce every layer; past a point, more GPUs mostly add communication.
• Pipeline parallelism can leave stages idle (bubbles) waiting for the previous stage.
• The right degree of parallelism is the smallest that meets your capacity and latency needs — over-splitting spends more on wires than it earns in compute.

Mental model

Split a model only when one GPU cannot hold it or cannot hit the SLO. Use replicas for throughput, tensor + pipeline parallelism to fit a giant model — and place the chatty split (tensor) on the fast wire (NVLink, inside a node) and the loose split (pipeline, replicas) on the slow wire (RDMA, across nodes). Communication is always the tax.

Common mistakes

• Splitting a model that fits on one GPU. You add communication overhead for nothing; a replica would have added throughput instead.
• Tensor parallelism over PCIe (or non-NVLink cards). The per-layer all-reduce over a slow link dominates — you get a slower model on more hardware.
• Expecting linear speedup. N GPUs give well under N× because of all-reduce and pipeline bubbles; the tax rises with the split degree.
• Mismatching split and wire. Chatty splits across nodes (or loose splits wasting NVLink) both leave performance on the table.

Practical guidance

• Decide why you are going multi-GPU first: capacity (must split — TP/PP) or throughput (replicate).
• Keep tensor parallelism within a node (NVLink/NVSwitch); use pipeline/replicas across nodes (RDMA).
• Choose the smallest parallelism degree that fits the model and meets latency — then measure the real, sub-linear speedup, don't assume it.
• If you are on non-NVLink cards and the model doesn't fit, the answer is different hardware, not more of the same cards.

Summary

• One GPU stops being enough on capacity (model too big) or throughput/latency; both lead to distributing the model.
• Replica parallelism adds throughput; tensor and pipeline parallelism fit a model too big for one card; expert parallelism scales MoE models.
• The interconnect — NVLink/NVSwitch (fast, in-node) vs PCIe/RDMA (slower, cross-node) — decides which split is viable; match the chatty split to the fast wire.
• Speedup is always sub-linear; use the smallest split that works.

Knowledge check

A 140 GB model won't load on your 80 GB GPUs. You have a box of 8 L4s (no NVLink). Can you serve it well by splitting it across them? Why or why not?

Not well. Splitting a 140 GB model needs tensor/pipeline parallelism, and tensor parallelism communicates across GPUs every layer — which requires a fast link. The L4s have no NVLink, so they can only talk over slow PCIe, and the per-step communication would dominate. The real fix is NVLink-class cards (e.g. H100/H200 in an NVSwitch node), not more L4s.

Your model already fits on one GPU, but you need to serve 5× the traffic. Which form of parallelism do you reach for, and which do you avoid?

Reach for replica (data) parallelism — full copies on more GPUs, each handling different requests — because your problem is throughput, not capacity. Avoid tensor or pipeline parallelism: splitting a model that already fits only adds cross-GPU communication overhead without solving anything.

Related chapters

• Inside an NVIDIA GPU — the NVLink/HBM levels this chapter scales beyond one die
• GPU architecture and inference hardware — why NVLink is the lever that makes multi-GPU viable
• The KV cache and context growth — the other thing that must fit alongside the weights
• Orchestration and disaggregation — splitting the prefill and decode phases across GPU pools