CUDA kernels for LLM inference

CUDA kernels are the executable building blocks behind NVIDIA GPU inference. Performance depends on which implementation runs for a specific operation, shape, precision, software stack, and GPU—not just whether a model “supports CUDA.” Architecture-specific tuning matters, but functional compatibility and performance portability are different questions.

Most application and platform engineers select, configure, and benchmark kernels rather than author them. Kernel, compiler, and inference-engine engineers may write or generate specialized kernels in CUDA C++, Triton, CUTLASS, or related systems.

What you will understand by the end

• What CUDA and GPU kernels contribute to LLM inference.
• How cubins, PTX, fat binaries, JIT compilation, and compute capability affect compatibility.
• How dispatch selects an implementation—and why correct execution may still be slow.
• What fusion can remove, what it can make worse, and how to benchmark it honestly.
• Why FlashAttention is an IO-aware algorithm, not merely a fused decode kernel.
• How torch.compile, optimized operators, and inference engines overlap and differ.

The serving stack has overlapping responsibilities

Product boundaries differ, and engines can span GPUs or nodes. A useful responsibility map is:

Serving platform / distributed runtime
  routing, autoscaling, worker coordination, disaggregation

Inference engine
  scheduling, continuous batching, KV management, parallel execution, model kernels

Framework / compiler / kernel libraries
  PyTorch, Inductor, Triton, cuBLASLt, CUTLASS, FlashInfer, custom operators

CUDA software stack
  programming model, runtime and driver APIs, compiler toolchain, libraries

GPU hardware

An inference engine such as vLLM, SGLang, or TensorRT-LLM may perform multi-GPU and multi-node execution. A distributed framework such as NVIDIA Dynamo coordinates serving components and integrates inference backends; it is not simply a layer above a one-node engine.

CUDA itself is a platform and programming model with runtime and driver APIs, a compiler toolchain, and libraries. A kernel is a GPU function launched over many parallel threads. The same model operation can have multiple kernel implementations with very different performance characteristics.

Hardware-sensitive does not mean universally hardware-locked

CUDA code can reach a GPU in several forms:

• A cubin contains native machine code for a target streaming-multiprocessor version such as sm_90. NVIDIA defines limited binary compatibility rules within a compute- capability major family; major generations are not generally cubin-compatible.
• PTX is a virtual instruction-set representation. Compatible PTX can be JIT-compiled by the driver for a later GPU, providing forward functional compatibility.
• A fat binary can package cubins for several targets plus PTX, allowing the loader to select the best available image and retain a forward-compatible path.

These mechanisms answer “can it run?” They do not guarantee “does it exploit this GPU?” A PTX-JIT or compatible implementation may miss new tensor-core modes, instructions, memory behavior, or preferred tile shapes. Conversely, a newer architecture-specific cubin may not run on an older major architecture.

Kernel dispatch: what actually runs

Dense Transformer linear layers are usually lowered to GEMM implementations selected from libraries or generated kernels—commonly cuBLAS/cuBLASLt, CUTLASS-based code, Triton, or engine-specific implementations. cuDNN supplies a separate set of deep-learning primitives. Selection can happen through heuristics, autotuning, ahead-of-time compilation, JIT compilation, or runtime dispatch.

The effective implementation depends on shapes, strides, dtype, quantization scheme, GPU, workspace, library versions, and engine configuration. Unsupported or unmatched combinations can fail, select a less-specialized native implementation, upcast or dequantize, JIT-compile, or fall back depending on the stack.

Inspect dispatch or engine logs and profiler traces, then benchmark. Logs tell you what the stack intended to select; traces and measurements show what executed and how it behaved.

Kernel fusion: fewer boundaries, conditional benefit

Separate kernels often materialize intermediate tensors in GPU memory:

UNFUSED:  read x → op A → write tmp → read tmp → op B → write out
FUSED:    read x → op A + op B while state is on-chip → write out

Fusion can remove intermediate global-memory traffic, allocation, kernel launches, and synchronization. That is valuable when those costs are material. It is not free: a fused kernel can increase register pressure, shared-memory use, recomputation, code size, compilation time, or specialization cost, and can reduce occupancy. Compute-bound work or an already efficient library epilogue may gain little; over-fusion can regress performance.

FlashAttention changes the execution algorithm

FlashAttention is an IO-aware exact attention algorithm. It tiles attention so blocks of scores and probabilities remain in on-chip SRAM, applies an online softmax, and avoids materializing the full attention matrix in HBM. Implementations fuse multiple stages into specialized kernels, but the central contribution is reducing IO through a different tiled execution algorithm—not generic operator fusion alone.

The original work demonstrates broad benefits for training and long-sequence attention. Decode attention has related IO concerns but a different shape: one or a small number of query tokens attends over an existing KV cache. Prefill/training and decode may therefore use different kernels, tiling, and scheduling strategies. “FlashAttention support” is not enough to infer which phase or shape gets the benefit.

torch.compile and inference engines solve overlapping problems

torch.compile can fuse many PyTorch operations. PyTorch's scaled-dot-product attention can dispatch to optimized CUDA backends, and compiled graphs can compose with registered custom operators and user-defined Triton kernels. The exact composability depends on how an operator is registered: an opaque custom-op boundary can prevent fusion through that call, while a traceable Triton operator can expose more optimization opportunity.

The compiler does not automatically invent every model-, quantization-, shape-, and hardware-specific kernel. But this limitation is not the sole reason inference engines exist. Engines add a serving runtime around model execution:

• request scheduling and continuous batching;
• paged KV-cache allocation, prefix reuse, and admission or preemption;
• distributed tensor, pipeline, expert, and multi-node execution;
• quantization and model-specific kernels;
• speculative decoding and CUDA graph management;
• observability and production-facing controls.

Workload context, not fusion evidence

How to benchmark a fusion claim

A trustworthy microbenchmark preserves semantics and measures GPU execution rather than Python launch timing:

1. Implement an explicitly unfused path that materializes the intended intermediate and a fused path with equivalent numerical semantics.
2. Prevent the compiler from silently rewriting the baseline; verify both graphs or traces.
3. Warm up JIT compilation, autotuning, allocator state, and caches.
4. Use CUDA events or a GPU-aware benchmark utility with correct synchronization.
5. Sweep representative tensor shapes, sequence/batch regimes, dtypes, and layouts.
6. Report latency and kernel count, plus achieved bandwidth and occupancy/resource use when they support the diagnosis.
7. Confirm with Nsight Systems or Nsight Compute that the expected kernels, launches, and memory transfers occurred.

End-to-end model and serving benchmarks still matter: a faster isolated kernel may be too small a fraction of the workload to move TTFT, ITL, throughput, or goodput.

Common mistakes

• Equating functional compatibility with tuning. PTX JIT can make code run on a newer GPU without exploiting its best execution modes.
• Treating fusion as a guaranteed bandwidth win. Verify the eliminated traffic and the fused kernel's resource use; another bottleneck may replace it.
• Describing FlashAttention as fusion alone. Its IO-aware tiling and online softmax change how exact attention is executed and materialized.
• Claiming torch.compile cannot use optimized or custom kernels. It can compose with several such paths; opaque boundaries narrow cross-op fusion.
• Reducing an engine to kernel integration. Scheduling, KV memory, batching, distributed execution, and serving controls are first-class engine responsibilities.
• Assuming every mismatch silently emulates. A stack may error, JIT, change precision, or dispatch another implementation. Inspect and measure.

Practical guidance

1. Record GPU model and compute capability, host driver, CUDA runtime/toolkit, framework, engine, library versions, architecture targets, model, dtype, shapes, and parallelism.
2. Pin a validated engine/runtime container for reproducibility, but separately verify host- driver compatibility, CUDA requirements, communication libraries, and the engine support matrix. Containers normally depend on the host NVIDIA driver.
3. For new hardware, choose a release that explicitly supports the GPU rather than assuming an older pinned image provides tuned performance.
4. Check dispatch logs and profiler traces for the actual GEMM, attention, quantization, and communication paths; benchmark representative workloads and SLO metrics.
5. Treat a kernel optimization as a hypothesis. Measure before and after, then verify that the end-to-end serving metric moves as predicted.

Summary

• CUDA kernels are GPU executable units selected for particular operations, shapes, precisions, software versions, and hardware.
• Cubins, PTX, and fat binaries have different compatibility behavior; correct execution on a new GPU does not guarantee tuned performance.
• Fusion can remove intermediate traffic and launches, but resource pressure or the wrong workload can erase or reverse the gain.
• FlashAttention is an IO-aware tiled exact-attention algorithm, not merely a fused decode kernel.
• torch.compile can compose with optimized attention and custom/Triton operators; inference engines add a much broader serving runtime.
• Logs, profiler traces, kernel benchmarks, and end-to-end measurements answer different parts of the performance question.

Knowledge check

A teammate says, “torch.compile means we no longer need an inference engine.” What is missing?

torch.compile can optimize model execution and compose with optimized attention, registered custom operators, and Triton kernels. An inference engine also supplies serving- time scheduling, continuous batching, KV-cache management, distributed execution, quantization, model-specific kernels, speculative decoding, and other runtime controls. The tools overlap but solve different portions of the system.

A CUDA kernel runs correctly on a new GPU but performs below expectations. How can that happen?

PTX JIT or a compatible packaged binary may preserve functional execution while the selected implementation lacks tuning for the new GPU's tensor-core modes, instructions, memory hierarchy, or preferred tile shapes. Verify the executed kernel, architecture target, precision path, library and engine versions, profiler counters, and benchmark result.

Why can fusion improve performance, and why is it not guaranteed to do so?

It can remove intermediate global-memory traffic, allocations, launches, and synchronization. It can also increase register or shared-memory pressure, reduce occupancy, add recomputation, or produce a poorly specialized kernel. The result depends on shapes, precision, hardware, and the original bottleneck and must be measured.

Primary sources

• CUDA Programming Guide: the CUDA platform — compute capability, cubins, fat binaries, PTX, and JIT compatibility.
• NVIDIA Blackwell Compatibility Guide — forward compatibility and architecture support.
• PyTorch scaled dot-product attention — optimized attention backend dispatch.
• PyTorch custom operators — registered custom operations and subsystem integration.
• User-defined Triton kernels with torch.compile — custom Triton composability and traceable operators.
• FlashAttention — IO-aware exact attention through tiling and reduced HBM access.
• TensorRT performance benchmarking — CUDA-event timing, profiling, and reproducible measurement.
• vLLM parallelism and scaling — multi-GPU and multi-node engine execution.
• NVIDIA Dynamo introduction — distributed inference framework responsibilities.

Related chapters

• GPU architecture and inference hardware — compute capability and performance resources
• Prefill and decode — phase shapes and regime-dependent bottlenecks
• Inference runtimes — scheduling, KV management, and engine execution
• Batching and concurrency — serving work selected around the kernels
• Quantization — precision paths and specialized kernel support