Model artifacts, compilation, and serving layers

Between the CUDA floor and a public inference API is not one exclusive route. It is a stack of choices. A production service might load safetensors into a PyTorch model, apply torch.compile and custom kernels, then let an engine schedule requests and manage the KV cache.

What you will understand by the end

• The four layers people commonly collapse into “the framework.”
• What safetensors protects—and what remains executable software.
• Where torch.compile, ONNX Runtime, TensorRT, Transformers, and LLM engines fit.
• How to decompose cold start before choosing a remedy.

The four-layer stack

Layer	Examples	The question it answers
Repository and artifacts	safetensors, GGUF, ONNX data, tokenizer and config files	What files came from the model producer?
Model implementation	Transformers/PyTorch, Diffusers, engine-native code, exported ONNX graph	What defines the computation?
Execution and optimization	eager PyTorch, torch.compile, ONNX Runtime, TensorRT, CUDA/Triton kernels	How is that computation executed?
Serving system	Transformers Serve, vLLM, SGLang, TensorRT-LLM	How are requests scheduled, state managed, and failures operated?

One concrete decomposition is:

Qwen checkpoint in safetensors
→ Qwen implementation in Transformers/PyTorch
→ vLLM model runner + torch.compile + custom kernels
→ vLLM scheduler + paged KV cache + streaming API

The arrows are boundaries, not mandatory handoffs. vLLM V1, for example, describes torch.compile as a critical, default-enabled component. Current TensorRT-LLM uses PyTorch as its sole execution backend and loads supported Hugging Face checkpoints without the legacy engine-build step. An inference engine can therefore contain a compiler and custom kernels rather than replace them.

Artifact taxonomy: do not compare unlike objects

Artifact	Contains	Requires at runtime	Typical reason to use
safetensors	Tensor data and limited metadata; no executable model graph	A compatible model implementation and loader	Safe, fast checkpoint interchange
ONNX	A portable computation graph plus embedded or external initializer tensors	An ONNX-compatible runtime and any custom operators	Portability across runtimes, languages, and devices
Runtime-specific artifact	A compiled or transformed representation; exact contents vary	Matching runtime and sometimes software or hardware constraints	Faster startup or optimized execution for a fixed target

Pickle-backed PyTorch checkpoints—often named pytorch_model.bin—can execute code during deserialization. Safetensors removes that code path from tensor loading and supports lazy access and zero-copy reads.

What torch.compile does—and does not do

torch.compile captures PyTorch graphs and generates optimized execution, including fusion where the graph and backend permit it. Registered custom CUDA operators and user-defined Triton kernels can compose with the compiler. An opaque custom kernel is usually a boundary: the compiler can call it, but does not automatically rewrite its internals.

That is compatible with an inference engine. Engines can add model-specific compiler passes and tuned kernels around the broader serving loop.

ONNX Runtime and TensorRT: graph-oriented runtimes

ONNX Runtime is a serious option when graph portability, cross-language deployment, or CPU, GPU, browser, mobile, or NPU execution matters. Its Generate API supports text generation mechanics including tokenization, sampling, logits processing, KV-cache management, chat templates, and structured output. The API is currently documented as preview, so bind a deployment decision to tested versions and supported models.

TensorRT is an NVIDIA-focused runtime for optimized graph deployments, especially when the model exports cleanly and hardware is fixed. It builds a serialized execution plan from a TensorRT network definition. ONNX parsing is a common import route, but the network can also be constructed programmatically.

Unsupported or highly dynamic operations can break export or require decomposition, graph rewriting, custom operators, fallback partitions, or another runtime. Validate export, numerical equivalence, dynamic shapes, and performance early.

Model library versus serving system

Hugging Face model-card snippets remain excellent correctness references, but current Transformers is no longer limited to single-request notebook inference. It includes an OpenAI-compatible server, continuous batching, paged attention, prefix caching, optimized attention backends, CUDA graphs, quantization, streaming, and tensor parallelism.

Capability	Minimal model-card script	Current Transformers serving	Specialized LLM engine
Correctness baseline	Strong	Strong	Depends on supported model
HTTP and streaming API	No	Yes	Yes
Continuous batching	No	Yes	Yes
Paged KV management	Usually no	Yes	Yes
Distributed and operational envelope	Application-built	Verify for the tested version	Often broader or deeper; verify
Peak workload-specific performance	Baseline	Benchmark	Benchmark

The distinction is therefore model library first versus specialized serving system with a broader operational envelope, not “learning” versus “serving.” A minimal notebook loop usually omits scheduling and operational controls; benchmark it as a reference baseline, not as proof that its library cannot serve efficiently.

Cold start is a pipeline

Safetensors can reduce deserialization copies and peak host-memory pressure. A pre-quantized checkpoint reduces bytes to download, read, and transfer. If weight movement dominates, halving bytes can approach a 2× reduction for that phase—but total startup rarely scales perfectly because fixed work, conversion, compilation, and warm-up remain.

Phase to time	Evidence to capture
Platform scheduling and process start	Platform and container timestamps
Model fetch	Cache state, source, bytes, download logs
Checkpoint read and deserialization	Storage type and loader timestamps
Host-to-device transfer	Device-transfer timestamps or profiler trace
Compile, autotune, and graph capture	Engine logs and cache hit/miss
Warm-up to readiness	Warm-up shape and readiness timestamps

A defensible selection procedure

1. Identify the required model code, tokenizer or processor, and modality.
2. Choose trusted artifacts; prefer safetensors for tensors and pin an immutable revision.
3. Decide whether graph, language, or device portability is required.
4. Filter runtimes by model, operator, hardware, precision, and deployment support.
5. Filter serving systems by concurrency, latency, cache, parallelism, and operational needs.
6. Benchmark correctness, startup phases, TTFT, inter-token latency, throughput, memory, and tail latency under the same workload.

Common mistakes

• Treating safetensors, compilation, ONNX, and an engine as competing choices.
• Assuming safetensors secures remote code or custom native operators.
• Comparing a synchronous notebook loop with a continuously batched server.
• Treating export failure as impossible—or as impossible to work around.
• Calling all startup delay “weight loading” without timing each phase.
• Using a permanent product ranking instead of a version-bound benchmark.

Summary

• Separate artifacts, model implementation, execution optimization, and serving; modern stacks combine them.
• Safetensors protects tensor deserialization, not the whole repository supply chain.
• torch.compile accelerates captured execution; an engine operates requests and state.
• Transformers now has a real serving path; benchmark specialized engines when you need broader scale, model coverage, or operational capabilities.
• ONNX Runtime is not restricted to vision, and cold start is not one phase.

Knowledge check

Why are “safetensors versus torch.compile versus an inference engine” not three alternatives?

They answer questions at different layers. Safetensors stores tensor data; torch.compile optimizes captured PyTorch execution; an inference engine schedules requests, manages KV state, and operates an API. One service can use all three.

What security risk remains after converting weights to safetensors?

Repository code, trust_remote_code=True, preprocessing, custom operators, native extensions, and dependencies may still execute software. Review and pin the repository revision, verify provenance, and isolate untrusted code or conversion.

How should you diagnose a two-minute cold start?

Timestamp scheduling, process start, download/cache lookup, checkpoint read, device transfer, compilation/autotuning/graph capture, and warm-up separately. Then optimize the measured dominant phase instead of assuming weight format is the cause.

Primary sources and version boundary

Product capabilities below were reviewed August 2, 2026 and should be rechecked when versions change.

• Transformers Serve and server optimizations
• Transformers continuous batching and paged attention
• vLLM torch.compile integration
• PyTorch custom operators and user-defined Triton kernels
• ONNX Runtime Generate API
• safetensors documentation and Hub pickle security
• ONNX IR and external tensor data
• TensorRT-LLM backend-removal guide

Related evidence

• Cold start as a serving-design constraint →
• Compare current inference engines →