NVMe-oF: crucial for multi-node LLM training performance

On a Tuesday in late May, we were wrestling with a multi-node Llama 3 70B pre-training job. The GPUs — eight H100s, running at over $200 an hour — were sitting at a dismal 65% utilization. The culprit wasn’t our code or the interconnect, but a plain old network file system, bottlenecking data ingest. We were paying top dollar for GPUs to wait on storage. It’s a familiar problem, and it’s why NVMe-oF for multi-node LLM training is no longer a ‘nice to have’ but a fundamental requirement for efficiency.

The multi-node llm training storage bottleneck

When you’re training large language models, especially those pushing hundreds of billions or even trillions of parameters, you hit a wall. Local NVMe drives on a single GPU instance are blisteringly fast, often delivering sequential reads upwards of 5-7 GB/s. That’s fine for a single A100 or H100, which might ingest data at 2-3 GB/s during peak training.

The problem starts when you scale out. Once you’re distributing your model across 8, 16, or even 64 GPUs, each needing to pull fresh data from a shared source, that single local NVMe drive on a head node (or even shared network attached storage) becomes a choke point. Your network file system — think NFS or SMB — can only push so much data. Even with 100 Gbps Ethernet, which theoretically offers around 12.5 GB/s, you’re battling protocol overhead, CPU contention, and the inherent latency of traditional TCP/IP stacks.

We’ve seen it repeatedly: a multi-node job that should be saturating GPUs at 95% utilization instead idles at 60-70% because the data isn’t arriving fast enough. This isn’t just an annoyance; it’s money draining away. Those idle GPU cycles are still on your bill. And if you’re not careful, the hidden costs of GPU instance storage can balloon quickly, turning what looks like a cheap hourly rate into an expensive, inefficient nightmare. Modern LLM training workloads, especially those using techniques like data parallelism or pipeline parallelism, demand a constant, high-throughput stream of data, and traditional network storage simply wasn’t built for it.

How nvme-of changes multi-node storage

NVMe-oF, or NVMe over Fabrics, is a game-changer because it takes the raw speed and low latency of NVMe SSDs and extends it across a network. Instead of communicating with storage through traditional block protocols (like SCSI over Ethernet) or file protocols (like NFS), NVMe-oF allows NVMe commands to travel directly over high-speed networks, often leveraging technologies like RDMA (Remote Direct Memory Access).

RDMA is key here. It allows data to move directly between memory in one machine and memory in another, bypassing the CPU, operating system, and kernel stack on both sides. This dramatically reduces latency and CPU overhead, which are major bottlenecks in traditional network storage. When you combine NVMe-oF with RDMA over InfiniBand or RoCE (RDMA over Converged Ethernet), you’re looking at network storage that can approach the performance characteristics of local storage.

Consider the difference: a typical NFS share might offer ~500 MB/s to a single client with ~200-500 microsecond latency. A well-tuned NVMe-oF setup, however, can provide multiple gigabytes per second to a single client (or aggregate many clients to tens of GB/s) with latency often in the single-digit microseconds, sometimes even sub-microsecond. For instance, CoreWeave states their NVMe-oF storage offers up to 400 GB/s bandwidth for large datasets [https://www.coreweave.com/products/storage]. This isn’t just faster; it’s a fundamental shift in how remote storage interacts with compute, making it feel almost local.

NVMe-oF’s impact on llm dataset loading and checkpointing

The direct beneficiaries of NVMe-oF are the I/O-intensive phases of LLM training. Let’s break down the common culprits:

1. Initial dataset loading: Before training can even begin, your multi-terabyte dataset needs to be loaded into memory or cached locally. If you’re pulling a 50 TB dataset over a traditional network file system, this can take hours. With NVMe-oF, that time shrinks dramatically, shaving off significant setup time from every training run. This is especially critical for iterative development where you might restart training jobs frequently.
2. Intermediate checkpointing: During long training runs, saving model checkpoints is crucial for fault tolerance and resuming training. These checkpoints can be massive, often hundreds of gigabytes for large models. Writing these synchronously to slow network storage can pause training or introduce significant stalls. NVMe-oF allows these large writes to complete much faster, minimizing disruption to GPU processing. This also improves the reliability of your training, as checkpoints are saved quicker and more frequently.
3. Model saving and loading: Beyond checkpoints, the final model saving and subsequent loading for inference or fine-tuning are also heavily I/O bound. If you’ve ever waited for LLM model load times from cloud block storage, you know how frustrating (and expensive) this can be. NVMe-oF accelerates this, reducing the time from job start to actual computation. NVIDIA’s GPUDirect Storage also plays a role here, working in concert with NVMe-oF by allowing direct data transfer between storage and GPU memory, bypassing the CPU. NVIDIA claims that GPUDirect Storage reduces CPU overhead by 90% and latency by 18% by enabling direct data transfer to GPUs [https://developer.nvidia.com/gpudirect-storage]. This combination means your GPUs spend more time crunching numbers, not waiting for data.

Cloud provider support and what to look for

Identifying cloud providers who genuinely offer NVMe-oF benefits for multi-node LLM training setups requires looking beyond marketing speak. Many providers offer