Distributed Data Preprocessing Pipelines for Petabyte Corpora

What a petabyte preprocessing pipeline looks like end to end

Preprocessing has stopped being the warm-up act before training starts. It decides whether the GPUs a company just spent tens of millions of dollars on are doing work or sitting idle, waiting for text to show up in the right shape. Meta measured its own infrastructure and found 56% of GPU cycles stalled, waiting on training data that hadn't arrived yet. That's the majority of purchased compute doing nothing, on hardware billed by the hour whether it works or not.

The old assumption was that preprocessing happens once, upstream, and storage I/O is the real bottleneck, so a little latency in the pipeline doesn't matter much because training runs don't happen often enough for it to add up. That assumption is wrong now, and clinging to it produces Meta's number. Training runs chew through terabytes of preprocessed data per job, repeatedly, and the pipeline feeding them has to keep pace continuously. Wire a 512-GPU job to a data loader that serializes preprocessing across a handful of CPU threads, and GPU utilization sinks below 40% the moment the job hits a data-heavy phase, because the accelerators finish a batch faster than the CPUs can prepare the next one.

A rough consensus has formed across the major labs about what a pipeline needs to contain, even if no two implementations look identical. NVIDIA's documented stages are as good a spine as any: download and extract raw text into JSONL, run preliminary cleaning (Unicode normalization, language separation), apply heuristic quality filters, deduplicate at exact, fuzzy, and semantic levels, run model-based quality filtering that covers PII redaction and task decontamination, then blend and shuffle the surviving data from multiple sources into one training-ready set.

Naver's HyperCLOVA X 32B Think shows how those six stages compress into something a production team actually ships, in four steps. Collection and normalization comes first: lightweight cleaning that renders documents from wildly different formats into one consistent schema. Then quality scoring, where document-level structural and linguistic signals get computed once and stored as metadata, alongside PII detection and masking. Then filtering, which combines simple threshold rules with those learned quality scores rather than picking one over the other. Finally, serialization: sharding the surviving data into files built for streaming during training, not for sitting in cold storage.

That's the tidy version. In practice the pipeline looks more like a maze of one-off jobs with no shared abstraction connecting the stages, where each run spits out a disposable intermediate table nobody reuses next time. Common Crawl is the test case that exposes this best. The corpus runs to roughly 7 petabytes, delivered as WARC files that often run into the gigabytes each, and loading one of those into memory whole isn't an option. The only workable approach is streaming it through a WARC-aware library, such as warcio or the considerably faster FastWARC, one record at a time. Language identification and heuristic feature computation don't run inline, record by record, either. They run as a separate batch pass afterward, because doing that work synchronously while streaming would slow the extraction itself to a crawl.

Storage and I/O: how the data gets to the transformation layer at all

Before any transformation happens, the data has to physically reach the machine doing the transforming, and at petabyte scale that's its own engineering problem, separate from anything downstream. Meta stores exabytes in Tectonic, its distributed file system, but even Meta doesn't keep enough local disk sitting next to its GPUs to hold a petabyte-scale training set. Disaggregated storage is the only option that scales here, and the network connecting storage to compute has to move fast enough that it doesn't just become the bottleneck under a new name.

A few numbers calibrate what "fast enough" actually means. WEKApod, a storage system built for this kind of workload, is documented at over 720 GB/s of throughput and 18 million IOPS, with sub-150 microsecond latency, running from just 8 storage nodes to feed 768 H100 GPUs. On the drive side, PCIe Gen5 NVMe SSDs pushing past 14 GB/s of sequential read are becoming the norm for training-tier storage, with x16 configurations reaching 64 GB/s. NVMe-oF is what carries that near-local latency across a network, and that's what makes a disaggregated pool viable instead of just theoretically fast on paper.

Treating every byte in a petabyte corpus as equally urgent wastes money, and most teams do it anyway out of habit. A tiered approach puts frequently touched features on flash: Meta's RSC cluster reportedly leans on 46 petabytes of cache storage just to keep its GPUs fed continuously. Cold data, the kind that can tolerate a slow first touch, sits on capacity-optimized HDD or object storage like AWS S3 or GCS, and only gets pulled into a faster tier right before heavy processing needs it. For workloads that need synchronous, I/O-heavy access, parallel file systems like Lustre or GPFS do the job. Google Cloud's Managed Lustre integrates directly with object storage rather than sitting apart from it, and that's the direction this kind of system has to move in, since nobody wants to manage two separate storage stacks for one pipeline.

Why CPU-bound transformation fails at petabyte scale

Routing transformation through CPUs while GPUs stay reserved purely for training math made sense under three conditions: training runs were infrequent, storage I/O was the real chokepoint, and transformation itself was cheap next to the surrounding model computation. None of those conditions hold anymore. Clinging to that division of labor is what produces the 40%-utilization failure mode described above, where a 512-GPU cluster stuck behind serialized CPU preprocessing drops below 40% utilization the moment it hits a data-heavy phase. That's the exact pattern Meta's measurement captured, and it isn't an edge case.

The CPU-bound approach forced a specific compromise. Because transforming data on the fly was too slow, teams had to precompute and store derived features ahead of time, whether or not those turned out to be the features a given training run actually needed. GPU-native processing, through the NVIDIA RAPIDS ecosystem and cuDF specifically, breaks that compromise by putting transformation on the same memory hierarchy as training itself, cutting out the PCIe transfer overhead that used to make on-the-fly work impractical. Complex augmentations, tokenization at scale, even embedding generation, can now run inside the training loop without dragging utilization down.

That shift has a knock-on effect further down the stack. Feature stores exist largely to compensate for slow transformation: precomputing and caching derived features so training doesn't have to wait on them. But feature stores carry their own weight, schema management, freshness guarantees, backfill jobs, and the constant chore of keeping offline and online feature versions consistent with each other. Where GPU-native preprocessing computes those features fast enough to skip precomputation entirely, that operational overhead disappears with it. The trade is more compute spent at training time, which costs little for a team sitting on reserved GPU capacity that would otherwise sit idle. It's closer to free.

Deduplication at petabyte scale: why it is the hardest stage to get right

Skipping deduplication is a liability. It's a liability that appears later, when memorized training text leaks out of a deployed model. Deduplicating a training corpus cuts how often a model reproduces memorized text verbatim by a factor of ten, and lets the model reach equal or better accuracy in fewer training steps. Duplicate PII sitting in a pretraining corpus is also a documented path for privacy attacks against the finished model, which makes deduplication as much a security control as a quality one.

Two families of method exist here, and they solve different problems rather than competing for the same job. Exact or byte-level matching finds documents that are completely identical, or hash-identical, and it's cheap and auditable: run the hash, compare, done. On a sample of 13,197 WildChat prompts, exact matching alone caught 5.81% of duplicates. MinHash-LSH, an approximate method built around Jaccard similarity, catches near-duplicates rather than identical ones, and on that same corpus it caught 31.32%, nearly six times what exact matching found alone, though it costs far more compute to run. Known duplication rates across standard corpora make the case for bothering with approximate matching at all: C4 runs 6.7% duplicate content, RealNews 18.6%, ROOTS 21.67%. Exact matching alone would leave most of that sitting untouched.

Memory is the catch, and it's the reason teams reach for shortcuts here that they wouldn't accept anywhere else in the pipeline. A standard MinHash-LSH index built over 5 billion documents needs on the order of 23 terabytes of memory, putting it out of reach for most clusters that aren't purpose-built for this one task. Bloom filters offer a way out: swapping the LSH index for a Bloom filter approach cuts memory use down to a small fraction of that 23-terabyte baseline, with a real speedup on top. A job that needs a dedicated high-memory cluster costs a team far more than one that runs on hardware it already owns, and that gap is the whole argument for choosing Bloom filters over LSH once a corpus gets large enough to make the difference matter. Anyone still defaulting to plain LSH at petabyte scale is paying for memory they don't need to buy.

Quality filtering: choosing between heuristics and model-based classifiers at scale

Quality filtering runs on a spectrum between cheap-and-dumb and expensive-and-smart, and the mistake most teams make is treating that spectrum as a choice instead of a sequence. Heuristic filtering is at the cheap end: threshold rules on word or character ratios, line-length cutoffs, stop-word density, keyword blacklists. It runs at the character level (normalizing symbols and numbers, capping repeated characters, standardizing newlines, stripping non-standard Unicode), the line and paragraph level (removing HTML and JavaScript tags, cutting lines choked with special characters, dropping short or broken fragments), and the document level (discarding anything under a length floor, anything with excessive repetition, anything with too high a share of out-of-vocabulary words). None of it needs a model, and every rejection traces back to a specific rule someone can read and check by hand.

Model-based classifiers sit at the other end. A fine-tuned DistilBERT or BERT-small model, trained on positive examples (Wikipedia, well-edited books) against negative examples (low-quality web scrapes), catches domain-specific quality signals no threshold rule would ever notice. That power costs more compute per document, and it depends on labeled training data that heuristics never needed.

Running classifiers first, on the full corpus, is the wrong order, and it wastes most of the compute a team paid for. Heuristics need to go first, since they're cheap and cut corpus volume down fast, so that classifiers running second, on whatever survives, get their higher per-document cost amortized over a much smaller set. HyperCLOVA X 32B Think gives a working example of this in production: stage-specific corpora built by combining threshold heuristics with learned quality scores computed once and stored as document metadata. That metadata-first design means a score, once computed, gets reused across every later filtering pass without anyone reprocessing the underlying document again.

Open-source toolkits for distributed preprocessing: what each one is built for

Picking a toolkit here isn't a matter of comparing feature checklists against each other. It comes down to what infrastructure a team already has: GPU cluster access, a SLURM scheduler, or just a pile of CPU machines and a deadline. A team without spare GPUs gains nothing from a GPU-native toolkit, no matter how fast its benchmarks look on someone else's cluster.

NVIDIA NeMo Curator is built specifically for GPU-accelerated data curation ahead of LLM training, which puts it squarely in the RAPIDS and cuDF lineage described above. The 26.02 release introduced a Ray-based pipeline architecture spanning all four modalities the toolkit handles (text, image, video, and audio), unifying what used to be separate pipelines under one execution model. The 26.04 release followed with a Cosmos-Xenna 0.2.0 upgrade and a simplified Resources API, cutting down the manual tuning needed to get a pipeline scheduled correctly across a cluster's GPUs. For a team already standardized on NVIDIA hardware and the RAPIDS stack, NeMo Curator is built to sit directly on top of that investment, not bolted on beside it, and switching away from it means giving up the GPU-native shortcuts that make petabyte-scale transformation affordable.