Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI

Bogdan Gaza: Engineering & Scaling Synthetic Data for Trillion-Token LLM Pre-Training

Bogdan Gaza (CTO/co-founder of Duality) explains engineering and scaling lessons for synthetic data used in LLM pre-training at trillion-token scale. The discussion covers why synthetic data is needed, how Duality’s “Beyond Web” recipe works, and the infrastructure/throughput bottlenecks they encountered while generating ~12T synthetic tokens across web, math, and code.

---

Why Synthetic Data (the “Data Wall”)

• Scaling laws suggest a “data wall”: to get only linearly better model performance, you may need exponentially more compute and data.
• Internet data becomes insufficient at the scale required for “mythical model” capabilities.
• High-quality synthetic data can supplement non-web data that can’t be freely found online.

---

Product / Service Framing (Duality)

Duality is positioned as a data curation service—an “oil refinery” for pre-/core training datasets.

They help clients:

1. Identify high-quality data points within client datasets
2. Run those points through a synthetic data pipeline/recipe to produce much larger datasets

---

“Beyond Web” Synthetic Data Recipe (Key Concept + Reported Results)

Their approach emphasizes paraphrasing / rewording:

• Find high-quality documents from the client dataset
• Restructure/paraphrase them using a prompting/recipe approach

Reported empirical results

• Models trained with Beyond Web synthetic data can achieve smaller-model accuracy comparable to larger models
  • Example cited: 3B vs 8B on Nemo Trans Sent.
• For token efficiency, they claim performance comparable to known datasets using:
  • ~2.7× fewer tokens for one dataset comparison
  • ~5.3× fewer tokens for another (Relative to baselines including Nemo Trans Sent and Cosmopedia-style datasets.)

Note on result freshness

They state earlier results were “a bit outdated” and that the recipe has improved since initial publication.

---

Engineering Lessons from Scaling to ~12 Trillion Synthetic Tokens

They report a recent system run generating about 12T tokens spanning web/math/code, then break down key bottlenecks and fixes.

---

1) Metadata Bottlenecks When Datasets Become “Millions of Partitions”

At extreme scale, data stored in Parquet (or similar) can result in millions of partitions.

This creates bottlenecks:

• Loading/processing partitions requires many S3/metadata operations
• Ray metadata/query handling slows down
• GPU downtime increases because GPUs idle waiting on metadata operations

They estimated that building/populating metadata at this scale would take ~9–11 days, which they deemed unacceptable.

Fix: use batched S3 API requests instead of per-item operations:

• Increase batch size (noted support: about ~1000 requests per list)
• Reduced metadata processing time from ~11 days to ~2 hours

Lesson #1: Plan/engineer metadata management and state tracking early to enable real scaling.

---

2) GPU Instability (Crashes Waste Huge Amounts of Time Without Recovery)

Large-scale synthetic generation will encounter:

• Malfunctioning GPUs
• Node failures / partial outputs

Their pipeline uses Spark to output “hints” and paraphrase targets, then runs GPU “read/paraphrase” tasks.

Because there’s a 1-to-1 mapping between input partitions and output work, failures can erase late progress:

• If a partition takes hours and the system crashes near completion, you can lose a large chunk of work.

Fixes:

• Checkpointing (periodically save progress back to S3)
• Tune partition sizes to reduce the “lost work” window

They summarize needed properties:

• Failover / resumption
• Idempotence (checkpoint overwrite behavior is safe)
• Smooth recovery mechanism

Lesson: design for restartable/robust GPU tasks, not “batch jobs that assume perfect runs.”

---

3) Cross-Infrastructure Orchestration (CPU/GPU Scheduling Across Multiple Clusters)

They describe a multi-cluster setup:

• A central management/orchestration cluster
• Spark/Ray CPU-oriented compute clusters
• A separate H100 GPU cluster (via AWS Hyperpod/EKS on Hyperpod)

Main challenge:

• Scheduling tasks makes it easy to allocate CPU resources (Ray tasks)
• But harder to allocate GPU capacity in the GPU pool

Naive scheduling can create dead-ends:

• You may have CPUs but not GPUs (or vice versa)

Fix concept: treat orchestration as a containerization/scheduling problem:

• Use dedicated resource pools for each cluster and stage
• Perform atomic scheduling that accounts for both CPU and GPU needs

Lesson: implement scheduling/placement logic that jointly considers CPU+GPU constraints across clusters/providers.

---

4) Inference Optimization for Throughput (vLLM / SGLang Tuning)

During synthetic generation, they run batched inference using vLLM (and are also testing SGLang).

Key idea: throughput depends heavily on vLLM parameters and disciplined experimentation.

They report:

• ~40% throughput gain from tuning vLLM flags

They also mention considerations for correctness/performance:

• Batch sizing
• Speculative decoding configuration
• Systematic parameter grid search / tuning focus for vLLM

Lesson: build a parameter-optimization toolkit and iterate on inference settings.

---

Reported Scaling Outcomes (Quantitative)

• End of 2024: only ~30B synthetic tokens for limited runs on their cluster
• After late-2024 scaling improvements, they can run jobs generating about:
  • ~7T tokens for web
  • ~5T tokens for many customers
• They continue working on better synthetic data recipes and plan to expand domain coverage (web vs multilingual vs math vs software vs legal, etc.).

---

Main Speakers / Sources

• Bogdan Gaza (Co-founder & CTO, Duality)