What does it actually cost to train a large language model? The figures thrown around in press coverage — "$100 million for GPT-4," "billions for frontier models" — are real but apply to a narrow category of labs operating at the absolute frontier. For the vast majority of organizations training serious production models in the 7B–70B parameter range, the economics are very different, very tractable, and very sensitive to infrastructure efficiency. This article breaks down the actual costs across all budget categories and shows where the optimization leverage is greatest.
1. Establishing the Baseline: Compute Requirements
Training compute requirements follow a well-established scaling relationship: for a transformer model with N parameters trained on D tokens, the approximate number of FLOPs required is 6 × N × D (the Chinchilla compute formula). This is the starting point for all cost estimation.
Concrete examples using this formula:
- 7B model, 140B tokens (Chinchilla-optimal): 6 × 7×10⁹ × 1.4×10¹¹ = 5.88 × 10²¹ FLOPs
- 13B model, 260B tokens: 6 × 1.3×10¹⁰ × 2.6×10¹¹ ≈ 2.03 × 10²² FLOPs
- 70B model, 1.4T tokens (Llama-3 scale): 6 × 7×10¹⁰ × 1.4×10¹² ≈ 5.88 × 10²³ FLOPs
Converting FLOPs to GPU-hours: an A100 80GB delivers approximately 312 TFLOPS in BF16. In practice, with communication overhead and typical 45–55% hardware utilization (model FLOP utilization, MFU), achievable throughput is 140–170 TFLOPS per GPU-hour.
- 7B model, Chinchilla-optimal: ~9,600–11,700 A100-hours
- 70B model, Llama-3 scale (1.4T tokens): ~960,000–1,170,000 A100-hours
At cloud spot pricing of $1.50–$2.00/A100-hour, the compute cost estimates are:
- 7B model: ~$14,400–$23,400 in compute
- 70B model: ~$1.44M–$2.34M in compute
2. Storage Costs
Storage costs are the most underestimated budget line for large-model training. For a 70B model training run, storage requirements include:
- Training dataset (1.4T tokens, tokenized): ~2.8TB (at 2 bytes per token). Stored in cloud object storage: ~$65/month.
- Training checkpoints: A BF16 70B model with ZeRO-1 optimizer states requires ~280GB per checkpoint. At 200 checkpoints over a 30-day run: 56TB of checkpoint storage. Peak storage: ~$1,300/month; archived long-term: ~$420/month on S3 Glacier.
- Profiling traces and experiment logs: ~100GB–500GB per run, depending on logging verbosity. Negligible cost but significant data management overhead.
Total storage costs for a 30-day 70B training run: $1,500–$3,000. Comparable to 1–3 hours of compute on the same cluster — often dismissed as rounding error but worth managing systematically to avoid compounding across many runs.
3. Networking and Egress Costs
For on-premise clusters, networking is a capital cost embedded in the cluster build (InfiniBand switches run $50,000–$500,000+ depending on scale). For cloud training, networking is an operational expense:
- Intra-region data transfer (EC2 to EC2): Free for most inter-instance communication within the same Availability Zone. Cross-AZ: $0.01/GB each way.
- S3 request costs: Reading a 2.8TB dataset with 100-byte average object size (individual tokenized sequences) means ~28 billion objects — at $0.0004/1,000 requests, that is $11,200 in S3 request fees alone. Using larger sharded files (WebDataset format) reduces object count by 10,000x, dropping S3 request costs to near-zero.
- Data egress (for multi-cloud or hybrid setups): $0.08–$0.09/GB out of AWS. Transferring 2.8TB of training data between clouds: ~$252 per transfer.
4. Engineering Labor Costs
Engineering labor is the most significant but least commonly cited cost component. Training a 70B model from scratch is not a set-and-forget operation. It requires:
- Infrastructure setup: 1–2 weeks of senior ML infrastructure engineering to configure the cluster, tune NCCL, validate networking, and establish monitoring. At $200/hour (fully-loaded), that is $16,000–$32,000.
- Training monitoring and intervention: Someone needs to watch the loss curves, respond to node failures, validate checkpoints, and adjust hyperparameters. For a 30-day run, expect 1–2 FTE-days per week of ongoing oversight: $20,000–$40,000.
- Data preparation: Curating, deduplicating, filtering, and tokenizing 1.4T tokens is a non-trivial engineering project. Depending on data sources, this can range from 2 weeks (existing curated dataset) to 3+ months (novel data collection from scratch).
- Evaluation and iteration: Training is never a single run. Budget for 3–5 hyperparameter-tuning runs (shorter, lower cost, but real compute) before committing to a full production run.
Realistic labor cost for a first production 70B training run from a team without prior experience: $150,000–$300,000 in engineering time. For a team with mature infrastructure and tooling: $50,000–$100,000.
5. Infrastructure Efficiency: The Multiplier on Everything
The cost figures above assume reasonable infrastructure efficiency (45–55% MFU). Teams operating at 30% MFU (common for first-time large-model training) pay 50% more in compute for identical output. The efficiency gap compounds:
- A 70B run at 30% MFU costs ~$3.5M–$4.7M in compute (vs. $1.44M–$2.34M at 50% MFU)
- Failed runs at 25% failure rate add another 33% overhead
- Idle reserved capacity at 70% utilization wastes 30% of capacity budget
Combined: a poorly optimized infrastructure doubles or triples effective training cost compared to a well-optimized one. The ROI on infrastructure investment is clear: $500K of engineering to improve MFU from 30% to 50% pays back in the first production run if compute budget exceeds ~$1M.
6. Total Cost of Ownership Model
Putting it all together for a representative 70B model, 1.4T token training run:
- Compute (cloud, spot): $1.5M–$2.5M
- Storage: $5,000–$10,000
- Networking/egress: $2,000–$5,000
- Engineering labor (experienced team): $75,000–$150,000
- Infrastructure efficiency overhead (failed runs, idle): $200,000–$500,000
- Total: $1.8M–$3.2M for an experienced, well-instrumented team
For a first-time team without mature infrastructure, the upper end can exceed $5M due to efficiency losses, failed runs, and engineering overhead. This is why the ROI of investing in purpose-built ML infrastructure platforms becomes compelling quickly at this scale.
Key Takeaways
- Compute is the largest line item, but not the only one. Engineering labor and efficiency overhead often exceed storage and networking combined.
- MFU is the most important efficiency metric. A 20-percentage-point improvement in MFU reduces compute cost by 30–40%.
- Storage is cheap per GB but expensive at scale without retention policies. Manage checkpoints actively; they compound silently.
- S3 request patterns matter. Naive per-sample file storage can cost more in requests than storage fees; use sharded binary formats.
- Infrastructure investment pays for itself within one large training run. At $1.5M+ compute budgets, even $200K of infrastructure improvement investment yields immediate positive ROI.
Conclusion
Training large models is expensive, but the cost structure is far more controllable than headlines suggest. The dominant variables are infrastructure efficiency (MFU), failed run rates, and engineering overhead — all addressable with the right tooling and practices. Teams that treat infrastructure efficiency as a product discipline rather than a best-effort side project consistently operate at 30–50% lower cost per trained model than teams that do not.
The Deepiix platform is designed specifically to optimize across these cost dimensions simultaneously — improving MFU, reducing failed runs, and automating the monitoring overhead that would otherwise consume engineering time. Contact us to understand what our infrastructure economics analysis looks like for your specific model scale.