Most ML teams budget for GPU hours. That is the visible line item — the one in the cloud invoice, the one in the capital plan. But in production deep learning infrastructure, GPU compute is often only 50–60% of total cost. The other 40–50% hides in places that rarely appear in budget presentations: storage I/O, network egress, engineering time, incident response, and the compounding overhead of poorly coordinated tooling. This article names and quantifies the costs that most teams discover only after they have already overspent.
1. Storage: The Silent Budget Killer
Training large models generates enormous amounts of data: datasets (often terabytes), training checkpoints (hundreds of GB per checkpoint, dozens of checkpoints per run), experiment logs, activation dumps, profiling traces, and model artifacts. Few teams cost-account for this when setting up their ML infrastructure.
On AWS, S3 storage costs $0.023/GB/month, but that is only the beginning. Data requests cost extra ($0.0004 per 1,000 GET requests — a training run reading millions of dataset files makes millions of GET requests). Data transfer out of S3 to EC2 within the same region is free; out to the internet or to another region costs $0.09/GB. A single 100TB dataset living in S3, read daily by training jobs, can cost $3,000–5,000/month in storage plus request fees — easily exceeding the compute cost of small-scale training.
Checkpoint bloat is the most common storage cost surprise. A typical LLM training run creates a checkpoint every 1,000 steps. With 300GB of model state per checkpoint and 100 checkpoints per run, that is 30TB of checkpoint storage per training run. At $0.023/GB/month, that is $690/month just to keep one run's checkpoints. Most teams accumulate many runs across months; the storage bill compounds silently in the background.
The fix: implement a checkpoint retention policy. Keep only the last N checkpoints plus selected milestone checkpoints. Use differential checkpointing to store only changed parameters between consecutive checkpoints. Use S3 Glacier or Azure Archive for long-term storage of final model weights. Incremental delta checkpointing (storing only weight differences) can reduce checkpoint storage by 70–80% for fine-tuning workloads where model weights change gradually.
2. Network Egress: The Invisible Tax
Cloud providers charge for data moving out of their network. This is well-known in theory but consistently underestimated in practice. Common egress scenarios in ML infrastructure:
- Dataset transfers: Downloading a 10TB dataset from S3 to an EC2 training instance in a different region costs $900 in egress fees. Same-region egress: $0.
- Model serving: A deployed model receiving 1M inference requests/day, each returning a 10KB response, generates 10GB/day of outbound traffic — $0.90/day, $27/month. At scale (100M requests/day), $2,700/month.
- Experiment tracking: Many experiment tracking tools (Weights & Biases, MLflow hosted) send metrics, gradients, and activations out of your VPC. A team running 50 concurrent experiments with detailed logging can generate 100GB+/day of outbound tracking data without realizing it.
- Cross-region replication: Teams that replicate training data or model checkpoints across regions for redundancy pay egress on every byte replicated.
Mitigation: co-locate training instances with data storage in the same region and availability zone. Use VPC endpoints for S3 and other managed services to avoid internet egress entirely. Audit your egress costs monthly — cloud dashboards make it easy to overlook until the bill arrives.
3. Engineering Time: The Largest Hidden Cost
A senior ML infrastructure engineer costs $200,000–$350,000/year fully-loaded in major tech hubs. A team of four infrastructure engineers costs over $1 million/year. What are they actually doing?
In a typical ML platform team without mature tooling, engineering time distributes roughly as follows: 25% on cluster provisioning and maintenance (replacing failed nodes, updating CUDA drivers, managing capacity reservations), 20% on incident response (debugging hung training jobs, tracking down OOM crashes, investigating corrupted checkpoints), 20% on tooling development (building internal dashboards, job submission interfaces, monitoring systems), 15% on cost optimization (identifying and eliminating waste in the GPU fleet), and only 20% on actual performance engineering that directly improves model training.
The opportunity cost is significant. Every hour an ML infrastructure engineer spends on routine cluster maintenance is an hour not spent on throughput improvements that compound across all future training runs. Investing in automation and managed infrastructure often delivers a 3–5x multiplier on engineering productivity.
4. Failed Runs: The Unbudgeted Compute Tax
Training runs fail. Hardware fails, code has bugs, data pipelines corrupt, and distributed jobs encounter deadlocks. In production LLM training at scale, industry data suggests 15–30% of GPU-hours are consumed by runs that do not complete successfully or produce usable models. This compute tax is rarely reflected in ML infrastructure budgets.
The compounding effect: a 30% failure rate means you need to provision 43% more compute to achieve your target model delivery rate (1/0.7 = 1.43). A $1M annual GPU budget effectively becomes $700K of productive training and $300K of waste. At enterprise scale, this represents millions in wasted compute annually.
Sources of failure: hardware faults (10–15% of failures), OOM crashes from mis-estimated memory requirements (20–25%), data pipeline failures (15–20%), numerical instability (gradient explosion/NaN) (10–15%), and infrastructure issues like storage timeouts, job scheduler bugs, and network partitions (remaining 25%). Each category requires a different mitigation strategy.
5. Idle Reserved Capacity: Paying for What You Don't Use
Teams that purchase reserved instances or dedicated GPU capacity often significantly underutilize that capacity. A common scenario: a team reserves 100 A100s for 12 months to get the reserved discount. During the contract, they use an average of 60–70 GPUs, leaving 30–40 GPUs sitting idle but being paid for. The per-hour savings from the reservation are entirely offset by the utilization loss.
The utilization math: on-demand A100 pricing of $3.00/hr vs. 1-year reserved at $1.80/hr (40% discount) looks compelling. But if reserved utilization drops to 65%, the effective per-hour rate for useful compute is $1.80/0.65 = $2.77/hr — barely below on-demand pricing, and with none of the flexibility. The reservation discount only pays off above ~75% utilization.
The solution is capacity planning sophistication: model your compute demand with uncertainty bounds, reserve only the base load you can confidently fill, and cover demand peaks with on-demand or spot instances. Most ML teams reserve too much and pay the idle penalty.
6. Tooling Fragmentation: The Integration Tax
A typical ML platform assembled from best-of-breed open-source tools includes: SLURM or Kubernetes for job scheduling, Weights & Biases or MLflow for experiment tracking, Prometheus + Grafana for infrastructure monitoring, custom scripts for data pipelines, a separate checkpointing library, a separate fault tolerance layer, and possibly multiple job submission CLIs. Each tool requires maintenance, upgrades, and integration with the others.
Integration failures between these layers are a chronic source of incidents and engineering time. A SLURM upgrade breaks the custom Python job submission wrapper. An MLflow upgrade changes the artifact API, breaking the checkpointing library's integration. A Prometheus exporter fails silently, blinding the team to GPU utilization problems until a bill arrives.
The integration tax is invisible in tool-level cost accounting because each individual tool appears "free" (open source) or cheap (SaaS). The true cost — integration engineering, incident response, and the cognitive overhead of operating a dozen loosely coupled systems — is absorbed by the engineering team's time budget.
Key Takeaways
- GPU compute is only 50–60% of true ML infrastructure cost. Budget for storage, egress, engineering time, and failed runs explicitly.
- Checkpoint bloat silently inflates storage bills. Implement retention policies and differential checkpointing from day one.
- Failed runs consume 15–30% of GPU budget. Fault-tolerant training infrastructure is not optional — it is a cost control measure.
- Reserved capacity discounts only work above 75% utilization. Most teams over-reserve and lose the discount benefit to idle capacity.
- Engineering time is often the largest single cost center. Tooling fragmentation multiplies overhead; integrated platforms reduce it.
Conclusion
The teams that control their ML infrastructure costs most effectively are the ones that measure everything — not just the GPU invoice. Comprehensive cost accounting across storage, egress, engineering time, and failed runs gives a complete picture and reveals the highest-leverage optimization opportunities. Frequently, the biggest wins come not from squeezing GPU utilization but from eliminating the structural waste that hides in the surrounding infrastructure.
Deepiix's platform gives ML infrastructure teams complete visibility into all cost components, not just compute. See our solutions for how we approach cost efficiency as a system property rather than a single metric.