Kubernetes has become the default container orchestration platform for most cloud-native applications. It is natural — and almost universal — for ML platform teams to ask: should we run our training workloads on Kubernetes too? The answer is nuanced. Kubernetes offers real advantages for certain ML workloads, particularly inference and data processing pipelines. But for large-scale distributed training, vanilla Kubernetes has significant limitations that require substantial additional investment to overcome. This article gives an honest assessment of both sides, and outlines when alternatives like SLURM, Ray, and Volcano make more sense.
1. Where Kubernetes Genuinely Helps ML Teams
The case for Kubernetes in ML infrastructure is real, and stems from the same properties that make it valuable for web services:
Infrastructure unification. If your organization already runs Kubernetes for application workloads, operating ML training and inference on the same cluster eliminates a second operations team, a second monitoring stack, a second deployment pipeline. The operational cost of running two fundamentally different infrastructure stacks is often underestimated. Kubernetes-as-common-platform has genuine value for organizations where platform team headcount is constrained.
Container-based reproducibility. Kubernetes' container model — immutable images with explicit dependencies — solves the "works on my machine" problem for ML experiments. Every experiment runs in an identical software environment, version-locked and reproducible months later. This is more valuable for research teams running many experiments than for production training teams running a small number of critical runs.
Inference serving. Kubernetes is genuinely good for ML inference workloads. Horizontal pod autoscaling, traffic-based scale-to-zero, rolling deployments, and canary releases are all first-class Kubernetes features that are directly applicable to model serving. For inference, Kubernetes' strengths align well with the workload characteristics — stateless, horizontally scalable, latency-sensitive request routing.
Hybrid data processing. Data preprocessing pipelines that mix GPU and CPU workloads — tokenization, image augmentation, feature extraction — map naturally to Kubernetes where heterogeneous node pools are straightforward to configure and mix.
2. Where Kubernetes Struggles with Training Workloads
Despite these advantages, vanilla Kubernetes has architectural limitations that create real problems for large-scale distributed training:
Gang scheduling. Kubernetes' default scheduler places pods independently, without awareness that a distributed training job requires all its pods to start simultaneously. In a loaded cluster, this leads to "partial placement" deadlocks: 60 out of 64 required pods are running, the remaining 4 pods are queued but blocked by other workloads, and the 60 running pods idle indefinitely waiting for their peers. Without gang scheduling, cluster utilization can be catastrophically low when multiple distributed jobs compete for resources.
The fix — Volcano scheduler, MCAD (Multi-Cluster Application Dispatcher), or the Kubernetes scheduler framework's Co-Scheduling plugin — adds gang-aware scheduling. But each of these introduces additional operational complexity and requires careful configuration to avoid its own failure modes.
Job priority and preemption. Kubernetes supports pod priority and preemption, but the default preemption semantics were designed for web services (kill a lower-priority pod to make room for a higher-priority pod). For training workloads, preempting one process in a distributed job does not free a useful amount of resources — it kills the entire job (since the remaining processes are blocked waiting for the preempted one). Implementing training-aware preemption requires custom logic not available in stock Kubernetes.
Network performance. Kubernetes networking abstractions (CNI plugins, kube-proxy, service meshes) add latency and reduce bandwidth between pods compared to direct MPI-over-InfiniBand communication. For AllReduce-intensive training jobs where inter-node communication is on the critical path, the Kubernetes networking layer can cost 10–30% of training throughput. GPUDirect RDMA, which bypasses the CPU and network stack entirely, is difficult to enable correctly in Kubernetes without privileged container configurations that compromise security posture.
Storage performance. Kubernetes Persistent Volumes (PVs) add an abstraction layer above the underlying storage system. High-throughput dataset access — reading millions of files per training step — can be significantly slower through PV abstractions than direct POSIX access to a local NVMe or NFS mount. PVC provisioning delays also add startup latency for training jobs.
3. SLURM: The HPC Standard
SLURM (Simple Linux Utility for Resource Management) is the dominant workload manager in high-performance computing environments, and for good reason. It was designed from the ground up for the exact problem of scheduling tightly-coupled parallel jobs on shared compute clusters — precisely the workload profile of distributed deep learning training.
SLURM's strengths for training: native gang scheduling (jobs wait until all required nodes are simultaneously available), MPI-aware process launch with direct InfiniBand binding, priority queues with fair-share scheduling across teams, and decades of operational hardening at the world's largest compute facilities.
SLURM's weaknesses in modern ML contexts: it predates the container era and lacks native container support (Singularity/Apptainer is the standard workaround, adding complexity). It has no native concept of GPU topology-awareness — a GPU job might be placed on nodes whose GPUs are connected via slow PCIe rather than NVLink, silently degrading performance. And SLURM's job definition language (sbatch scripts) is less flexible than Kubernetes YAML for complex multi-stage ML pipelines.
The practical verdict: SLURM is the right choice for organizations that already have HPC infrastructure and a team with SLURM expertise, particularly for large-scale training jobs where communication performance is critical. For cloud-native teams starting from scratch, Kubernetes with Volcano often has a lower total onboarding cost.
4. Ray: Python-Native Distributed Computing
Ray is an open-source distributed computing framework from Anyscale designed specifically for ML workloads. Ray Cluster provides a Python-native API for distributed execution; Ray Train adds distributed training abstractions for PyTorch and TensorFlow; Ray Tune handles hyperparameter search at scale.
Ray's key advantages: it is Python-native, so ML researchers can write distributed code that looks like regular Python with minimal boilerplate. Ray handles worker failures and restarts automatically. Ray's task graph model is more flexible than Kubernetes pod-based scheduling for dynamic ML workflows where the compute graph changes based on intermediate results.
Ray's limitations: it adds another process management layer on top of whatever scheduler runs the Ray cluster itself (typically Kubernetes or SLURM). For pure large-scale training, Ray Train's performance lags behind hand-tuned NCCL-based PyTorch DDP by 5–15% due to its abstraction overhead. And Ray's operational complexity — managing the Ray head node, worker nodes, object store, and GCS — is non-trivial at production scale.
5. The Hybrid Architecture Pattern
The most common production ML infrastructure pattern at established organizations is hybrid: Kubernetes for inference serving and data processing pipelines, plus SLURM or a custom training scheduler for large-scale distributed training. This captures the strengths of both systems while avoiding their weaknesses in the other's domain.
The integration challenge: experiment metadata, model artifacts, and data pipelines need to flow between the Kubernetes layer and the training layer. A well-designed MLflow or custom metadata store serves as the integration point — Kubernetes-based data preprocessing jobs write processed datasets, SLURM training jobs read them, and Kubernetes-based inference services consume trained models.
See how the Deepiix platform abstracts over these infrastructure layers, providing a unified interface for training, experiment tracking, and deployment without requiring teams to operate two separate scheduler ecosystems.
Key Takeaways
- Kubernetes is excellent for ML inference, mediocre for large-scale training. The gang scheduling and networking limitations are real and require significant investment to overcome.
- Volcano or MCAD is mandatory if you use Kubernetes for training. Vanilla Kubernetes scheduling deadlocks are a common failure mode for distributed training teams.
- SLURM remains the gold standard for communication-intensive training workloads. Its HPC heritage gives it real advantages for AllReduce-heavy large-model training.
- Ray is compelling for research teams prioritizing flexibility. For production training at scale, its abstraction overhead is a concern.
- Hybrid architectures are the most common production pattern. Kubernetes for serving + SLURM (or equivalent) for training captures the best of both worlds.
Conclusion
There is no universally correct answer to "Kubernetes or not for ML?" The right choice depends on your team's existing expertise, workload profile, scale requirements, and operational constraints. What is clear is that vanilla Kubernetes — without training-specific extensions — is insufficient for serious large-scale distributed training. Whether the answer is Volcano on Kubernetes, SLURM alongside Kubernetes, or a purpose-built ML platform that abstracts the scheduling layer entirely, the default configuration will underperform your requirements.
Deepiix has built an orchestration layer specifically designed for deep learning training workloads, incorporating gang scheduling, topology-aware placement, and elastic recovery without requiring teams to maintain two separate infrastructure ecosystems. Get in touch to discuss your infrastructure architecture.