Container orchestration: Best Practices

Container orchestration has become the de facto standard for deploying AI and machine learning workloads at scale. What began as a tool for managing web services has evolved into a sophisticated platform for running GPU-accelerated training jobs, auto-scaling inference endpoints, and managing the full ML lifecycle. A 2024 CNCF Survey found that 96% of organizations are either using or evaluating Kubernetes, and Datadog's 2024 Container Report shows that 65% of Kubernetes clusters now include GPU-enabled nodes, up from 35% in 2022.

Yet the gap between running Kubernetes and running it well for AI workloads remains significant. The organizations that close this gap treat orchestration not as infrastructure plumbing but as a strategic capability, one that directly determines model deployment velocity, GPU cost efficiency, and competitive time-to-market.

Kubernetes as the AI/ML Platform Foundation

Kubernetes has established itself as the orchestration layer for AI workloads, but running ML effectively requires purpose-built extensions that go well beyond a vanilla cluster.

The most fundamental of these extensions is GPU scheduling and management. The NVIDIA GPU Operator automates GPU driver installation, container toolkit setup, and device plugin deployment across the cluster. When combined with NVIDIA's Multi-Process Service (MPS), multiple inference workloads can share a single GPU, improving utilization from a typical 30-40% to 70-80%. For most organizations, this single optimization represents the highest-leverage infrastructure investment available.

On top of GPU scheduling, Kubeflow provides Kubernetes-native Custom Resource Definitions for training jobs (TFJob, PyTorchJob, MPIJob), hyperparameter tuning (Katib), and model serving (KServe). As of 2024, Kubeflow has accumulated over 13,000 GitHub stars and is deployed at organizations including Google, Bloomberg, and Spotify, a testament to its maturity as a production-grade ML platform layer.

Kubernetes 1.30 introduced stable support for Dynamic Resource Allocation (DRA), a capability that enables fine-grained allocation of hardware accelerators. DRA allows pods to request specific GPU models, memory quantities, and interconnect topologies, moving beyond the binary "one GPU per pod" model that constrained earlier versions. This granularity matters as organizations run increasingly diverse GPU fleets spanning multiple hardware generations.

Finally, scheduling extensions like the Volcano batch scheduler address a critical gap in native Kubernetes scheduling. Volcano supports gang scheduling, where all pods in a distributed training job start simultaneously or none do, along with fair-share queuing and preemption policies tailored to ML workloads. Gang scheduling is essential for distributed training because partial pod launches waste expensive GPU resources while the job waits for the remaining pods to be placed.

Scaling Inference Workloads

Production AI inference has distinct scaling characteristics that differ fundamentally from traditional web services. The standard CPU-based Horizontal Pod Autoscaler (HPA) is inadequate because inference bottlenecks manifest in GPU utilization, request queue depth, and tail latency rather than CPU load. Configuring HPA to scale on these custom metrics, exposed through Prometheus-based adapters, is the first step toward responsive inference scaling.

For asynchronous inference pipelines, KEDA (Kubernetes Event Driven Autoscaling) offers a more natural scaling model. KEDA scales based on message queue depth across Kafka, RabbitMQ, or SQS, making it ideal for batch inference workloads where throughput matters more than latency. Critically, KEDA can scale deployments to zero during idle periods, eliminating GPU costs entirely for intermittent workloads. For organizations running dozens of specialized models with sporadic traffic, this capability alone can justify the implementation effort.

Workloads with predictable traffic patterns benefit from predictive autoscaling, which pre-provisions capacity before demand spikes rather than reacting to them. Both Google's Autopilot for GKE and Amazon's Karpenter support schedule-based scaling policies, enabling organizations to align GPU capacity with known usage windows such as business-hours inference or batch processing cycles.

Perhaps the most impactful scaling strategy is multi-model serving, where multiple models share the same infrastructure. NVIDIA Triton Inference Server supports dynamic batching across models, and Seldon Core enables A/B testing and canary deployments for model versions on shared compute. Organizations adopting multi-model serving report 40-60% infrastructure cost reduction compared to dedicated-instance deployments, a savings magnitude that compounds rapidly as model counts grow.

GPU Resource Management

GPUs are the most expensive and constrained resource in AI infrastructure. A single NVIDIA H100 commands thousands of dollars per month in cloud pricing, making effective GPU management not merely an operational concern but a financial imperative.

NVIDIA's time-slicing feature allows multiple containers to share a single GPU through temporal multiplexing. While time-slicing does not provide memory isolation, it significantly improves utilization for inference workloads that do not need full GPU memory. A 2024 benchmark by NVIDIA showed that time-slicing supports up to 7 inference workloads per GPU with acceptable latency overhead.

For environments requiring stronger isolation, Multi-Instance GPU (MIG) technology on NVIDIA A100 and H100 GPUs partitions a single GPU into up to 7 isolated instances, each with dedicated compute, memory, and memory bandwidth. MIG provides hardware-level isolation, making it suitable for multi-tenant environments where workload interference must be eliminated entirely.

At the cluster level, implementing namespace-level GPU quotas prevents resource monopolization. The recommended practice is to create separate GPU pools for training (requiring full GPUs or multi-GPU allocations) and inference (using time-slicing or MIG). This separation prevents large training jobs from starving latency-sensitive inference endpoints, a failure mode that can cascade into customer-facing service degradation.

Cloud GPU spot pricing offers 60-80% discounts compared to on-demand pricing, representing an enormous cost optimization lever for fault-tolerant workloads. Using Kubernetes node affinity and tolerations, organizations can schedule interruptible training jobs on spot instances while keeping inference on stable on-demand nodes. Karpenter can automatically provision spot instances when training jobs are submitted, closing the loop between workload submission and cost-optimized placement.

Container Image Optimization for ML

ML container images are notoriously large. A typical PyTorch GPU image exceeds 10GB, and without deliberate optimization, image pulls become a deployment bottleneck that delays model updates and slows incident recovery.

Multi-stage builds separate build-time dependencies (compilers, development headers) from runtime requirements. A well-optimized PyTorch inference image can be reduced from 10GB+ to 3-4GB, cutting image pull time from 5 minutes to under 90 seconds on standard networks. The compounding effect on deployment frequency is substantial: teams that optimize images deploy more often simply because the friction of each deployment is lower.

Dockerfile instruction ordering determines layer cache hit rates during CI/CD. The optimal sequence moves from least to most frequently changing: base OS, CUDA drivers, ML framework, application dependencies, model weights, and finally application code. This ordering ensures that the expensive base layers are rebuilt only when the underlying platform changes, not on every code commit.

Model weights should never be baked into container images. Storing models in object storage (S3, GCS, MinIO) and pulling them at container startup using init containers decouples model updates from image builds. This separation enables model versioning without image rebuilds, a critical capability for organizations iterating rapidly on model quality.

Google's Distroless images and NVIDIA's CUDA minimal images eliminate unnecessary OS packages, reducing both attack surface and image size. The NVIDIA CUDA 12.3 runtime image is 2.3GB compared to the full development image at 8.7GB, a reduction that translates directly into faster startup times and lower storage costs across the cluster.

Networking and Storage for AI Workloads

Distributed training and high-throughput inference place unique demands on networking and storage that standard Kubernetes configurations cannot satisfy.

For distributed training across multiple nodes, RDMA (Remote Direct Memory Access) networking bypasses the kernel network stack, reducing inter-node communication latency by 10-50x. NVIDIA's GPUDirect RDMA takes this further by enabling GPU-to-GPU communication across nodes without CPU involvement. InfiniBand interconnects deliver 400Gbps bandwidth per port, and for large-scale training runs spanning dozens of nodes, this interconnect performance often determines whether a job completes in hours or days.

Training jobs reading large datasets require high-throughput parallel file systems that can saturate GPU compute capacity. WekaFS, GPFS, and Lustre provide the throughput, 100+ GB/s in aggregate, needed for data-intensive training. For Kubernetes-native storage, Rook-Ceph with NVMe backing provides 10+ GB/s throughput while maintaining the operational simplicity of a Kubernetes-managed storage layer.

Long-running training jobs must checkpoint regularly to survive preemptions and failures. Persistent volumes backed by NVMe storage provide the fast write speeds necessary for efficient checkpointing. A checkpoint-aware scheduler can preempt training jobs gracefully, allowing them to save state before eviction, turning what would otherwise be lost compute hours into recoverable progress.

Between object storage and compute, caching layers such as Alluxio or JuiceFS reduce data loading bottlenecks that would otherwise leave GPUs idle while waiting for training data. Alluxio's documentation reports that its distributed cache can improve training data throughput by 3-10x compared to direct object storage access, an improvement that translates directly into faster training iteration cycles and lower per-experiment GPU costs.

Observability and Cost Management

AI workloads on Kubernetes require specialized observability that extends well beyond standard CPU and memory metrics. Without GPU-aware monitoring, organizations operate blind to their most expensive resources.

NVIDIA DCGM (Data Center GPU Manager) exports per-GPU metrics including utilization, memory usage, temperature, power draw, and error counts. Feeding these metrics into Prometheus and Grafana provides real-time dashboards and alerts that surface underutilization and hardware degradation before they impact training jobs or inference quality.

Experiment tracking tools such as MLflow, Weights & Biases, and Neptune should be integrated with Kubernetes metadata so that every training job carries experiment IDs, model versions, and cost attribution labels. This integration enables per-experiment cost analysis, a capability that transforms GPU spending from an opaque infrastructure line item into a measurable input tied to specific model improvements.

Cost allocation by team and project is where observability meets financial governance. Kubernetes labels and namespaces, combined with cost management tools such as Kubecost, OpenCost, or CloudHealth, attribute GPU spending to specific teams, projects, and experiments. According to Kubecost's published benchmarks, organizations implementing Kubernetes cost management reduce cloud spending by an average of 30%, primarily by identifying and eliminating idle or oversized GPU allocations.

Right-sizing recommendations complete the observability picture. Monitoring actual GPU utilization versus requested resources reveals persistent over-provisioning that accumulates into significant waste. While Kubernetes VPA (Vertical Pod Autoscaler) can recommend resource adjustments for CPU and memory, GPU workloads require manual review because VPA does not natively understand GPU memory requirements. Establishing a regular right-sizing review cadence, monthly at minimum, prevents cost drift as workload patterns evolve.

Security and Isolation

Multi-tenant AI clusters require robust security boundaries that account for the unique threat surface of GPU workloads and model assets.

Kubernetes NetworkPolicies, enforced through Calico or Cilium, isolate training and inference namespaces at the network level. Proper segmentation prevents training jobs from accessing inference model endpoints and vice versa, limiting the blast radius of a compromised workload. This isolation is particularly important in environments where multiple teams or business units share GPU infrastructure.

Pod security standards should be enforced at the restricted level to prevent privileged container access. GPU workloads require specific device plugin permissions but should not run as root. Configuring securityContext with readOnlyRootFilesystem and explicit GPU device allowlists provides the minimum privilege necessary for GPU access without exposing the broader node.

Supply chain security for container images is non-negotiable in production ML environments. Sigstore Cosign provides image signing, and admission controllers such as Kyverno or OPA Gatekeeper enforce signature verification at deployment time. This chain of trust prevents deployment of unauthorized or tampered images, a risk that grows as organizations adopt more third-party model containers.

Secrets management deserves particular attention in ML environments where API keys, model registry credentials, and cloud provider tokens proliferate across pipelines. External secrets operators (External Secrets Operator, Sealed Secrets) inject credentials at runtime without embedding them in container images or Kubernetes ConfigMaps. Baking credentials into images is one of the most common and most dangerous anti-patterns in containerized ML, and eliminating it should be a foundational security requirement.

Container orchestration for AI workloads is a rapidly evolving discipline, but the principles that separate high-performing organizations are already clear. The teams extracting the most value combine Kubernetes-native tooling with ML-specific extensions, invest heavily in GPU resource management, and treat observability and cost management as first-class concerns rather than afterthoughts. As models grow larger and inference demand increases, the efficiency of the container orchestration layer directly determines both time-to-market and total cost of ownership.