AI & GPU Workloads
20 articles from official documentation
Navigating AI Conformance in Kubernetes: From WGs to SIGs
AI workloads are evolving, and Kubernetes is stepping up to meet the challenge with the AI Conformance initiative. This program introduces a tiered model for standardizing AI infrastructure requirements, focusing on dynamic resource allocation. Dive in to understand how this impacts your deployments.
- →Understand AI Conformance as a response to the evolution of AI workloads on Kubernetes.
- →Utilize the tiered graduation model to experiment with new AI infrastructure concepts before they become mandatory.
Mastering Multi-Tenant GPU Kubernetes with Kamaji and HAMi
Unlock the power of multi-tenant GPU workloads in Kubernetes using Kamaji and HAMi. This setup allows you to efficiently allocate GPU resources across multiple tenants, leveraging simulated hardware for testing and development. Dive into the specifics of configuration and deployment to streamline your GPU management.
- →Utilize Kamaji to run each tenant's Kubernetes control plane as pods for better resource management.
- →Leverage HAMi to allocate GPU slices, optimizing GPU utilization across multiple pods.
Maximize Your AI SRE Evaluation with Contextual Awareness
Before diving into AI SRE evaluations, ensure your existing agent has the right context. By leveraging Radar's read-only MCP tools, you can access critical insights like current failures and change timelines that raw kubectl can't provide. This context is essential for effective decision-making.
- →Leverage Radar's MCP tools for richer context beyond kubectl.
- →Access current failures and change timelines to improve AI SRE evaluations.
Unlocking BackstageCon: Insights for KubeCon + CloudNativeCon 2026
BackstageCon is set to be a pivotal gathering for Backstage enthusiasts at KubeCon + CloudNativeCon 2026. Dive into AI features and the Model Context Protocol that are reshaping software development workflows. This is your chance to connect with the community and deepen your understanding of Backstage.
- →Explore AI features that are transforming software development workflows.
- →Engage in discussions about the Model Context Protocol (MCP) for AI agents.
Transforming AI Workloads: My Journey from Attendee to Speaker at KubeCon India 2026
KubeCon + CloudNativeCon India 2026 was a turning point for me, moving from attendee to speaker. I shared insights on building a self-hosted AI cluster with NVIDIA's DGX Spark, leveraging Dynamic Resource Allocation (DRA) for optimal GPU scheduling.
- →Leverage Dynamic Resource Allocation (DRA) to optimize GPU scheduling.
- →Build a Kubernetes cluster on NVIDIA's DGX Spark for robust AI workloads.
Scaling GPU AI Workloads with ECS Managed Instances: A Deep Dive
Running GPU workloads at scale can be a nightmare without the right tools. Amazon ECS Managed Instances streamline this process, reducing operational overhead while ensuring efficient resource allocation. Discover how Ramp leverages this to power its Bore ML inference platform.
- →Leverage Amazon ECS Managed Instances to reduce operational overhead in GPU workload management.
- →Utilize dedicated capacity providers for each GPU workload family to optimize resource allocation.
Building a Reliable Cloud Native Foundation for Distributed AI Training
Unlock the potential of distributed AI training with Kubernetes. By leveraging RDMA for high-throughput communication and Lustre for efficient data access, you can streamline your ML workflows. Discover how to set up a robust infrastructure that minimizes management overhead.
- →Leverage RDMA for lower-overhead, high-throughput communication between GPU nodes.
- →Utilize Lustre for high-throughput shared access to training data and checkpoints.
Secure Multi-Tenant GPU Metrics in Kubernetes: A Deep Dive
In a multi-tenant Kubernetes environment, managing GPU metrics securely is crucial. By leveraging MetricAccess and kube-rbac-proxy, you can ensure that each team only sees its own metrics. This article breaks down how to implement these features effectively.
- →Implement MetricAccess to declare specific metrics for each team.
- →Use kube-rbac-proxy for authentication and authorization in a Kubernetes-native way.
Transforming Kubernetes: From Cloud Native to AI Native
As AI continues to evolve, so must our infrastructure. Discover how Kubernetes can adapt to support AI-native applications while avoiding common pitfalls. Learn why relying solely on simplified tools can lead to serious security issues.
- →Recognize that AI-native infrastructure often lacks the robustness of true cloud-native setups.
- →Monitor for security vulnerabilities, as seen in the exposure of user data due to misconfigured applications.
Scaling AI with Kubernetes: The Role of CNCF Silver Members
As enterprises scale AI from training to inference, operational efficiency becomes critical. Kubernetes plays a vital role in managing these workloads, and the support from CNCF Silver Members enhances this infrastructure.
- →Understand the role of CNCF Silver Members in building cost-efficient AI infrastructure.
- →Leverage Kubernetes for automating deployment and scaling of AI workloads.
Unifying AI Training and Inference on Kubernetes: Lessons from China Merchants Bank
China Merchants Bank has cracked the code on unifying AI training and inference using Kubernetes. By leveraging tools like Kueue and Fluid, they efficiently manage nearly 10,000 heterogeneous accelerator cards. Dive into how this unified control plane can revolutionize your AI workflows.
- →Leverage Kueue to manage training admission and prevent resource wastage.
- →Utilize KEDA and Prometheus for dynamic scaling of online inference based on demand.
China's Cloud Native Surge: Kubernetes in the Age of AI Inference
China is rapidly advancing in cloud native technologies, especially as AI transitions to inference. Kubernetes plays a pivotal role in this evolution, providing the infrastructure for scalable applications and resilient data pipelines.
- →Leverage Kubernetes for scalable application infrastructure across various cloud environments.
- →Implement chaos engineering to enhance system resilience during distributed inference.
Kubernetes and AI: Navigating the New Frontier
Kubernetes isn't new, but the rise of AI workloads is reshaping how we think about it. Managing job placement and GPU utilization is crucial to prevent resource starvation and maintain application performance. Dive into the complexities of running AI in Kubernetes and what you need to watch out for.
- →Manage job placement actively to optimize resource usage.
- →Utilize GPUs efficiently to avoid costly idling.
Navigating Heterogeneous Infrastructure for AI with Kubernetes
AI workloads are complex, requiring both CPU and GPU resources to function optimally. Understanding how Dynamic Resource Allocation (DRA) can help you manage these resources is crucial for effective AI platform engineering.
- →Understand the distinct roles of CPU and GPU in AI workloads.
- →Utilize Dynamic Resource Allocation (DRA) to manage specialized device requests.
Accelerate AI Inference: Fast Model Loading on Amazon EKS
Speed is crucial for AI inference, and inefficient model loading can bottleneck your applications. By leveraging tools like Run:ai Model Streamer and torch.compile, you can significantly reduce startup times. Discover how to optimize your Kubernetes deployments for faster performance.
- →Leverage Run:ai Model Streamer for efficient model loading.
- →Optimize weights loading from S3 to reduce startup time.
Predictive Autoscaling for GPU Workloads: Stay Ahead of Demand in Kubernetes
In a world where GPU workloads can spike unexpectedly, predictive autoscaling is a game changer. By leveraging a Bi-LSTM model, Kubernetes can forecast demand and pre-provision capacity, ensuring your applications are ready when it matters most.
- →Leverage the Predictive Controller to forecast demand every 60 seconds.
- →Utilize Bi-LSTM models to analyze past metrics for better predictions.
Is Your Kubernetes Cluster AI-Ready? Here's What You Need to Know
As AI workloads surge, Kubernetes must adapt. Dynamic Resource Allocation (DRA) offers a game-changing way to request specialized hardware for these demanding tasks. Discover how to leverage this feature effectively.
- →Leverage Dynamic Resource Allocation (DRA) for flexible hardware requests.
- →Monitor resource utilization to optimize performance for AI workloads.
Building an AI Factory on Kubernetes: Optimizing Resource Allocation
Transform your AI workloads with Kubernetes by leveraging Dynamic Resource Allocation and HAMi. Discover how these tools can optimize resource use and tenant isolation in your AI factory setup.
- →Leverage Dynamic Resource Allocation (DRA) to treat accelerators as rich devices with attributes.
- →Implement HAMi to enforce per-pod memory and compute limits for better resource management.
Kubeflow's Graduation: The New Standard for Cloud Native AI Operations
Kubeflow has officially graduated, marking a significant milestone for cloud native AI operations. This platform standardizes the entire AI and ML lifecycle, from data processing to model serving, making it a game-changer for production environments.
- →Leverage Kubeflow's capabilities for standardizing the AI and ML lifecycle.
- →Utilize native tools for data processing and model serving to streamline operations.
Who Owns the AI Pipeline? Navigating LLMOps and Platform Engineering
Understanding who should own the AI pipeline is crucial for effective LLMOps. This article dives into the lifecycle of large language model operations, from data prep to monitoring, and highlights the importance of treating prompts as versioned artifacts.
- →Understand LLMOps as a comprehensive lifecycle from data prep to monitoring.
- →Treat prompts as versioned artifacts to ensure consistency in model performance.
Industry-standard certifications built by the people behind Linux and Kubernetes. Earn the CKA — the gold standard Kubernetes administrator cert. OpsCanary readers get 30% off year-round with code OPSCANARY3.
Get CKA certified →Get the daily digest
One email. 5 articles. Every morning.
No spam. Unsubscribe anytime.