GPU‑Accelerated Workloads on KubeVirt: Scaling ML/AI in Kuberne... Amandeep Singh and Shivani Tiwari

GPU‑Accelerated Workloads on KubeVirt: Scaling ML/AI in Kubernetes

The video titled “GPU‑Accelerated Workloads on KubeVirt: Scaling ML/AI in Kubernetes” features a lightning talk by Amandeep Singh (founder at Wellin and former senior data scientist at PayPal) and Shivani Tiwari (Developer Relations at Wellin). The session focuses on integrating GPU acceleration with KubeVirt to efficiently scale machine learning (ML) and artificial intelligence (AI) workloads in Kubernetes environments.

---

Key Technological Concepts and Product Features

1. Introduction to KubeVirt

• KubeVirt extends Kubernetes by enabling management of virtual machines (VMs) alongside containers within the same Kubernetes cluster.
• It provides a unified platform allowing VMs and containers to interoperate, facilitating CPU and GPU workload management.

2. Challenges with CPU and Containers for AI/ML Workloads

• CPUs struggle with parallel computations required by deep learning models, leading to longer processing times.
• Containers alone cannot handle GPU workloads efficiently due to the need for custom GPU drivers on the host OS.
• Migrating GPU workloads between VMs and containers is complex and often requires downtime, which is problematic in production.

3. Role of GPUs in AI/ML

• GPUs accelerate inferencing and training by handling parallel processing tasks efficiently.
• Integrating GPUs with Kubernetes requires device plugins and drivers to expose GPU resources to containers and VMs.

4. Enabling GPU Workloads on KubeVirt

• GPU Device Plugin Installation: For example, Nvidia’s GPU operator plugin is deployed on Kubernetes nodes to expose GPU resources.
• Hardware Access Configuration: Ensuring GPU drivers (e.g., Nvidia drivers and CUDA libraries) are installed on Kubernetes nodes.
• Virtual Machine Manifest Configuration: A YAML file defines the VM instance with GPU resource requests and limits.
• VM Scheduling and GPU Pass-Through: Kubernetes schedules the VM on nodes with available GPUs, passing physical GPUs through to the VM.
• Inside the VM, the guest OS recognizes GPUs as physical devices, allowing ML/AI applications to utilize GPU acceleration for faster computation.

5. Additional Tools and Monitoring

• CNCF tools such as Prometheus and Grafana can be integrated with KubeVirt to monitor GPU usage and workloads, enhancing observability and management.

6. Challenges and Limitations

• Complexity in storage management.
• Migration difficulties between containers and VMs.
• Complex installation and setup processes.

---

Summary of the Process to Enable GPU Acceleration with KubeVirt

1. Install GPU device plugins on Kubernetes nodes.
2. Ensure GPU drivers and necessary libraries are installed.
3. Define GPU resource requests in VM YAML manifests.
4. Launch VMs that are scheduled on GPU-enabled nodes.
5. Enable GPU pass-through to VMs for direct hardware access.
6. Run AI/ML workloads inside VMs leveraging GPU acceleration.

---

Review/Guide/Tutorial Elements

• The talk serves as a brief guide on how to enable GPU acceleration in KubeVirt for AI/ML workloads.
• It outlines step-by-step configurations and architectural considerations.
• It highlights common problems and solutions related to GPU integration in Kubernetes environments.

---

Main Speakers

• Amandeep Singh – Founder at Wellin, former senior data scientist at PayPal.
• Shivani Tiwari – Developer Relations at Wellin.

The session was cut short but invited further detailed discussion in the KubeVirt community meetings.