# Lab 06 — Serve an LLM with vLLM on Kubernetes (GPU) Hands-on, runnable lab for an experienced Kubernetes engineer (CKA-level) who is newer to GPU scheduling. Goal: stand up a real GPU node in a real cluster, schedule a `vllm/vllm-openai` pod against `nvidia.com/gpu`, and curl the OpenAI-compatible endpoint. Pick ONE path, then run the shared manifests. **This is the combined LLMOps + Cloud-Infra-Ops centerpiece.** > Accuracy note: cloud accelerator names and driver-install flags change often. Treat every `gcloud` accelerator flag and device-plugin URL below as "verify against current docs" - do not trust this file as ground truth for exact flag spelling. --- ## Prerequisites (both paths) - `kubectl` (matching your cluster minor version) and a working kubeconfig. - A Hugging Face token if you serve a gated model. Export it: `export HF_TOKEN=hf_xxx`. - Start with a SMALL model so the lab is cheap and fast, e.g. `Qwen/Qwen2.5-0.5B-Instruct` or `TinyLlama/TinyLlama-1.1B-Chat-v1.0`. A single T4/L4 (16-24GB) handles these easily. - Budget discipline: GPU nodes bill by the second-ish and add up fast. Do the whole lab in one sitting, then tear down. --- ## PATH A - Managed cloud (GKE) - closest to most LLMOps JDs This mirrors what production teams do: a standard cluster plus a dedicated GPU node pool. ```bash # 0. Set project/zone (verify zone actually has your GPU type available) export PROJECT=your-gcp-project export ZONE=us-central1-a gcloud config set project "$PROJECT" # 1. Create a small CPU control/base cluster (cheap default node pool) gcloud container clusters create vllm-lab \ --zone "$ZONE" \ --num-nodes 1 \ --machine-type e2-standard-4 # 2. Add a GPU node pool. # VERIFY the current --accelerator type value and driver flags against GKE docs. # Common types: nvidia-tesla-t4 (cheapest), nvidia-l4 (better $/token), nvidia-tesla-a100. # GKE can auto-install drivers via --accelerator ...,gpu-driver-version=default (a managed DaemonSet). gcloud container node-pools create gpu-pool \ --cluster vllm-lab \ --zone "$ZONE" \ --machine-type n1-standard-8 \ --accelerator type=nvidia-tesla-t4,count=1,gpu-driver-version=default \ --num-nodes 1 \ --node-labels=gpu=true # 3. Get credentials gcloud container clusters get-credentials vllm-lab --zone "$ZONE" ``` Driver install: with `gpu-driver-version=default` GKE runs a DaemonSet that installs NVIDIA drivers and the device plugin for you. If you omit it (or on older setups), you apply the device plugin manually - see the "device plugin" note in Path B. Verify which behavior your GKE version uses. Confirm the node advertises GPUs: ```bash kubectl get nodes -o wide kubectl describe node | grep -A5 -i allocatable # expect nvidia.com/gpu: 1 ``` Rough cost: a single T4 node is roughly USD 0.35-0.60/hr, L4 roughly USD 0.70-1.00/hr, A100 far more (verify live pricing). The CPU base pool adds a little. Same pattern applies to EKS (`eksctl create nodegroup` with a `g4dn`/`g5` instance type) and AKS (`az aks nodepool add` with an `NC`/`ND` VM size) - you still end up with nodes advertising `nvidia.com/gpu`. Teardown (do this when done): ```bash gcloud container clusters delete vllm-lab --zone "$ZONE" --quiet ``` --- ## PATH B - Cheapest / local-ish - one rented GPU box + k3s For real GPU-scheduling experience at roughly USD 1-2/hr. Rent a single GPU box on Vast.ai or RunPod (a T4/RTX-3090/L4 is plenty for a tiny model). Caveat: some Vast/RunPod instances are containers that will NOT run nested Kubernetes cleanly (no systemd, restricted cgroups, or the host NVIDIA runtime not exposed to nested containers). Prefer an instance type that gives you a real VM / bare host with SSH and root. If `k3s` refuses to start or the device plugin cannot see the GPU, that instance is the problem - pick a bare-metal / VM offering instead. ```bash # On the rented box (has NVIDIA drivers + nvidia-container-toolkit preinstalled on good images): nvidia-smi # confirm the GPU + driver are visible on the host first # Install k3s (single node). k3s bundles containerd; ensure the nvidia runtime is wired in. curl -sfL https://get.k3s.io | sh - export KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get nodes # Apply the NVIDIA device plugin DaemonSet so pods can request nvidia.com/gpu. # VERIFY the current image tag / URL against the k8s-device-plugin repo before applying. kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin//deployments/static/nvidia-device-plugin.yml ``` The device plugin is the piece that makes the kubelet advertise `nvidia.com/gpu` as a schedulable resource. Without it, the GPU is invisible to the scheduler even though `nvidia-smi` works on the host. Confirm: ```bash kubectl get nodes -o wide kubectl describe node | grep -i nvidia.com/gpu # expect it under Capacity/Allocatable ``` Teardown: STOP or DESTROY the rented instance from the Vast/RunPod console. A stopped GPU box can still bill for storage; destroy it if you are truly done. --- ## Common: the Kubernetes manifests ### GPU scheduling essentials (read before applying) - `resources.limits: nvidia.com/gpu: 1` - GPUs are a countable, non-oversubscribable extended resource. You request whole GPUs; you cannot request 0.5 (unless using MIG/time-slicing). Request and limit must match. - Taint your GPU nodes, e.g. `nvidia.com/gpu=present:NoSchedule`, so ordinary workloads do NOT land on expensive GPU hardware. Only pods with a matching toleration schedule there. On GKE, GPU node pools are auto-tainted with `nvidia.com/gpu=present:NoSchedule` - your pod MUST tolerate it. - Pair the toleration with a `nodeSelector` or affinity (e.g. the `gpu=true` label from Path A, or `cloud.google.com/gke-accelerator`) so the pod targets the GPU pool specifically. ### Deployment + Service (copy-paste, edit placeholders) ```yaml # vllm.yaml apiVersion: apps/v1 kind: Deployment metadata: name: vllm labels: { app: vllm } spec: replicas: 1 selector: { matchLabels: { app: vllm } } template: metadata: labels: { app: vllm } spec: # Target the GPU node pool (adjust label to your cluster). nodeSelector: gpu: "true" # Tolerate the GPU node taint so we are allowed onto the expensive nodes. tolerations: - key: "nvidia.com/gpu" operator: "Exists" effect: "NoSchedule" containers: - name: vllm image: vllm/vllm-openai:latest # pin a real version tag in production args: - "--model" - "Qwen/Qwen2.5-0.5B-Instruct" # small model = cheap/fast lab - "--gpu-memory-utilization" - "0.90" # fraction of VRAM vLLM may claim - "--max-model-len" - "4096" ports: - containerPort: 8000 env: - name: HUGGING_FACE_HUB_TOKEN value: "REPLACE_OR_USE_SECRET" # prefer a Secret + valueFrom in real use resources: limits: nvidia.com/gpu: 1 # <-- schedules onto exactly one GPU readinessProbe: httpGet: { path: /health, port: 8000 } initialDelaySeconds: 60 # model load/download takes time periodSeconds: 10 livenessProbe: httpGet: { path: /health, port: 8000 } initialDelaySeconds: 120 periodSeconds: 20 volumeMounts: - name: hf-cache mountPath: /root/.cache/huggingface volumes: # emptyDir is fine for a throwaway lab (cache dies with the pod). # For real use, swap for a PVC so model weights survive restarts. - name: hf-cache emptyDir: {} --- apiVersion: v1 kind: Service metadata: name: vllm spec: type: ClusterIP selector: { app: vllm } ports: - port: 8000 targetPort: 8000 ``` Apply and watch it schedule + pull the model: ```bash kubectl apply -f vllm.yaml kubectl get pods -w # Pending -> ContainerCreating -> Running (readiness passes after model load) kubectl logs -f deploy/vllm # watch weight download + "Uvicorn running on ..." ``` If the pod stays `Pending`, `kubectl describe pod` almost always shows the reason: `Insufficient nvidia.com/gpu` (no GPU advertised - device plugin/driver problem) or a taint the pod does not tolerate. ### Reach the endpoint ```bash # From your laptop via port-forward: kubectl port-forward svc/vllm 8000:8000 # In another terminal - OpenAI-compatible chat completion: curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role":"user","content":"Say hello in one sentence."}] }' ``` To expose beyond your laptop, front it with an Ingress or a LoadBalancer Service - but for a lab, `port-forward` is safest (no public GPU endpoint left running). --- ## Scaling notes - HPA basics: an HPA can scale a serving Deployment on CPU or a custom metric (queue depth, tokens/sec). Example: `kubectl autoscale deploy/vllm --min=1 --max=3 --cpu-percent=70`. But CPU is a poor proxy for LLM load - production teams scale on a real serving metric via the custom-metrics API. - The real constraint: each replica pins a whole GPU (`nvidia.com/gpu: 1`). So HPA replica count is bounded by available GPUs. When you run out, the Cluster Autoscaler has to add GPU NODES - and GPU nodes are the expensive, slow-to-provision unit. You are effectively autoscaling by whole GPUs, not fractions. MIG or time-slicing lets multiple pods share one GPU, but adds complexity and is not free performance. - NVIDIA GPU Operator (the production way): instead of hand-applying the device plugin, the GPU Operator (a Helm chart) installs and lifecycle-manages the whole GPU stack - NVIDIA drivers (via the driver container), the `nvidia-container-toolkit`, the `k8s-device-plugin`, `dcgm-exporter` (GPU metrics for Prometheus/Grafana), node feature discovery, and the MIG manager. On a self-managed cluster you would deploy the GPU Operator rather than wiring each component by hand; managed clusters (GKE with `gpu-driver-version`) do much of this for you. --- ## Cost & teardown (do NOT skip) - Path A: `gcloud container clusters delete vllm-lab --zone "$ZONE" --quiet`. Confirm no GPU node pools linger. - Path B: STOP or DESTROY the rented box from the provider console; a merely-stopped box may still bill for disk. - GPU nodes bill fast - a forgotten A100 overnight is a real bill. Tear down the same day. --- ## What you'll be able to say in the interview 1. "GPUs are a countable extended resource - I request them with `resources.limits: nvidia.com/gpu: 1`; you get whole GPUs, no oversubscription unless you use MIG or time-slicing." 2. "I taint GPU nodes (`nvidia.com/gpu=present:NoSchedule`) and add matching tolerations plus a nodeSelector/affinity so only GPU workloads land on expensive hardware and everything else stays off it." 3. "The NVIDIA device plugin DaemonSet is what makes the kubelet advertise `nvidia.com/gpu`; on GKE the managed driver DaemonSet handles drivers, and in production I'd use the NVIDIA GPU Operator to manage drivers, device plugin, dcgm-exporter metrics, and MIG." 4. "I served an OpenAI-compatible endpoint with the `vllm/vllm-openai` image, tuned `--gpu-memory-utilization`, and health-probed `/health` for readiness/liveness so it only takes traffic after the weights load." 5. "Autoscaling a serving deployment is really autoscaling by whole GPUs - HPA replicas are capped by GPU count, and past that the cluster autoscaler must add GPU nodes, which are the slow, expensive unit." --- ## Artifacts to capture (screenshots for your portfolio / interview) - `kubectl get nodes -o wide` showing the GPU node(s). - `kubectl describe node ` with `nvidia.com/gpu` under Capacity/Allocatable. - `kubectl get pods` showing the `vllm` pod `Running` and readiness passing. - The `curl` request and the JSON completion response from the model. - (Bonus) `kubectl describe pod vllm-...` showing the toleration/nodeSelector actually placed it on the GPU node.