# Lab 04 — MIG: Partition a Single A100 into Isolated GPU Instances and Serve a Model on One Slice **Purpose:** Hands-on, runnable walkthrough that enables MIG on a rented A100/H100, carves it into isolated GPU instances, and serves a small LLM pinned to a single slice - so you can defend MIG claims in an LLMOps/AI-infra interview from real experience. ## Cost and teardown (read first) - A100 80GB on Vast.ai or RunPod is roughly USD 1-2/hr. Budget about 1-2 hours; this lab is doable in under an hour once the box is up. - MIG only works on A100 and H100-class GPUs (data-center Ampere/Hopper). It is NOT available on T4, L4, RTX 3090/4090, A10, etc. Confirm the rented box is an A100 or H100 before you start. - MIG requires root and a driver reset (sometimes a reboot). Many rented instances are containers without the privileges MIG needs. If `sudo nvidia-smi -mig 1` fails with a permissions/insufficient-privileges error, you likely need a bare-metal or root-enabled instance (RunPod "Secure Cloud" bare-metal, or a Vast.ai host that grants full device access). Filter for that when renting. - STOP when done. Disable MIG (`sudo nvidia-smi -mig 0`) and destroy the instance so billing stops. GPU rentals bill by the hour whether idle or busy. --- ## What MIG is (concept) Multi-Instance GPU (MIG) is HARDWARE partitioning. A single physical A100 is split into up to 7 GPU Instances (GIs), each with its own dedicated slice of streaming multiprocessors, L2 cache slices, and framebuffer memory. Instances are isolated - a workload on one slice cannot starve or crash a workload on another, and they have separate memory address spaces. This is different from time-slicing. Time-slicing oversubscribes one GPU by round-robin scheduling many processes onto the full device; there is no memory isolation and one process can OOM the whole card. MIG gives true isolation at the cost of flexibility (fixed profiles). Rule of thumb: MIG for multi-tenant isolation and right-sizing; time-slicing for bursty dev/test where isolation does not matter. **Where MIG matters:** multi-tenant GPU serving (each tenant a slice), right-sizing small models that would waste a full 80GB card, and a team of developers sharing one A100 without stepping on each other. **Profile naming:** profiles are named `g.gb`, e.g. `1g.10gb`, `2g.20gb`, `3g.40gb`, `7g.80gb` on an 80GB A100 (the 40GB A100 shows `1g.5gb`, `2g.10gb`, `3g.20gb`, `7g.40gb`). The `g` count is GPU-instance compute slices; the `gb` is dedicated memory. Exact profiles depend on the specific GPU - always list them on the actual box rather than assuming. --- ## Step 0 - Confirm the GPU and driver ```bash nvidia-smi # confirm it says A100 or H100 nvidia-smi --query-gpu=name,driver_version,mig.mode.current --format=csv ``` If `mig.mode.current` shows `[N/A]`, the GPU does not support MIG (wrong card) - stop and rent an A100/H100. A recent driver (R470+, ideally R525+) is expected on these boxes. ## Step 1 - Check MIG support and current mode ```bash nvidia-smi -i 0 -q | grep -i mig # look for the MIG Mode block (Current/Pending) nvidia-smi --query-gpu=mig.mode.current --format=csv ``` `Enabled`/`Disabled` (or `1`/`0`) tells you the current state. Fresh rentals are almost always `Disabled`. ## Step 2 - Enable MIG mode (root, no GPU workloads running) MIG mode cannot be enabled while any process holds the GPU. Kill workloads first, then: ```bash nvidia-smi # verify NO processes are listed under "Processes" sudo nvidia-smi -i 0 -mig 1 # enable MIG on GPU 0 ``` You will often see `Pending` state and a message that a GPU reset is required. Apply it with: ```bash sudo nvidia-smi --gpu-reset -i 0 # may fail if the driver is busy ``` If `--gpu-reset` fails (common in containerized rentals where the driver is shared), reboot the box: ```bash sudo reboot ``` After reboot/reset, re-check: ```bash nvidia-smi --query-gpu=mig.mode.current --format=csv # expect: Enabled ``` If enabling MIG returns "Insufficient Permissions" or similar, this instance does not grant the privileges MIG needs - switch to a bare-metal/root instance (see Cost note). ## Step 3 - List available GPU instance profiles ```bash sudo nvidia-smi mig -lgip # list GPU Instance profiles + how many of each are free ``` Read the `Instances Free/Total` columns. A typical 80GB A100 shows profiles like: ``` Profile Name Instances Free/Total Memory 19 1g.10gb 7/7 ~10 GB 14 2g.20gb 3/3 ~20 GB 9 3g.40gb 2/2 ~40 GB 0 7g.80gb 1/1 ~80 GB ``` Profile IDs are what you pass to create instances. IDs and exact sizes are GPU-specific - use what `-lgip` prints on your box, not the table above. ## Step 4 - Create GPU instances + compute instances The `-C` flag auto-creates a matching Compute Instance (CI) inside each GPU Instance (GI), which is what CUDA actually runs on. Create three `1g.10gb` slices by profile name (or by ID): ```bash sudo nvidia-smi mig -cgi 1g.10gb,1g.10gb,1g.10gb -C # equivalent by profile ID (verify IDs from -lgip on your box): # sudo nvidia-smi mig -cgi 19,19,19 -C ``` You can mix sizes as long as they fit (e.g. `3g.40gb,2g.20gb,1g.10gb`). The placement engine rejects layouts that do not fit - if creation fails, list profiles again to see what is still free. ## Step 5 - List the instances you created ```bash sudo nvidia-smi mig -lgi # list GPU Instances sudo nvidia-smi mig -lci # list Compute Instances nvidia-smi # top table now shows a "MIG devices" section nvidia-smi -L # lists each MIG device with its MIG- ``` `nvidia-smi -L` output looks like: ``` GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-xxxx) MIG 1g.10gb Device 0: (UUID: MIG-abcd1234-...) MIG 1g.10gb Device 1: (UUID: MIG-efgh5678-...) MIG 1g.10gb Device 2: (UUID: MIG-ijkl9012-...) ``` Each slice is addressed by its `MIG-`. Copy one UUID for the next step. ## Step 6 - Pin a process to ONE slice (host) ```bash # CUDA sees ONLY this slice - confirm isolation: CUDA_VISIBLE_DEVICES=MIG-abcd1234-... nvidia-smi -L CUDA_VISIBLE_DEVICES=MIG-abcd1234-... python -c "import torch; print(torch.cuda.get_device_name(0), torch.cuda.mem_get_info())" ``` `mem_get_info()` should report roughly the slice size (about 10GB for `1g.10gb`), not 80GB - proof the process is boxed into its instance. ## Step 7 - Run a container pinned to one MIG slice Requires the NVIDIA Container Toolkit (`nvidia-ctk` / `--gpus` support), which is preinstalled on most GPU rental images. Pass the slice by its MIG UUID: ```bash docker run --rm --gpus '"device=MIG-abcd1234-..."' \ nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi -L ``` Equivalent via the older env-var form: ```bash docker run --rm --runtime=nvidia \ -e NVIDIA_VISIBLE_DEVICES=MIG-abcd1234-... \ nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi -L ``` The container should see exactly one MIG device. Note: for MIG the device string must be the full `MIG-` - integer GPU indexes do not address MIG slices. ## Step 8 - Serve a small model on the slice with vLLM Serve a small model (Qwen2.5-1.5B-Instruct fits comfortably in a `1g.10gb` slice) and size `--gpu-memory-utilization` to the slice, not the whole card. Start conservative (0.5) and raise if it fits: ```bash docker run --rm --gpus '"device=MIG-abcd1234-..."' \ -p 8000:8000 \ vllm/vllm-openai:latest \ --model Qwen/Qwen2.5-1.5B-Instruct \ --gpu-memory-utilization 0.6 \ --max-model-len 4096 ``` Notes: - vLLM computes its KV-cache budget from the MEMORY IT SEES, which is the slice (about 10GB), so a fraction like 0.6 keeps weights + KV inside the slice. If it OOMs, lower the fraction or `--max-model-len`. - If the image needs an HF token for the model, add `-e HF_TOKEN=...`. - vLLM on MIG runs single-slice only - MIG slices cannot be aggregated across a tensor-parallel group, so keep it to one slice per server. To serve more, launch a second container pinned to a different MIG UUID. Once it is up, hit it from another shell: ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"Qwen/Qwen2.5-1.5B-Instruct","messages":[{"role":"user","content":"Say hi in 5 words."}]}' ``` Then run `nvidia-smi` on the host: you will see the vLLM process attributed to ONE MIG device, with the other two slices idle - visible proof of isolation. ## Step 9 - Teardown (do not skip - billing) ```bash # stop containers first, then destroy MIG instances (CIs before GIs) sudo nvidia-smi mig -dci # destroy compute instances sudo nvidia-smi mig -dgi # destroy GPU instances sudo nvidia-smi -i 0 -mig 0 # disable MIG mode (may need reset/reboot) ``` Then DESTROY the rented instance in the Vast.ai/RunPod console so the hourly charge stops. --- ## Kubernetes angle (conceptual) In a cluster you do not run these commands by hand. The NVIDIA GPU Operator, driven by a MIG config profile, applies the MIG layout on labeled nodes and exposes slices to the scheduler through the device plugin. Two strategies: `single` (every GPU on a node is carved into the same uniform profile, and slices advertise as `nvidia.com/gpu`) and `mixed` (heterogeneous profiles advertise as distinct resources like `nvidia.com/mig-1g.10gb`). Pods then request a slice as a normal resource limit, e.g. `nvidia.com/mig-1g.10gb: 1`, and land on one isolated instance. This is how you offer right-sized, isolated GPU to many teams from a small A100/H100 fleet. --- ## What you will be able to say in the interview - "I have enabled MIG on a real A100, which requires root and a driver reset with no active GPU workloads, and carved it into seven isolated 1g.10gb instances - I know the profile naming and that exact profiles are GPU-specific." - "I understand MIG vs time-slicing: MIG is hardware isolation with dedicated memory and compute per instance, time-slicing is oversubscription with no isolation - I choose MIG for multi-tenant or right-sizing, time-slicing for bursty dev." - "I served a model with vLLM pinned to a single MIG slice via its MIG-UUID, sizing gpu-memory-utilization to the slice, and verified from nvidia-smi that the process was confined to one instance while others stayed idle." - "In Kubernetes I would drive this with the GPU Operator using single or mixed MIG strategy so slices become schedulable resources that pods request by name." ## Artifact to capture - `nvidia-smi` and `nvidia-smi -L` output showing MIG enabled and multiple MIG device UUIDs (the full instance list). - A screenshot/log of vLLM serving on one slice plus the successful `curl` completion, and a host `nvidia-smi` showing the process bound to a single MIG device. > Accuracy note: exact profile IDs, memory sizes, and whether a full reboot vs `--gpu-reset` is needed vary by GPU, driver version, and host. If any flag behaves differently than shown, verify against the current NVIDIA MIG User Guide rather than forcing the command.