# Lab 00b — Local-First on Your RTX 5060 Ti (16GB): Save the Cloud Money

**Purpose:** Do most of this program on the GPU you already own, for free, and rent cloud GPUs only for the handful of labs that physically need data-center hardware. The skills and commands are identical to the cloud versions - you lose only raw scale and two data-center-only features (MIG, multi-GPU). For interview prep that trade is almost always worth it.

**Honest assessment (you asked for quality + useful, not just cheap):** running vLLM/SGLang, tuning serving knobs, quantizing models, benchmarking the throughput-latency curve, and standing up monitoring on your own 16GB card is **genuinely quality practice** - the method, the commands, and the talking points transfer 100% to a rented A100. The only things you cannot do locally are MIG (needs A100/H100) and multi-GPU tensor parallelism (needs 2+ GPUs). Do everything else at home for $0.

---

## What your card is

- **RTX 5060 Ti 16GB** = NVIDIA **Blackwell** (consumer, compute capability sm_120). 16GB VRAM.
- **Fits comfortably:** 7B models quantized to 4-bit (AWQ/GPTQ) with room for a healthy KV-cache; 8B-14B quantized; 7B at FP16 is tight (~14GB weights + KV, borderline - quantize it).
- **Blackwell caveat (verify current support):** Blackwell is a newer architecture, so you need **current CUDA (12.8+), a recent NVIDIA driver, recent vLLM, and a matching PyTorch (cu128 build)**. By now this is mature, but if vLLM errors with "no kernel image for sm_120" you are on too-old a build - upgrade vLLM/PyTorch or use the latest `vllm/vllm-openai` container. Verify the current minimum against the vLLM release notes.

---

## Local setup (Windows workstation)

Two good options; pick one:

**Option A - Docker Desktop + WSL2 (recommended, matches the cloud labs):**
1. Install WSL2 + a recent NVIDIA Windows driver (WSL2 CUDA passthrough is built in).
2. Install Docker Desktop with the WSL2 backend and enable GPU support.
3. Verify the GPU reaches containers:
   ```bash
   docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu22.04 nvidia-smi
   ```
   If that prints your 5060 Ti, every `docker run --gpus all ...` command in the other labs works locally, unchanged.

**Option B - native WSL2 Ubuntu + pip:**
1. In a WSL2 Ubuntu shell with the NVIDIA driver installed, `pip install vllm` (a current version) in a venv.
2. `vllm serve Qwen/Qwen2.5-7B-Instruct --quantization awq --gpu-memory-utilization 0.90` (use an AWQ model id for 4-bit).

**Ollama (fastest local smoke test):** `ollama run qwen2.5:7b` gives you a served model in one command - good for confirming the card works before the heavier vLLM labs. Ollama is not the interview skill (vLLM is), but it is a zero-friction warm-up.

---

## What to run LOCALLY (free) vs what to RENT (cloud)

| Lab | Local on 5060 Ti? | Notes |
|---|---|---|
| 01 Modal serverless | Cloud (free tier) | It IS the serverless lab - run on Modal's free credits |
| V vLLM core: serve + quantize + **tune** + benchmark | **LOCAL** (mostly) | Serve a 7B-AWQ, tune knobs, benchmark - all local. Only the **2-GPU tensor-parallelism** step needs cloud |
| 02 SGLang vs vLLM | **LOCAL** | Small model, both engines, compare - fully local |
| 03 GPU monitoring (DCGM/Grafana) | **LOCAL** (with caveat) | Works locally; note GeForce cards expose **fewer DCGM profiling fields** than data-center cards. `nvidia-smi` + core metrics always work; the method transfers |
| 04 MIG | **CLOUD ONLY** | MIG is A100/H100 hardware - impossible on a 5060 Ti. Rent for this one |
| 05 NIM + Triton | Mixed | **Triton runs local.** Many **NIMs target data-center GPUs** - check the NIM's NGC page; rent if it needs a bigger card |
| 06 vLLM on Kubernetes | **LOCAL** (Path B) | Run k3s/kind + the NVIDIA device plugin on your box. GKE path (Path A) only if you want the managed-cloud experience |
| 07 Tuning + benchmarking playbook | **LOCAL** | The whole point is method, not scale - do it at home |

**Bottom line:** you can do roughly **6 of the 8 labs locally for free.** Rent cloud GPUs only for: **MIG (Lab 04)**, the **2-GPU tensor-parallelism step** of the vLLM core, **A100-scale benchmark numbers** if you want them, and **NIM** if the specific microservice needs a data-center card. That is maybe 3-5 hours of cloud time total (~$5-15), not a whole weekend of rentals.

---

## The one honest limitation to name in an interview

Your local numbers (tokens/sec, max concurrency) will be smaller than an A100's - that is scale, not skill. Say it straight: *"I run and tune vLLM and SGLang locally on a 16GB Blackwell card and rent A100s when I need MIG, multi-GPU tensor parallelism, or data-center-scale throughput numbers."* That is a strong, credible, cost-aware answer - exactly the mindset an infra team wants.

---

## What you'll be able to say in the interview

- "I do most of my LLM-serving work on a local 16GB GPU - serving, 4-bit quantization, serving-knob tuning, benchmarking, and monitoring - and rent cloud A100s only for MIG and multi-GPU work. Same tools, same commands, cost-aware."
- "I know the practical VRAM math: a 7B model quantized to 4-bit fits a 16GB card with room for KV-cache, where FP16 would be too tight - so I pick precision to fit the hardware."