# Lab V — vLLM Core Build: Serve, Parallelize, Quantize, Tune, Benchmark

**Purpose:** A concrete, runnable checklist to stand up vLLM on a GPU, serve an open-weight LLM, exercise tensor parallelism + quantization, tune the serving knobs, and benchmark it under load. This is the single biggest gap-closer in the program — it turns "I read about vLLM" into "I ran it." Most of it runs on a **local 16GB card**; only the 2-GPU tensor-parallelism step needs a rented multi-GPU box.

**Cost reality:** a single A100 is roughly $1 to $2/hour on Vast or RunPod. A few hours is about $10 to $30. Or run the single-GPU steps free on your own card. Tear any rented box down when done so billing stops.

**Model choice:** use **Qwen2.5** (not gated, no Hugging Face token dance). Llama needs a Hugging Face license click + token; skip that friction for the first run.

---

## Step 0 — pick the box

- **Local:** a 16GB+ NVIDIA card serves a 7B model quantized to 4-bit comfortably (see the local-first lab).
- **Cloud provider:** RunPod or Vast (hourly, simple). Modal works too but is more code.
- **First run:** 1x A100 80GB (or 40GB, fine for 7B). Enough for serving + quantization.
- **Tensor-parallelism run:** 2x A100 so you can actually split a model. Do this second.
- **Image:** pick a PyTorch/CUDA base, or run vLLM's own container (below) so you don't fight CUDA installs.

---

## Step 1 — verify the GPU

SSH in (or open a local shell), then:

```bash
nvidia-smi          # confirm the GPU(s), driver, VRAM. Screenshot this.
nvcc --version      # CUDA toolkit (optional; the container carries its own)
```

Keep that `nvidia-smi` output — it's an artifact.

---

## Step 2 — run vLLM (OpenAI-compatible server)

Easiest path is the official container (no pip/CUDA fighting):

```bash
docker run --gpus all -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 --port 8000
```

Or bare pip if the image already has Python/CUDA:

```bash
pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --port 8000
```

**Read the startup logs.** vLLM prints the number of **KV-cache blocks**, the **GPU memory utilization** target, and the model's max length. That log line IS the KV-cache/PagedAttention story you'll talk about. Screenshot it.

---

## Step 3 — confirm it serves

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role":"user","content":"Say hi in one sentence."}]
  }'
```

You now have a self-hosted, OpenAI-compatible LLM endpoint. That alone is the "stood up vLLM on GPU infrastructure" line.

---

## Step 4 — tensor parallelism (the 2-GPU run)

On a 2x A100 box, serve a bigger model split across both cards:

```bash
vllm serve Qwen/Qwen2.5-32B-Instruct \
  --tensor-parallel-size 2 \
  --host 0.0.0.0 --port 8000
```

Watch `nvidia-smi` in another shell: **both GPUs fill with memory and show utilization.** That is tensor parallelism working, and now you can describe it from having seen it, not from a diagram. Screenshot both-GPU `nvidia-smi`.

---

## Step 5 — quantization (fit a model to less hardware)

Serve a pre-quantized AWQ model and compare VRAM:

```bash
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
  --quantization awq \
  --host 0.0.0.0 --port 8000
```

Note the VRAM drop versus the FP16 run in Step 2 (roughly 14GB to about 5 to 6GB). If you're on a Hopper/Ada card, also try FP8 on-the-fly:

```bash
vllm serve Qwen/Qwen2.5-7B-Instruct --quantization fp8 --host 0.0.0.0 --port 8000
```

Now the AWQ-vs-GPTQ-vs-FP8 answer is grounded in a VRAM number you watched change.

---

## Step 6 — tune the serving knobs (KV-cache / batching, hands-on)

Restart the server with different values and watch throughput move. These are the exact "tuning continuous batching and managing KV-cache memory" words a serving JD uses:

```bash
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --gpu-memory-utilization 0.90 \   # how much VRAM vLLM claims for weights + KV-cache
  --max-num-seqs 256 \              # max concurrent sequences in the running batch
  --max-model-len 8192 \            # caps KV-cache size per sequence
  --enable-chunked-prefill          # smooths prefill vs decode under load
```

Raise `--max-num-seqs` and `--gpu-memory-utilization`, re-benchmark (next step), and note the throughput/latency change. That is the tuning story. Change **one knob at a time** so you can attribute the result.

---

## Step 7 — benchmark under load (tokens/sec, TTFT, p99)

vLLM ships a serving benchmark. Point it at your running server:

```bash
# from the vllm repo, or: pip install vllm then use the bundled script
python benchmarks/benchmark_serving.py \
  --backend vllm \
  --model Qwen/Qwen2.5-7B-Instruct \
  --dataset-name random \
  --num-prompts 500 \
  --request-rate 20          # requests/sec; sweep this: 5, 10, 20, 40
```

It reports **throughput (tokens/sec), TTFT, TPOT/ITL, and p50/p99 latency.** Run it at several request rates to draw the throughput-vs-latency curve. Log each run to an experiment tracker (e.g. MLflow) recording rate, tokens/sec, TTFT, and p99. Those runs are your benchmarking artifact.

---

## Step 8 — wire it into a client / observability stack

Any OpenAI-compatible client or gateway can point its `base_url` at the vLLM endpoint (expose it via the provider's public URL or a tunnel). From there you can:

- Compare vLLM against another engine (Ollama, an API model) side by side on latency + tokens.
- Log every vLLM call's tokens + latency into an observability dashboard.

That closes the loop: a self-hosted vLLM service, monitored and benchmarked by your own tools.

---

## Step 9 — capture the artifacts (this is the point)

Save these; they're what make the claims interview-proof:

- `nvidia-smi` before/after, single-GPU and 2-GPU (tensor parallelism).
- vLLM startup log showing KV-cache blocks + GPU memory utilization.
- VRAM numbers: FP16 vs AWQ vs FP8.
- Benchmark runs: tokens/sec + TTFT + p99 across request rates.
- A one-page notes file mapping each JD term (PagedAttention, continuous batching, tensor parallelism, KV-cache, AWQ/FP8, tokens/sec) to the one thing you saw or changed.

---

## Step 10 — tear down

Stop the container, terminate any rented instance, confirm billing stopped. Done.

---

## What you'll be able to say in the interview

- "I stood up vLLM serving an open-weight model on GPU infrastructure, turned on tensor parallelism across two GPUs, served a quantized (AWQ/FP8) variant and watched VRAM drop, tuned continuous batching and the KV-cache memory budget, and benchmarked tokens/sec and p99 under a request-rate sweep."
- "I can read a vLLM startup log — KV-cache blocks, GPU-memory-utilization target, max length — and reason about the throughput-vs-latency trade-off from numbers I measured, not a diagram."