# Lab 07 — Performance Tuning & Benchmarking Playbook **Purpose:** The "tuning" skill Paul emphasized, made explicit. This is the decision guide that sits on top of the runnable steps in the vLLM core build (`A100_VLLM_BUILD.md`, Steps 6-7): what knob to turn, when, and how to read the result. Run it on your **local 5060 Ti** (the method is what matters, not the scale) or on a rented box for real A100 numbers. **What "tuning" means here:** given a model, a GPU, and a traffic pattern, get the best throughput and latency you can without falling over. It is a loop: change one knob, benchmark, read the metrics, decide the next change. --- ## The mental model: three things compete for VRAM 1. **Model weights** (fixed once you pick the model + quantization). 2. **KV-cache** (grows with concurrent requests x context length - this is your throughput headroom). 3. **Activation/working memory** (overhead). Tuning is mostly about **trading VRAM between weights and KV-cache**, and about **how aggressively you batch**. Quantize the weights -> more room for KV-cache -> more concurrent requests -> higher throughput. That is the whole game. --- ## The knobs (vLLM), and what each one does | Knob | What it controls | Turn it up when | Turn it down when | |---|---|---|---| | `--quantization awq` (or gptq/fp8) | Weight precision -> weight VRAM | You need room for more KV-cache / bigger batch | Quality matters more than throughput | | `--gpu-memory-utilization` (0.0-1.0) | Fraction of VRAM vLLM claims for weights + KV-cache | You have headroom and want more KV-cache | You see OOM or the box does other work | | `--max-num-seqs` | Max concurrent sequences in the running batch | Throughput is the goal and you have KV-cache room | p99 latency is climbing / OOM | | `--max-model-len` | Max context length (caps KV-cache per sequence) | You genuinely need long context | You want more concurrent short requests | | `--enable-chunked-prefill` | Interleaves prefill with decode | Long prompts are starving decode (TTFT spikes) | (usually leave on) | | `--tensor-parallel-size N` | Split model across N GPUs | Model does not fit one GPU / need more compute | Single GPU is enough (cloud, 2+ GPUs) | Change **one at a time**, then benchmark. Changing several at once means you cannot tell what helped. --- ## The metrics (what you are optimizing) - **Throughput** - tokens/sec (or requests/sec) the server sustains. The headline number. - **TTFT (time to first token)** - how long before the user sees anything. Driven by prefill + queueing. - **TPOT / ITL (time per output token / inter-token latency)** - how fast tokens stream after the first. Driven by decode. - **p50 / p99 latency** - median and tail. The tail (p99) is what users actually feel and what SLOs are written against. **The core tradeoff:** bigger batches (`--max-num-seqs` up) raise **throughput** but worsen **p99 latency** - each request waits behind more work. Tuning is finding the batch size where throughput is high but p99 is still inside your SLO. --- ## The benchmarking loop (runnable) Serve the model, then drive load at increasing request rates and record the curve. ```bash # vLLM ships a serving benchmark. Sweep the request rate. python benchmarks/benchmark_serving.py \ --backend vllm \ --model Qwen/Qwen2.5-7B-Instruct \ --dataset-name random \ --num-prompts 500 \ --request-rate 5 # then re-run at 10, 20, 40, 80 ``` For each rate, record: throughput (tokens/sec), TTFT, TPOT, p50, p99. Plot throughput (x) against p99 (y). You will see: - A flat region where throughput rises with little latency cost (you have headroom). - A knee where p99 starts climbing fast (you are saturating - the KV-cache is full and requests queue). **The knee is your operating point.** Set your target concurrency just below it. That single plot is the strongest artifact you can bring to an interview - it is "I found the max sustainable load." --- ## A concrete tuning session (do this) 1. **Baseline:** serve at defaults, benchmark at rate 10. Record throughput + p99. 2. **Quantize:** switch to a 4-bit (AWQ) model. Re-benchmark. VRAM for weights drops - note the freed memory. 3. **Spend the freed VRAM on KV-cache:** raise `--gpu-memory-utilization` (e.g. 0.85 -> 0.92) and/or `--max-num-seqs`. Re-benchmark. Throughput should rise. 4. **Push until it breaks:** keep raising `--max-num-seqs` until p99 crosses your SLO or you OOM. Back off one step. That is your tuned config. 5. **Log every run to MLflow** (the existing `llm-eval-pipeline`): record the knob values + throughput + TTFT + p99 per run. Now you have a tracked experiment, not just terminal scrollback. --- ## Reading the GPU while you tune (ties to Lab 03) Watch DCGM/`nvidia-smi` during the sweep: - **VRAM (FB_USED) pinned near cap + p99 climbing** = KV-cache full, you are memory-bound. Quantize or shorten `--max-model-len`, do not just add batch. - **GPU-util high + power near cap + throughput flat** = compute-bound. A bigger/faster GPU or tensor parallelism is the lever, not more batching. - **GPU-util low while requests queue** = you are under-batching or bottlenecked upstream (tokenization, network). Raise `--max-num-seqs`. That diagnosis - memory-bound vs compute-bound vs under-batched - is the single most valuable thing to be able to say out loud. --- ## When tuning is not the answer - Model does not fit one GPU -> **tensor parallelism** (multi-GPU, cloud) or a smaller/quantized model, not knob-twiddling. - Steady traffic far above one GPU's knee -> **scale out** (more replicas / GPUs), which on Kubernetes means autoscaling by whole GPUs (Lab 06). - Bursty/low traffic -> **serverless** (Modal, Lab 01), where scale-to-zero beats a tuned always-on box on cost. --- ## What you'll be able to say in the interview - "Tuning vLLM is trading VRAM between weights and KV-cache, then finding the batch size at the knee of the throughput-vs-p99 curve - past the knee, throughput flattens and tail latency explodes." - "I quantize to free VRAM, spend it on KV-cache via gpu-memory-utilization and max-num-seqs, and benchmark each change - one knob at a time, logged to MLflow." - "I diagnose memory-bound vs compute-bound from the GPU metrics: FB_USED pinned with rising p99 means quantize or cap context; util+power pinned with flat throughput means I need a bigger GPU or tensor parallelism, not more batching." ## Artifact to capture - The throughput-vs-p99 plot across request rates, with your chosen operating point marked. - The MLflow runs comparing baseline vs quantized vs tuned configs (knobs + metrics per run). - One sentence of diagnosis: "at rate X the GPU went memory-bound (FB_USED at cap), so I quantized and raised KV-cache headroom, moving the knee from Y to Z tokens/sec."