# Lab 02 — Serve Qwen2.5-7B on SGLang and Compare to vLLM

**Purpose:** Serve the same model you already ran on vLLM, this time on SGLang, then run a quick throughput/latency comparison so you can speak to how the two inference engines differ.

**Cost & teardown:** A single mid-range GPU (A10/A6000/L40S/4090-class, 24GB+) on Vast.ai or RunPod runs about $1-2/hr. This lab is ~30-45 min of GPU time. STOP or DESTROY the instance the moment you capture your numbers - idle GPUs bill by the second.

> Assumes you have ALREADY completed the vLLM serving lab (A100_VLLM_BUILD.md). This is the "serve the same model on the other major engine and compare" lab.

---

## 0. Prereqs

- A rented single-GPU box with NVIDIA drivers + Docker + the NVIDIA Container Toolkit (RunPod/Vast GPU templates ship this).
- Model: `Qwen/Qwen2.5-7B-Instruct` (fits comfortably in 24GB).
- Confirm the GPU is visible:

```bash
nvidia-smi
```

> Version note: SGLang launch flags and image tags change often. Before you run anything, verify the current launch command and image tag against the SGLang docs and GitHub (https://github.com/sgl-project/sglang). Do NOT invent flags - if a flag below is stale, use the one the docs show.

---

## 1. Start the SGLang server

SGLang serves an OpenAI-compatible API on port 30000 by default. Two ways to launch.

**Option A - official Docker image (preferred):**

```bash
docker run --gpus all --rm \
  -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --ipc=host \
  lmsysorg/sglang:latest \
  python3 -m sglang.launch_server \
    --model-path Qwen/Qwen2.5-7B-Instruct \
    --host 0.0.0.0 --port 30000
```

**Option B - pip install:**

```bash
pip install "sglang[all]"
python -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 --port 30000
```

Wait for the log line saying the server is ready (it downloads weights on first run).

---

## 2. Curl the OpenAI-compatible endpoint

```bash
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "In one sentence, what is RadixAttention?"}],
    "max_tokens": 128
  }'
```

You should get a normal chat completion JSON back. Same shape as vLLM - that is the point of the OpenAI-compatible surface.

---

## 3. Serve the SAME model on vLLM

In a second shell (keep SGLang running - if you only have 24GB, tear SGLang down first and run these back-to-back instead of side-by-side):

```bash
docker run --gpus all --rm \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 --port 8000

# smoke test
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen2.5-7B-Instruct","messages":[{"role":"user","content":"hello"}],"max_tokens":32}'
```

---

## 4. Quick throughput / latency check

The point is NOT to win a benchmark. It is to get real numbers on YOUR box you can talk about. Run the same load against each port.

**Simple concurrent curl loop (works against either engine, just change the port):**

```bash
PORT=30000   # 8000 for vLLM
time ( for i in $(seq 1 20); do
  curl -s http://localhost:$PORT/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"Qwen/Qwen2.5-7B-Instruct","messages":[{"role":"user","content":"Write a haiku about GPUs."}],"max_tokens":64}' \
    -o /dev/null &
done; wait )
```

Record the wall-clock time for each port.

**Better - use the bundled benchmark tools** (identical model, so it is apples-to-apples):

```bash
# SGLang (verify flags against current docs)
python -m sglang.bench_serving --backend sglang \
  --model Qwen/Qwen2.5-7B-Instruct --port 30000 \
  --num-prompts 100

# vLLM ships benchmark_serving.py in its repo; point it at port 8000
# e.g. python benchmarks/benchmark_serving.py --backend vllm \
#   --model Qwen/Qwen2.5-7B-Instruct --port 8000 --num-prompts 100
```

Capture requests/sec, mean latency, and p99 (or TTFT if the tool reports it) for both.

---

## 5. What makes SGLang distinct

Both SGLang and vLLM are peers: both do paged KV-cache (PagedAttention-style) + continuous batching. SGLang adds:

- **RadixAttention** - automatic KV-cache reuse across requests that share a prefix. It keeps a radix tree of cached prefixes, so a shared system prompt, few-shot examples, or an agent's growing context are computed once and reused. Big win when many requests share a long prefix.
- **Structured-generation frontend** - constrained / JSON / regex-guided decoding as a first-class feature for reliable structured output.
- Both are production-grade OpenAI-compatible servers - this is engine choice, not a toy-vs-real gap.

**When you would reach for each:**

- **SGLang** - heavy shared-prefix workloads (agents, RAG with a big fixed system prompt, few-shot prompting) and structured/JSON output at scale.
- **vLLM** - the broad, safe default: widest model + hardware coverage, huge community, most integrations.

---

## 6. What you'll be able to say in the interview

1. "SGLang's RadixAttention automatically reuses the KV cache across requests that share a prefix, so shared system prompts and few-shot/agent contexts don't get recomputed - that is where it pulls ahead of a stock paged-KV setup."
2. "For heavy shared-prefix or structured-output workloads I'd reach for SGLang; for a broad default with the widest model and hardware coverage I'd start with vLLM. They're peers, not tiers."
3. "I've served the same Qwen2.5-7B model on both engines against their OpenAI-compatible endpoints and benchmarked them on the same GPU, so I can speak to the tradeoff from having actually run it, not just read about it."

---

## 7. Artifact to capture

- Terminal output showing BOTH servers returning a valid `/v1/chat/completions` response (SGLang :30000, vLLM :8000).
- Your quick comparison numbers side by side (req/sec, mean latency, p99/TTFT) with the GPU type noted.
- One sentence on what YOU observed - e.g. "on an L40S with a shared 500-token system prompt, SGLang's prefix reuse showed up as lower TTFT under concurrency."

---

## 8. Teardown (do NOT skip)

```bash
docker ps          # find the running containers
docker stop <id>   # stop SGLang and vLLM
nvidia-smi         # confirm GPU memory is freed
```

Then STOP or DESTROY the rented instance in the Vast.ai / RunPod console. Verify it shows stopped so billing halts.