# Lab 01 — Serve an Open-Weight LLM on Modal (Serverless GPU with vLLM)

**Purpose:** Deploy an open-weight LLM (Qwen2.5-7B-Instruct) on a rented cloud GPU using Modal's serverless platform - define the GPU and container image in Python, get an HTTP endpoint, and see cold starts plus scale-to-zero autoscaling in action.

**Cost note (read first):** Modal is the cheapest hands-on entry point for renting a GPU because it grants **monthly free compute credits** (check the current amount at modal.com/pricing - do not trust a hardcoded dollar figure, it changes). This lab is sized to run inside that free tier. Key economics:
- GPUs bill **per second, only while a container is running**.
- Idle apps **scale to zero** - you pay nothing when no requests are in flight.
- A short test session (a few requests, one model load) costs cents of credit or less.
- No monthly minimum, no cluster to keep alive, no reserved instance.

Compare that to a dedicated GPU box (billed 24/7 whether or not you use it) or a managed K8s GPU node pool (billed while nodes are up). For interview prep and bursty experiments, per-second serverless is the right cost model. **Start here — it is the cheapest way to touch a real GPU.**

---

## Mental model

You write a **normal Python file**. You decorate functions and classes to tell Modal *where* to run them:

```python
@app.function(gpu="A10G", image=my_image)
def do_work(): ...
```

Modal ships your code into a container it builds from your `Image` definition, schedules it onto a cloud GPU **on demand**, runs it, streams results back, then tears the container down when idle. There is no Dockerfile to hand-maintain, no YAML, no node pool. This is **infrastructure-as-Python**: the GPU type, the CUDA/Python image, the secrets, and the autoscaling policy are all arguments in your source file.

---

## Setup

```bash
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install modal

modal token new      # opens a browser, links this machine to your Modal account
```

`modal token new` writes credentials to `~/.modal.toml`. That is the only auth step.

> **API-drift warning:** Modal's Python API evolves. Decorator and parameter names below (`@modal.web_endpoint`, `@modal.asgi_app`, `@app.cls`, `gpu=`, `scaledown_window`/`container_idle_timeout`, `min_containers`/`keep_warm`) have changed names across versions. **Before running, confirm the current names at modal.com/docs.** The *shape* of the program below is correct; the exact identifiers are what you verify.

---

## The runnable example

Save as `app.py`. It builds an image with vLLM, loads Qwen2.5-7B-Instruct on an A10G GPU, and exposes an OpenAI-compatible HTTP endpoint.

```python
import modal

MODEL_NAME = "Qwen/Qwen2.5-7B-Instruct"
MODEL_REVISION = "main"          # pin a real commit SHA in production for reproducibility

# 1) Define the container image entirely in Python.
vllm_image = (
    modal.Image.debian_slim(python_version="3.12")
    .pip_install(
        "vllm==0.6.6",           # pin; verify a current compatible version on PyPI
        "huggingface_hub[hf_transfer]",
        "fastapi[standard]",
    )
    .env({"HF_HUB_ENABLE_HF_TRANSFER": "1"})   # faster model downloads
)

app = modal.App("vllm-qwen-lab")

# 2) Cache downloaded weights on a Modal Volume so later cold starts skip the download.
hf_cache = modal.Volume.from_name("hf-cache", create_if_missing=True)
vllm_cache = modal.Volume.from_name("vllm-cache", create_if_missing=True)

N_GPU = 1
MINUTES = 60  # seconds

with vllm_image.imports():
    from vllm import LLM, SamplingParams


@app.cls(
    image=vllm_image,
    gpu=f"A10G:{N_GPU}",         # swap to "A100" or "H100" for bigger/faster models
    volumes={
        "/root/.cache/huggingface": hf_cache,
        "/root/.cache/vllm": vllm_cache,
    },
    scaledown_window=5 * MINUTES,  # stay warm 5 min after last request, then scale to zero
    timeout=10 * MINUTES,
    # min_containers=1,           # uncomment to KEEP ONE WARM (kills cold starts, costs $)
)
class Model:
    @modal.enter()
    def load(self):
        # Runs ONCE per container start. This is the cold-start cost you will observe.
        self.llm = LLM(
            model=MODEL_NAME,
            revision=MODEL_REVISION,
            tensor_parallel_size=N_GPU,
            max_model_len=4096,
            gpu_memory_utilization=0.90,
        )

    @modal.method()
    def generate(self, prompt: str, max_tokens: int = 256) -> str:
        messages = [{"role": "user", "content": prompt}]
        params = SamplingParams(temperature=0.7, max_tokens=max_tokens)
        out = self.llm.chat(messages, params)
        return out[0].outputs[0].text


# 3) Expose a plain HTTP endpoint. Confirm 'web_endpoint' vs current name in docs.
@app.function(image=vllm_image)
@modal.fastapi_endpoint(method="POST")   # older versions: @modal.web_endpoint
def infer(data: dict):
    prompt = data.get("prompt", "Say hello in one sentence.")
    text = Model().generate.remote(prompt, data.get("max_tokens", 256))
    return {"model": MODEL_NAME, "response": text}


# Optional: a local entrypoint so `modal run app.py` works without HTTP.
@app.local_entrypoint()
def main(prompt: str = "Explain serverless GPUs in two sentences."):
    print(Model().generate.remote(prompt))
```

Notes on what each Modal-specific piece does:
- `@app.cls(...)` binds a class to a GPU container. `@modal.enter()` runs once at container startup - the natural home for the expensive model load.
- `.remote(...)` executes the method on Modal's GPU, not locally.
- The `Volume` caches Hugging Face weights, so the **first** cold start downloads ~15GB but later ones only pay model-load-into-VRAM time.
- `scaledown_window` controls how long a container lingers after the last request before scaling to zero.

---

## Run it

**Dev loop (live-reload, ephemeral):**
```bash
modal serve app.py
```
Modal prints a temporary URL and hot-reloads on file save. Great for iterating.

**Persistent deploy:**
```bash
modal deploy app.py
```
This prints a stable URL like `https://<you>--vllm-qwen-lab-infer.modal.run`. It stays deployed and scales to zero when idle.

**Hit the endpoint:**
```bash
curl -X POST https://<you>--vllm-qwen-lab-infer.modal.run \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Give me three tips for a fast vLLM cold start.", "max_tokens": 200}'
```

The **first** request triggers a cold start - watch the Modal dashboard: container spins up, GPU attaches, model loads into VRAM (tens of seconds), then responds. Fire a **second** request immediately and it returns fast (warm container). Wait past `scaledown_window` and the next request is cold again.

**Local sanity check (no HTTP):**
```bash
modal run app.py --prompt "Test."
```

---

## Ops concepts this demonstrates

- **Serverless GPU / scale-to-zero:** No requests -> zero containers -> zero GPU cost. You pay per second only during actual execution. This is the whole cost argument versus an always-on box.
- **Cold starts:** A cold start = container boot + image pull + model weights loaded into GPU memory. For a 7B model this is typically tens of seconds. Mitigations, in order of cost:
  1. **Volume-cache the weights** (done above) - removes the download from the critical path.
  2. **`scaledown_window`** - keep the container warm longer so bursts of traffic reuse it.
  3. **`min_containers` / `keep_warm`** - pin one always-warm container. Eliminates cold starts entirely but you pay to keep a GPU parked. Verify the current param name in docs.
  4. **Memory snapshotting** (if available on your Modal version) - snapshot post-load state to shrink restore time.
- **GPU selection in code:** `gpu="A10G"` -> `gpu="A100"` is a one-line change. No node pool to reconfigure, no instance type to provision. Pick the smallest GPU that fits the model in VRAM to minimize per-second cost.
- **Autoscaling for free:** Concurrent load spins up more containers automatically; idle scales them back down. You define the policy in decorator args instead of writing an HPA or a scaler.

---

## How this differs from a long-running vLLM box or K8s

| Dimension | Modal (serverless) | Dedicated GPU box | Kubernetes GPU deployment |
|---|---|---|---|
| Idle cost | Zero (scale to zero) | Full 24/7 | Node-pool cost while up |
| Cold starts | Yes - must manage | None (always warm) | Depends on scaling |
| Ops burden | None (managed) | You own the VM | You own the cluster |
| Control / tuning | Medium | Full | Full |
| Data residency | Modal's regions | Your choice | Your choice |
| Best traffic shape | Bursty / low / spiky | Steady high | Steady high, multi-service |

**When Modal fits:** bursty or low-traffic inference, fast experiments, demos, batch jobs, and anyone who wants zero cluster ops. Cheapest way to touch a real GPU.

**When a dedicated box or K8s fits:** steady high-QPS production where a GPU is busy anyway (scale-to-zero saves nothing), strict data-residency or networking requirements, or when you need full control over drivers, kernels, and scheduling.

---

## What you'll be able to say in the interview

1. "I deployed vLLM serving an open-weight 7B model on a serverless GPU platform - defining the GPU type, container image, and autoscaling policy as Python decorators, no Dockerfile or cluster."
2. "I understand the cold-start problem concretely: container boot plus loading weights into VRAM. I mitigated it by caching weights on a persistent volume and tuning the idle/scaledown window, and I know the cost tradeoff of pinning a warm container."
3. "I can reason about when serverless-GPU (bursty, scale-to-zero, per-second billing) beats an always-on box or a K8s node pool (steady high traffic, full control, data residency) - it is fundamentally a utilization and ops-burden decision."
4. "I know the OpenAI-compatible serving pattern with vLLM and how to expose it as an HTTP endpoint, so swapping the model or GPU is a one-line change."

---

## Artifact to capture

- The **deployed endpoint responding** - save the `curl` request and its JSON response.
- The **Modal dashboard** screenshot showing container runs, GPU type, and per-run duration/cost - this visibly proves scale-to-zero and per-second billing.
- Two timing samples: **cold-start latency vs warm latency** for the same prompt. That contrast is the strongest single talking point.

> Final reminder: pin your `vllm` version, pin the model revision to a commit SHA for reproducibility, and re-check every Modal decorator/parameter name against modal.com/docs before you run - the API moves.