# Lab 07 — GPU Monitoring: DCGM Exporter → Prometheus → Grafana Under LLM Load

**Purpose:** Stand up a real GPU observability stack (NVIDIA DCGM exporter scraped by Prometheus, visualized in Grafana) and watch the metrics move while an LLM is being served on the GPU.

**Cost & teardown:** A single GPU on Vast.ai or RunPod runs roughly $1-2/hr. This lab takes about 1-2 hours. When you are done capturing artifacts, STOP or DESTROY the instance from the provider console. A stopped-but-not-destroyed instance may still bill for storage; destroy it if you are finished. Do not leave it running overnight.

---

## Prerequisites

- A running single-GPU box (Vast.ai or RunPod) with SSH access.
- Docker installed and the NVIDIA Container Toolkit configured so `--gpus all` works. Verify:
  ```bash
  nvidia-smi
  docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
  ```
  Both should print your GPU. If the second fails, fix `nvidia-container-toolkit` before continuing (verify with the NVIDIA Container Toolkit install docs for your distro).
- An LLM already serving on the GPU. This lab assumes the companion vLLM lab has a model listening on port 8000. If you do not have one running, start a small model quickly:
  ```bash
  docker run -d --gpus all -p 8000:8000 --name vllm \
    vllm/vllm-openai:latest \
    --model Qwen/Qwen2.5-0.5B-Instruct
  ```
  (Any served model works; the point is to put load on the GPU.)

> Note on ports: cloud GPU providers firewall most ports by default. On Vast/RunPod you must map or expose ports through their console (or use SSH tunnels). Metrics (9400), Prometheus (9090), and Grafana (3000) are all reached most safely over an SSH tunnel from your laptop rather than opened to the internet.

---

## Step 1 - Run the DCGM exporter

DCGM (Data Center GPU Manager) is NVIDIA's official GPU telemetry daemon. `dcgm-exporter` wraps it and exposes metrics in Prometheus format on port 9400.

Use the official image. NVIDIA publishes it on NGC as `nvcr.io/nvidia/k8s/dcgm-exporter` and mirrors it on Docker Hub as `nvidia/dcgm-exporter`. Pin a tag (do not rely on `latest`):

```bash
docker run -d --rm \
  --gpus all \
  --cap-add SYS_ADMIN \
  -p 9400:9400 \
  --name dcgm-exporter \
  nvcr.io/nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04
```

Notes:
- `--gpus all` gives the container the GPU. `--cap-add SYS_ADMIN` is required by DCGM to read certain low-level profiling/telemetry counters. If your tag or provider needs a different capability set, verify with the DCGM exporter docs rather than guessing.
- Check available tags on the NGC/Docker Hub page and pick a current one; the exact `3.3.9-3.6.1-...` string above is an example and may be superseded.

Confirm it is up:
```bash
docker logs dcgm-exporter --tail 20
```

## Step 2 - Verify metrics at /metrics

```bash
curl -s http://localhost:9400/metrics | head -40
```
You should see lines like `DCGM_FI_DEV_GPU_UTIL`, `DCGM_FI_DEV_FB_USED`, `DCGM_FI_DEV_POWER_USAGE`, each with a `{gpu="0",...}` label and a numeric value. If curl returns nothing, re-check the container logs and that port 9400 is published.

## Step 3 - Run Prometheus to scrape the exporter

Create a scrape config. `172.17.0.1` is the default Docker bridge gateway, which lets the Prometheus container reach the exporter published on the host. (If that does not resolve on your box, put all three containers on one user-defined Docker network and scrape by container name instead.)

```bash
mkdir -p ~/gpu-mon
cat > ~/gpu-mon/prometheus.yml <<'EOF'
global:
  scrape_interval: 5s
scrape_configs:
  - job_name: dcgm
    static_configs:
      - targets: ['172.17.0.1:9400']
EOF
```

Run the official Prometheus container:
```bash
docker run -d --rm \
  -p 9090:9090 \
  -v ~/gpu-mon/prometheus.yml:/etc/prometheus/prometheus.yml \
  --name prometheus \
  prom/prometheus:latest
```

Verify the target is UP: open `http://localhost:9090/targets` (over your SSH tunnel), or:
```bash
curl -s 'http://localhost:9090/api/v1/query?query=DCGM_FI_DEV_GPU_UTIL' | head
```

## Step 4 - Run Grafana and import the DCGM dashboard

```bash
docker run -d --rm \
  -p 3000:3000 \
  --name grafana \
  grafana/grafana:latest
```

Then, over an SSH tunnel, browse to `http://localhost:3000` (default login `admin` / `admin`, change on first login):

1. Add a data source: Connections -> Data sources -> Prometheus. Set URL to `http://172.17.0.1:9090` (or the Prometheus container name if you put them on one network). Save & test.
2. Import the dashboard: Dashboards -> New -> Import -> enter dashboard ID **12239** ("NVIDIA DCGM Exporter Dashboard", the community/official DCGM board on grafana.com). Select your Prometheus data source and import.

You now have a live GPU dashboard.

## Step 5 - Put load on the GPU and watch it move

Drive the served model so metrics react:
```bash
for i in $(seq 1 200); do
  curl -s http://localhost:8000/v1/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","prompt":"Explain GPU memory bandwidth in one paragraph.","max_tokens":128}' \
    >/dev/null &
done; wait
```
Watch the Grafana panels and `watch -n1 nvidia-smi` side by side. Utilization, power, and framebuffer-used should climb during generation.

---

## What the key metrics mean for an LLM server

- **`DCGM_FI_DEV_GPU_UTIL`** - percent of time the GPU was executing at least one kernel. High and sustained (near 100%) during inference means the GPU is busy. Note: this is a coarse "busy or not" signal, not true compute efficiency.
- **`DCGM_FI_DEV_FB_USED` / `DCGM_FI_DEV_FB_FREE`** - framebuffer (VRAM) used and free, in MiB. For LLM serving this is the number that matters most: model weights + KV cache live here. FB_USED sitting near total VRAM with FB_FREE near zero means you are **memory-bound** and one more concurrent request or a longer context can trigger an out-of-memory error.
- **SM occupancy / activity** (e.g. `DCGM_FI_PROF_SM_OCCUPANCY`, `DCGM_FI_PROF_SM_ACTIVE`, exposed when profiling metrics are enabled) - how much of the streaming-multiprocessor capacity is actually engaged. Utilization can read 100% while occupancy is low, which hints the workload is latency/memory-bound rather than compute-bound. Confirm exact field names against your exporter's metric list, as available profiling fields depend on GPU and DCGM version.
- **`DCGM_FI_DEV_POWER_USAGE`** (watts) - real work indicator. A GPU pinned at its power cap during inference is genuinely saturated.
- **`DCGM_FI_DEV_GPU_TEMP`** (Celsius) - thermal headroom. Rising temperature toward the throttle threshold predicts clock throttling and latency spikes.

Reading these together: **compute-bound** looks like high UTIL + high power + high SM activity; **memory-bound** looks like FB_USED pinned near cap with UTIL/power lower than you would expect. That distinction drives whether you scale by batching, by a bigger GPU, or by sharding the model.

---

## What you would ALERT on in production

- **VRAM near cap:** `DCGM_FI_DEV_FB_USED / (FB_USED + FB_FREE) > 0.9` sustained -> risk of OOM-killed requests. Page or autoscale before you fall off the cliff.
- **Thermal throttling / high temp:** `DCGM_FI_DEV_GPU_TEMP` above a board-specific threshold, or any throttle/clock-reduction signal from DCGM -> latency degradation and hardware risk.
- **Utilization pattern anomalies:** utilization pinned at 100% with a growing request queue (serving-side metric) means saturation; utilization near 0% on a box you are paying for means wasted/idle GPU. Both are worth alerting on for different reasons (capacity vs. cost).
- **XID / hardware errors:** DCGM surfaces GPU health and XID error events; alert on any, they often precede a dead GPU.

How DCGM fits a real ops stack: in Kubernetes you do not run these containers by hand. The **NVIDIA GPU Operator** deploys `dcgm-exporter` as a DaemonSet automatically on every GPU node, Prometheus (often via kube-prometheus-stack) scrapes it, and Grafana dashboards plus Alertmanager rules ride on top. This lab is the single-node, hand-wired version of exactly that production pattern, which is why doing it by hand is worth it.

---

## What you will be able to say in the interview

- "I stood up DCGM exporter -> Prometheus -> Grafana against a GPU serving an LLM, and I can read the dashboard: I use FB_USED vs FB_FREE to catch memory pressure before OOM, and I compare utilization against power and SM activity to tell compute-bound from memory-bound."
- "I know DCGM is NVIDIA's telemetry layer and that `dcgm-exporter` needs `--gpus all` plus `SYS_ADMIN` to read the low-level counters, and that in Kubernetes the GPU Operator ships it as a DaemonSet so you get this for free per node."
- "For GPU alerting I focus on VRAM saturation, thermal throttling, and idle-versus-pinned utilization patterns, because those map directly to reliability risk and to GPU spend."
- "I can wire the same stack into kube-prometheus-stack and Alertmanager, so this scales from one rented box to a fleet without changing the mental model."

---

## Artifacts to capture

- Screenshot of the **Grafana DCGM dashboard (ID 12239) under active LLM load** - utilization, power, and framebuffer panels visibly elevated.
- Screenshot or saved text of **`curl http://localhost:9400/metrics`** showing the raw `DCGM_FI_DEV_*` lines.
- Optional: a side-by-side of `nvidia-smi` and the Grafana panels during the load loop, to prove the dashboard tracks reality.

## Teardown

```bash
docker rm -f grafana prometheus dcgm-exporter vllm
```
Then STOP or DESTROY the GPU instance in the Vast.ai / RunPod console. Confirm in the console that billing has stopped.