# Lab 05 — NVIDIA NIM + Triton Inference Server on a Rented GPU Box A hands-on, runnable lab for standing up NVIDIA NIM microservices and NVIDIA Triton Inference Server on a single rented A100/L40S-class GPU. Written for an experienced DevOps/infra engineer who knows Linux and Docker cold but is newer to the NVIDIA GPU stack. Goal: make your NIM/Triton resume claims interview-proof by actually running both. Estimated time: 60-90 minutes. Estimated cost: ~$1-2/hr for a single GPU on Vast.ai or RunPod. STOP the instance when you finish (see teardown). --- ## 0. Provision the box (Vast.ai or RunPod) 1. Rent a single-GPU instance: A100 40/80GB or L40S is ideal. 24GB (e.g. L4/A10) works for small models but some NIMs need more VRAM - check the NIM's NGC page for VRAM requirements. 2. Pick a template that already includes the NVIDIA Container Toolkit and a recent CUDA driver. On RunPod, any "CUDA 12.x" or "PyTorch" base template is fine. On Vast.ai, filter for a CUDA 12.x image. 3. SSH in and confirm the GPU + toolkit are visible to Docker before anything else: ```bash nvidia-smi # driver + GPU present on the host docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi ``` If the second command prints the GPU table, `--gpus all` works and you can proceed. If it errors, the NVIDIA Container Toolkit is not wired into Docker - fix that first (pick a different template; it is rarely worth debugging by hand on a rented box). --- ## PART A - NVIDIA NIM (NVIDIA Inference Microservices) ### What NIM is NIM is a set of prebuilt, GPU-optimized, containerized inference microservices published by NVIDIA. Each NIM wraps a model behind an **OpenAI-compatible HTTP API** (`/v1/chat/completions`, `/v1/completions`, `/v1/models`). Under the hood NVIDIA has already done the hard optimization work - selecting and tuning a TensorRT-LLM or vLLM engine profile for your specific GPU - so you get a production-grade, supported endpoint without building or tuning an inference stack yourself. The operational pitch: NIM is the **fastest path to an optimized, supported endpoint**. You trade some control for batteries-included performance and NVIDIA support. ### Prerequisite: NGC account + API key (this is the real gate) NIM images live in NVIDIA's private registry `nvcr.io`, not Docker Hub. You need: 1. A free **NGC account** at https://ngc.nvidia.com (sign up / log in). 2. An **NGC API key**: NGC profile -> Setup -> Generate API Key. Save it - it is shown once. 3. Honest caveat: some NIMs are **gated**. Access is generally available via the NVIDIA Developer Program / NGC, but specific models may require you to accept terms or request approval on that model's catalog page before the pull succeeds. If a pull returns `unauthorized` or `not found`, that is the access gate, not a typo. Log Docker into `nvcr.io` using your key. The username is the literal string `$oauthtoken`: ```bash export NGC_API_KEY=nvapi-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin ``` ### Run a NIM LLM container IMPORTANT: NIM image names and tags change often, and each NIM's exact `docker run` command is published on its own NGC catalog page. **Do not trust a hardcoded tag from this doc.** Go to the NIM's NGC page, copy the exact image + tag + run command, and paste it. The block below is the SHAPE of that command so you understand each flag: ```bash # ILLUSTRATIVE SHAPE - get the exact image:tag from the NIM's NGC catalog page. export LOCAL_NIM_CACHE=~/.cache/nim mkdir -p "$LOCAL_NIM_CACHE" docker run -d --rm --name nim-llm \ --gpus all \ --shm-size=16GB \ -e NGC_API_KEY="$NGC_API_KEY" \ -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \ -u "$(id -u)" \ -p 8000:8000 \ nvcr.io/nim//: ``` What each flag does and why it matters: - `--gpus all` - exposes the host GPU to the container (requires the NVIDIA Container Toolkit from step 0). - `-e NGC_API_KEY` - the container calls back to NGC at startup to authenticate and to select/download the optimized engine profile for your GPU. - `-v $LOCAL_NIM_CACHE:/opt/nim/.cache` - **model cache volume**. First start downloads gigabytes of weights + prebuilt engine; caching means restarts are fast and you do not re-pull. - `-p 8000:8000` - exposes the OpenAI-compatible API (8000 is the NIM default; confirm on the NGC page). - `--shm-size` - shared memory for the inference runtime; too small causes crashes on larger models. First boot can take several minutes (download + engine selection). Watch it: ```bash docker logs -f nim-llm ``` Wait for a "ready"/"Uvicorn running"/health-serving log line, then check readiness: ```bash curl -s http://localhost:8000/v1/models | jq . ``` ### Call the OpenAI-compatible endpoint Use the exact model id returned by `/v1/models` in the `"model"` field: ```bash curl -s http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "PASTE-MODEL-ID-FROM-/v1/models", "messages": [{"role":"user","content":"In one sentence, what is NVIDIA NIM?"}], "max_tokens": 64 }' | jq . ``` Because the API is OpenAI-compatible, the same endpoint works unchanged with the `openai` Python SDK by pointing `base_url` at `http://localhost:8000/v1` - a strong talking point about drop-in migration. ### NIM vs rolling your own vLLM - **NIM** = batteries-included. Prebuilt, per-GPU-tuned engine; supported by NVIDIA; OpenAI API out of the box; least effort to a production endpoint. Cost: less control over the serving internals, and you live inside NVIDIA's release cadence and access gates. - **Plain vLLM** = maximum control. You pick the model, quantization, engine args, and version; nothing is gated. Cost: you own the tuning, benchmarking, and support. Interview framing: "NIM is the fast, supported path to an optimized endpoint; I reach for hand-rolled vLLM when I need control the NIM does not expose." --- ## PART B - NVIDIA Triton Inference Server ### What Triton is Triton is a **general-purpose inference server**. Unlike NIM (one optimized model, turnkey), Triton is infrastructure you configure to serve **many models across many frameworks** from one process: TensorRT, PyTorch (LibTorch), ONNX Runtime, a Python backend for arbitrary code, and the **vLLM and TensorRT-LLM backends** for LLMs. It gives you a standard model-repository layout, dynamic batching, concurrent model execution, and a Prometheus metrics endpoint - one standardized ops surface for a whole fleet of models. ### Minimal model repository layout Triton loads everything from a **model repository** - a directory it scans. Each model gets a folder containing a `config.pbtxt` and one or more numbered version directories holding the model artifact. Minimal ONNX example: ``` model_repository/ |-- simple_onnx/ |-- config.pbtxt |-- 1/ |-- model.onnx ``` A minimal `config.pbtxt` (adjust names/shapes/dtypes to your actual model): ``` name: "simple_onnx" backend: "onnxruntime" max_batch_size: 8 input [ { name: "input", data_type: TYPE_FP32, dims: [ 3, 224, 224 ] } ] output [ { name: "output", data_type: TYPE_FP32, dims: [ 1000 ] } ] dynamic_batching { max_queue_delay_microseconds: 5000 } instance_group [ { count: 2, kind: KIND_GPU } ] ``` - `dynamic_batching {}` turns on server-side batching (see below). - `instance_group { count: 2 }` runs two concurrent instances of this model on the GPU (concurrent model execution). If you do not have an ONNX file handy, grab a small one (for example, export a torchvision resnet to ONNX, or download any public ONNX classifier) and drop it in `1/model.onnx`. The point of the lab is Triton loading a model and exposing metrics, not the model itself. For LLMs you would instead use the **vLLM backend** (a `config.pbtxt` with `backend: "vllm"` and a `model.json`) or the **TensorRT-LLM backend** - same repository pattern, different backend. ### Run Triton ```bash docker run -d --rm --name triton \ --gpus all \ --shm-size=16GB \ -p 8000:8000 \ -p 8001:8001 \ -p 8002:8002 \ -v "$PWD/model_repository:/models" \ nvcr.io/nvidia/tritonserver:-py3 \ tritonserver --model-repository=/models ``` Get the exact `-py3` tag (for example a recent monthly release) from the Triton NGC catalog page - **do not invent one**; the tags are monthly and the backend contents differ per tag. The three ports: - `8000` - HTTP/REST inference API - `8001` - gRPC inference API - `8002` - Prometheus metrics Verify it loaded your model and is serving: ```bash curl -s http://localhost:8000/v2/health/ready # 200 = ready curl -s http://localhost:8000/v2/models/simple_onnx | jq . # model metadata curl -s http://localhost:8002/metrics | head -n 40 # Prometheus metrics ``` ### The three concepts to be able to explain - **Dynamic batching** - Triton holds incoming requests for a tiny configurable window (`max_queue_delay_microseconds`) and fuses them into one larger batch before hitting the GPU. This raises throughput and GPU utilization at the cost of a small, bounded latency increase. It is server-side and transparent to clients. - **Concurrent model execution** - `instance_group` lets Triton run multiple instances of a model (and multiple different models) simultaneously on the same GPU, overlapping compute and keeping the device busy instead of idle between requests. - **Metrics endpoint (8002)** - Triton exports Prometheus metrics: per-model request counts, queue time, compute time, GPU utilization, and memory. This is what you scrape into Prometheus/Grafana for SLOs and autoscaling - the "standardized ops" argument for Triton. ### When to choose what - **NIM** - you want one specific optimized LLM endpoint, turnkey and supported. Least effort. - **Plain vLLM** - a single LLM, simplest possible self-managed serving, maximum control over that one model. - **Triton** - you are serving **multiple models and/or multiple frameworks** and want one standardized server for batching, concurrency, and metrics across all of them. Fleet-scale ops. --- ## Cost and teardown A single A100/L40S-class GPU runs roughly **$1-2/hr** on Vast.ai or RunPod. The meter runs whether or not you are inferring. When done: ```bash docker rm -f nim-llm triton 2>/dev/null || true ``` Then **STOP or DESTROY the instance in the Vast.ai / RunPod console** - stopping the containers does not stop billing for the rented box. Verify the instance shows stopped/terminated. Cached weights on ephemeral disk are lost on destroy; that is fine for a lab. --- ## What you'll be able to say in the interview 1. "I have run NVIDIA NIM microservices - authenticated to `nvcr.io` with an NGC key, launched a NIM LLM container with `--gpus all` and a model cache volume, and hit its OpenAI-compatible `/v1/chat/completions`. I can explain why NIM is the fast, supported path to a per-GPU-optimized endpoint versus rolling my own vLLM." 2. "I have stood up Triton Inference Server from a model repository - `config.pbtxt` plus versioned model dirs - and served it over HTTP and gRPC, with the multi-framework backends (TensorRT, ONNX, PyTorch, Python, vLLM/TensorRT-LLM)." 3. "I can explain dynamic batching and concurrent model execution as throughput/utilization levers, and I scraped Triton's Prometheus metrics on port 8002 for per-model latency and GPU utilization." 4. "I can make the build-vs-buy call: NIM for turnkey optimized single endpoints, plain vLLM for a single LLM with full control, Triton when I need standardized multi-model multi-framework serving with batching, concurrency, and metrics." ## Artifact to capture Save these as proof you actually ran the lab: - The NIM `/v1/models` output plus a real `/v1/chat/completions` response (redact the API key). - The `nvidia-smi` output showing GPU memory in use while the NIM/Triton container is loaded. - Triton's `/v2/health/ready` = 200, the `/v2/models/` metadata, and the first ~40 lines of `curl :8002/metrics`. - Optionally a screenshot of GPU utilization moving under a small load test. Drop these in a private gist or repo so you can reference specifics in the interview. --- Accuracy note: NGC image names, NIM tags, VRAM requirements, and NIM availability/gating change frequently. Whenever a tag or run command matters, **copy the exact command from that model's NGC catalog page** rather than trusting any tag written here.