AI Infrastructure & Platform Engineer
LLMOps + GPU / Cloud Ops — the combined role
One workspace for a single job: stand up, tune, monitor, and maintain a self-hosted + cloud LLM serving stack for a team that ranges from engineers to PhDs. It pulls together the hands-on GPU/LLMOps lab track, the concept decks behind the vocabulary, and the cert paths that feed the role — so nothing gets lost.
The honest read: the gap most people have here isn't the concepts — it's having run it. The lab track is built to close exactly that, and it's designed free-tier-first: roughly 6 of 8 labs run on a consumer 16GB GPU at $0, and you rent a data-center GPU only for MIG, multi-GPU tensor parallelism, and A100-scale numbers — a few hours of cloud time, not a whole weekend of rentals.
The role at a glance
verified July 2026Also posted asLLMOps · MLOps Platform · GPU Infrastructure · Inference Platform Engineer
Core questionCan you keep a model-serving stack up, fast, and affordable?
Primary stackvLLM/SGLang · NVIDIA (CUDA, DCGM, MIG, NIM) · Kubernetes · Terraform · Linux
Interview shapeSystems + troubleshooting, not LeetCode — 'why is this slow / OOM / pending?'
Feeder certsNCA-AIIO · CKA · Terraform Associate (none are required)
Cost to prepare$0 for most of it — 6 of 8 GPU labs run on a 16GB consumer card
Hands-on GPU / LLMOps lab track
the centerpieceTen runnable runbooks. Each opens as a raw markdown file you run the commands from in your editor. The badge says where it runs: Local-free on your own card, Cloud · free tier (Modal credits), or Rented GPU for the data-center-only labs. Start at 00, then do V (the biggest gap-closer) — Lab 01 first if you want the cheapest possible first touch of a real GPU.
00Provider Setup & Cost Discipline No GPU
Proves: You rent GPUs cost-consciously — smallest GPU that fits, rent-run-teardown in one sitting, verify the meter is off.
no GPU — read first
- ›Modal vs RunPod vs Vast, and when to use each
- ›The golden rules that prevent a surprise overnight bill
- ›A one-command sanity check for GPU + Docker + NVIDIA runtime
00bLocal-First on a 16GB Card — Save the Cloud Money Local-free
Proves: Roughly 6 of 8 labs run free on a consumer 16GB GPU; rent only for MIG, multi-GPU, and A100-scale numbers.
runs on your own card
- ›What fits 16GB (7B at 4-bit with KV-cache headroom)
- ›Docker+WSL2 GPU passthrough that matches the cloud labs
- ›The one honest scale-not-skill limitation to name in an interview
Watch first
~6mWhat is Ollama? Running Local LLMs Made Simple · IBM TechnologyPrimer — what Ollama is and why run models locally (privacy, cost, no rate limits).~11mRun AI Models Locally with Ollama: Fast & Simple Deployment · IBM TechnologyThe hands-on step up — actually pulling and running a model locally on consumer hardware.~170mOllama Course - Build AI Apps Locally · freeCodeCampDeep dive — install, Modelfile customization, REST API, Python library and real projects (RAG, agents).
01Serverless vLLM on Modal Cloud · free tier
Proves: You deploy vLLM as infrastructure-as-Python — GPU type, image, and autoscaling as decorators — and understand cold starts + scale-to-zero.
cloud, free tier
- ›Serve Qwen2.5-7B behind an OpenAI-compatible endpoint
- ›Observe cold-start vs warm latency and the cost of a pinned warm container
- ›When serverless beats an always-on box or a K8s node pool
VvLLM Core Build — Serve, Parallelize, Quantize, Tune, Benchmark Local-free
Proves: The single biggest gap-closer: turns "I read about vLLM" into "I ran it" — serving, tensor parallelism, quantization, KV/batch tuning, and a benchmark sweep.
mostly local; 2-GPU step needs cloud
- ›Read a vLLM startup log: KV-cache blocks + GPU-memory-utilization
- ›Watch VRAM drop across FP16 vs AWQ vs FP8
- ›Benchmark tokens/sec, TTFT, p99 across a request-rate sweep
Watch first
~17mvLLM Serving Tutorial: High-Performance LLM Inference with Paged Attention and LoRA · Ready TensorA practical vLLM serving walkthrough — launching an open-weight model and the serving flags this lab builds on.~18mUnderstanding vLLM with a Hands On Demo · KodeKloudA second hands-on pass at launching and serving a model with vLLM, architecture explained.~15mContinuous Batching: Optimize LLM Serving Throughput and Latency · Ready TensorReinforces the serving knobs (batching/throughput vs latency) you tune when launching a vLLM server.
02SGLang vs vLLM Local-free
Proves: You've served the same model on both engines and can speak to the trade-off from having run it — RadixAttention, structured output, peers not tiers.
runs on your own card
- ›RadixAttention: automatic KV reuse across shared prefixes
- ›Benchmark both engines apples-to-apples on one GPU
- ›When to reach for SGLang vs vLLM
Watch first
~16mSGLang Serving Tutorial: Build Structured Agentic LLM Applications · Ready TensorA hands-on intro to SGLang and its structured-output / constrained-generation features that distinguish it from vLLM.~19mSGLang Cookbook Full Tutorial: Your One-Click Model Deployment Guide · LMSYS OrgHands-on model deployment straight from SGLang's own creators (LMSYS) — the one-click serving workflow, complementing the primer above.
03GPU Monitoring: DCGM → Prometheus → Grafana Local-free
Proves: You stand up real GPU observability and read it — FB_USED vs FB_FREE for memory pressure, compute-bound vs memory-bound, what to alert on.
runs on your own card
- ›DCGM exporter needs --gpus all + SYS_ADMIN
- ›Import the community DCGM dashboard (ID 12239) and watch it move under load
- ›In K8s the GPU Operator ships this as a DaemonSet
04MIG — Partition an A100 into Isolated Slices Rented GPU
Proves: You've enabled MIG on a real A100, carved isolated instances, and served a model pinned to one slice — the hardware-isolation claim, defensible.
A100/H100 only — rent it
- ›MIG (hardware isolation) vs time-slicing (oversubscription)
- ›Profile naming (1g.10gb …) and pinning by MIG-UUID
- ›How the GPU Operator makes slices schedulable in K8s
Watch first
~11mGPU Partitioning Explained: MIG vs Virtual Compute Server · Parallel RoutinesApproved-channel visual primer on Multi-Instance GPU — how MIG carves one physical GPU into isolated instances, contrasted with a vGPU / virtual compute server.~10mTutorial: Run multiple workloads using a single GPU · LambdaPractical demo of enabling MIG and carving one A100 into isolated instances for concurrent workloads.
05NVIDIA NIM + Triton Inference Server Rented GPU
Proves: You've run NVIDIA's prepackaged NIM microservices and Triton — the batteries-included, supported serving path — so the NIM/Triton claims are real.
Triton local; many NIMs need a data-center GPU
- ›NIM = prebuilt, GPU-optimized, OpenAI-compatible microservices from nvcr.io
- ›NGC account + API key is the real gate
- ›Triton vs NIM vs vLLM — packaging, optimization, lock-in
Watch first
~10mIntro to NVIDIA NIM for AI Builders · NVIDIA DeveloperOfficial NVIDIA intro to running optimized NIM inference-microservice containers.~15mLLM Inference Deep Dive: TensorRT-LLM, KV Cache, Prefill vs Decode, TTFT, TPOT · PreporatoThe inference engine (TensorRT-LLM) that NIM/Triton are built on — KV cache, prefill/decode and the latency metrics they optimize.~22mBuilding Inference-as-a-Service on Kubernetes · DevOps ToolkitA containerized inference-serving platform on Kubernetes — the same architectural role NIM microservices / Triton fill.
06vLLM on Kubernetes (GPU) — the combined centerpiece Local-free
Proves: The LLMOps + Cloud-Infra-Ops centerpiece: schedule a vLLM pod against nvidia.com/gpu on a real GPU node and curl the endpoint.
local k3s/kind, or managed GKE
- ›GKE GPU node pool path (closest to most JDs) or local k3s + device plugin
- ›Request GPU as a pod resource limit
- ›The NVIDIA device plugin / GPU Operator wiring
Watch first
~7mWhat is vLLM? Efficient AI Inference for Large Language Models · IBM TechnologyPrimer on vLLM — the serving engine — before you deploy it on a cluster.~13mGPUs in Kubernetes for AI Workloads · DevOps ToolkitHands-on: create GPU nodes and schedule nvidia.com/gpu pods on Kubernetes — the exact mechanics this lab targets.~18mUnderstanding vLLM with a Hands On Demo · KodeKloudServing a model on GPU with vLLM — bridges the primer to a k8s deployment.~22mBuilding Inference-as-a-Service on Kubernetes · DevOps ToolkitDeeper dive — a full self-hosted GPU inference platform on Kubernetes (routing, scaling, serving): the production shape of vLLM-on-k8s.
07Performance Tuning & Benchmarking Playbook Local-free
Proves: The "tuning" skill made explicit: which vLLM knob to turn, when, and how to read the result — the loop of change-one-knob, benchmark, decide.
method matters, not scale — runs local
- ›The three things competing for VRAM (weights, KV-cache, overhead)
- ›Each vLLM knob and when to turn it up vs down
- ›Throughput vs TTFT vs TPOT vs p50/p99 — what you're optimizing
Watch first
~11mLLM Inference Performance: Latency and Throughput Metrics · Ready TensorBreaks down TTFT, inter-token latency, end-to-end latency and throughput/tokens-per-sec for benchmarking.~15mLLM Inference Deep Dive: TensorRT-LLM, KV Cache, Prefill vs Decode, TTFT, TPOT · PreporatoTies the latency metrics (TTFT/TPOT, prefill vs decode) to the engine internals that drive them.~15mContinuous Batching: Optimize LLM Serving Throughput and Latency · Ready TensorThe throughput-vs-latency tradeoff that inference benchmarking measures.
Runbooks open as raw markdown — run the commands from your editor. Every provider API, image tag, and cloud flag moves fast; each runbook tells you to verify against current docs rather than trusting a hardcoded value.
Concept decks
the vocabulary + depthThe reading behind the labs — drill these so you can explain weights vs vectors, tokenization → attention → decoding, PagedAttention/KV-cache, MIG/DCGM, and the GCP ramp out loud.
The Numbers: Weights, Parameters, Vectors & Embeddings (the vocabulary everyone assumes you know)16 cardsThe vocabulary that trips people up — weights/params vs vectors/embeddings, tokens, VRAM, quantization.
Watch first
~6mHow Large Language Models Work · IBM TechnologyQuick primer framing params/weights and how a trained model turns input into predictions.~9mWhat are Word Embeddings? · IBM TechnologyExplains embeddings/vectors as numeric representations — the other half of 'the numbers', distinct from raw weights.~9mHow KV Cache Speeds Up LLMs for Faster AI Models on GPUs · IBM TechnologyConnects model size to GPU VRAM/precision — the practical 'numbers' side of running LLMs. (No approved quantization primer exists yet.)
Transformers & 'Attention Is All You Need' (self-attention, the architecture, and how modern LLMs evolved)30 cardsUnder-the-hood fluency: self-attention / Q-K-V, the architecture, and the O(n²) cost that leads into the KV-cache.
Watch first
~8mLarge Language Models explained briefly · 3Blue1BrownShort visual primer on what an LLM is and how next-token prediction works before diving into attention.~27mTransformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1BrownBuilds the full Transformer picture (tokens, embeddings, data flow) with 3B1B's signature visualizations.~26mAttention in transformers, step-by-step | Deep Learning Chapter 6 · 3Blue1BrownThe definitive visual walkthrough of self-attention — queries, keys, values and multi-head attention.~130mBuild an LLM from Scratch 3: Coding attention mechanisms · Sebastian RaschkaHands-on deep dive coding self-, causal- and multi-head attention from scratch in PyTorch.
vLLM Production Serving (PagedAttention, batching, tensor parallelism, quantization)35 cardsThe core serving engine — PagedAttention, continuous batching, tensor/pipeline parallelism, AWQ/GPTQ/FP8, benchmarking, plus SGLang.
Watch first
~7mWhat is vLLM? Efficient AI Inference for Large Language Models · IBM TechnologyPrimer on vLLM and why PagedAttention/continuous batching make serving efficient.~9mHow KV Cache Speeds Up LLMs for Faster AI Models on GPUs · IBM TechnologyExplains the KV-cache / paged-attention idea vLLM is built around.~15mContinuous Batching: Optimize LLM Serving Throughput and Latency · Ready TensorFocused deep dive on continuous batching — the core scheduling technique behind vLLM's throughput.~12mStatic Batching: Why Your GPU Is Sitting Idle During LLM Inference · Ready TensorOptional context — the naive-batching problem that continuous batching / PagedAttention solve.
LLM App Orchestration (LangChain, LangGraph, LlamaIndex, agents, RAG)34 cardsThe app layer above serving — LangChain/LangGraph/LlamaIndex, agents, and RAG orchestration.
Watch first
~8mWhat is LangChain? · IBM TechnologyPrimer on the core orchestration framework — chains, prompts, memory, and how components wire an LLM app together.~12mWhat are AI Agents? · IBM TechnologyThe agent loop (reason, use tools, act) sitting on top of the chain layer — the bridge from chains to agents.~9mLangChain vs LangGraph: A Tale of Two Frameworks · IBM TechnologyContrasts linear chains with LangGraph's stateful graph execution for multi-step agentic workflows.~8mWhat is Agentic RAG? · IBM TechnologyDeeper dive — how the RAG retrieval layer folds into agent orchestration (the full application layer).
NVIDIA GPU Ops Stack (CUDA, Triton, DCGM, MIG, NIM)39 cardsThe NVIDIA GPU-ops stack: CUDA, Container Toolkit, GPU Operator, DCGM, MIG, NIM/Triton.
Watch first
~6mParallel Processing: Why GPUs Dominate AI · PreporatoShort primer on why GPU parallelism (thousands of cores, matrix math) is the foundation of the whole NVIDIA AI stack.~13mGPUs in Kubernetes for AI Workloads · DevOps ToolkitHands-on core of the stack on Kubernetes — GPU nodes, driver/device-plugin exposure, and consuming nvidia.com/gpu.~20mThe Engineering Behind LLM Inference: Inside the GPU · The Commit LogDeeper dive into GPU internals (SMs, memory hierarchy, execution) so the driver/CUDA/MIG layers make sense, not just how to install them.
AI/ML Vocabulary51 cardsThe shared AI/ML language — training vs inference, params, precision, transformers.
Watch first
~10mAI, Machine Learning, Deep Learning and Generative AI Explained · IBM TechnologyNests all four terms into one hierarchy (AI > ML > DL > GenAI) — ideal orientation before the vocabulary deck.~8mMachine Learning vs Deep Learning · IBM TechnologyPins down the ML-vs-DL boundary specifically — the distinction learners most often blur.~9mMachine Learning vs. Deep Learning vs. Foundation Models · IBM TechnologyExtends the vocabulary chain to foundation models and LLMs, connecting classic ML terms to the GenAI era.
NVIDIA NCA-AIIO: AI Infrastructure & Operations (Associate)50 cardsThe NCA-AIIO cert deck — AI infrastructure & operations, domain-weighted.
Watch first
~7mGPUs: Explained · IBM TechnologyCert-altitude primer on GPUs vs CPUs, cores and parallelism — the opening fundamentals of the NCA-AIIO blueprint.~5mCPU vs GPU vs TPU · ByteByteGoShort animated take on why GPUs/TPUs beat CPUs for parallel matrix math — the core hardware intuition.~14mNVIDIA AI Certifications Explained: Which One Should You Get? (2026) · PreporatoOverview of the NVIDIA cert ladder incl. NCA-AIIO — scope and what infra/ops fundamentals it covers, from a dedicated cert-prep channel.~234mNVIDIA-Certified Associate AI Infrastructure and Operations (NCA-AIIO) Free Study Course · freeCodeCampOptional deep dive — a full ~4h NCA-AIIO study course covering GPUs, CUDA, networking, containers and Triton at exam depth.
GCP for Cloud Infrastructure Engineers (projects, IAM, VPC networking, GKE, ops)34 cardsGCP cloud-infrastructure fundamentals — the one cloud ramp for the combined role.
Watch first
~6mNever Feel Lost in GCP Again #GCPSketchnote · The Cloud GirlA concise big-picture orientation to Google Cloud's core services from a Google Cloud advocate.~390mGoogle Cloud Digital Leader Certification Course · freeCodeCampDeeper dive — a foundational tour of GCP core services, resource hierarchy and cloud concepts (incl. GKE).
Skill map — where to practice each thing
role skill → the repServe an LLM and tune throughputLab V — vLLM core buildvLLM Serving deck
Deploy serverless / autoscaled inferenceLab 01 — serverless vLLM on Modal
Partition and share a GPU between tenantsLab 04 — MIGGPU Ops Stack deck
Monitor GPU health at fleet scaleLab 03 — DCGM monitoring
Run inference on KubernetesKubernetes labsCKA path
Containers underneath all of itDocker lab
Write the infrastructure as codeTerraform pathHCL drills
Configure nodes and fleetsAnsible pathYAML drills
Handle secrets without leaking themSecrets drillsSecrets Management
Schedule jobs on a shared GPU clusterSlurm lab
Explain the model layer crediblyTransformers deckThe Numbers deck
Say it out loud under pressureAI Infra interview drill
A study routine
sustainableSession 1 — before anything elseRead Lab 00 and 00b. Knowing what runs free on your own card, and how to rent-run-teardown without a surprise bill, decides how far the rest of this goes.
Weekly — one labOne runbook per week, start to finish, on real hardware. The lab track is the centrepiece; the decks exist to make the labs make sense, not the other way round.
Daily — 10 minutes of recallRun whatever is due in the review queue. The concept decks are ordered on the subject hub — work down them rather than picking at random.
Alongside — one cert at a timePick a single feeder cert and finish it. CKA if you want the strongest signal, Terraform Associate if you want the fastest win.
After each labWrite two sentences on what broke and how you found it. That is the answer you will actually be asked for — nobody asks you to recite what PagedAttention is.
Cert & skill paths that feed this role
go deeper NVIDIA AI Infrastructure & OperationsThe GPU-infra ops spine — NCA-AIIO → CKA → NCP-AIO, labs, and reading. Certified Kubernetes Administrator (CKA)GPU orchestration runs on Kubernetes — the hands-on break-fix cert. Terraform Associate (003)Infrastructure-as-code for GPU infrastructure. Ansible AutomationAgentless config management for node/fleet operations. AWS ML Engineer – Associate (MLA-C01)Operating ML on AWS (SageMaker-centric). Google Cloud PMLEThe ML lifecycle on Vertex AI — the GCP side of the combined role. AI-300 — Azure ML OperationsMLOps + GenAIOps on Azure ML + Foundry. AZ-104 Azure AdministratorCore Azure administration — the strongest cloud base.
Private interview prep
owner onlyThese are personal, sign-in-gated prep workspaces — they map a specific job description and resume to the content above. Linked here for completeness; they show a lock unless you own them.