AI curriculum
Course outline for AI Infrastructure
Operate, measure, size, and choose inference infrastructure. Start a vLLM service on a prepared GPU host. Change its serving configuration and measure the effects on memory, latency, and useful throughput. Establish a bounded operating envelope. Choose compute from workload, data, persistence, availability, and effective cost.
AI Infrastructure Course Objectives
- Configure vLLM from first use and verify actual model readiness and responses.
- Check weight and context-memory estimates against measured GPU and runtime observations.
- Measure batching, concurrency, queueing, latency, and useful throughput under controlled workloads.
- Establish safe engine limits and verify bounded overload, restart, and infrastructure recovery.
- Compare notebook, specialist-GPU, and major-cloud compute against documented workload constraints.
- Defend capacity and placement using effective cost, measured limits, and explicit unknowns.
Pre-requisites
- Basic model inference, tokens, context windows, and output limits.
- Python scripts, HTTP requests, JSON responses, terminal use, and basic tests.
- Cloud compute, storage, networking, permissions, quotas, and usage-cost arithmetic.
- No prior vLLM operation or GPU programming is assumed.
- A prepared authorized supported GPU host is required; remote access is valid.
- No personal GPU purchase is required. GPU access and budget remain unapproved.
Before this course
Complete the required courses or bring equivalent experience. Recommended courses add useful depth; they are not enrolment gates.
Required preparation
Recommended preparation
- Local and Small Language Models
- Cloud-Native Engineering for AI Workloads
- Platform Engineering for AI Teams
Readiness checks
- Basic model inference, tokens, context windows, and output limits.
- Python scripts, HTTP requests, JSON responses, terminal use, and basic tests.
- Cloud compute, storage, networking, permissions, quotas, and usage-cost arithmetic.
- No prior vLLM operation or GPU programming is assumed.
- A prepared authorized supported GPU host is required; remote access is valid.
- No personal GPU purchase is required. GPU access and budget remain unapproved.
Basic model inference, tokens, context windows, and output limits.
Find your place in the learning mapLab Setup
Delivery requirements: the environment, access, and exercise materials must be prepared and verified before delivery. This outline does not certify that a lab package is ready.
Prepared authorized local or remote supported GPU host, compatible stable vLLM image and driver, licensed small model revision, tokenizer/chat template, restricted endpoint, sufficient VRAM/RAM/disk, tested request driver and sampler, safe restart/limit controls, minimal success/data rubric, independent task, dated cost profiles, and cleanup inventory. Explicit budget, quotas, stop owner, compatibility, actual memory/timeout/recovery observations, and timing require verification. No personal GPU purchase is required; no silent CPU or llama.cpp replacement meets the required GPU observations.
Detailed Course Outline
Module 1: Start one inference service for a defined workload (2 hours)
- Workload and data constraints: Workload inputs, output budget, arrival pattern, minimum success, and data restrictions.
- vLLM first-use configuration: vLLM server, model revision, tokenizer, chat template, and first-use configuration.
- Prepared host compatibility: Prepared host access, supported stable runtime, driver compatibility, and resource inventory.
- Actual service requests: First start, model listing, actual request, streaming response, and readiness checks.
- Prefill decoding and warmup: Prefill versus decoding and cold versus warm request observations.
- Configuration change and run identity: One learner configuration change with a checked response and recorded run identity.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Start actual vLLM, send a checked request, and save a learner-authored configuration with its observations.
Module 2: Fit weights and growing context into memory (2 hours)
- Weights precision and overhead: Model weights, precision, supported quantization, and runtime overhead estimates.
- Context and key-value memory: Key/value cache, input context, generated tokens, and concurrent sequence demand.
- GPU and host resources: GPU VRAM, host RAM, compute, bandwidth, and observable utilization boundaries.
- Allocated and active memory: Engine cache allocation versus active token memory and reported process memory.
- Controlled memory comparison: Controlled context changes, configured limits, observed memory, and estimate reconciliation.
- Headroom and uncertainty: Supported-model constraints, headroom, safe failure avoidance, and capacity uncertainty.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Calculate a memory budget, vary context safely, and reconcile estimates with actual diagnostics and unavailable fields.
Module 3: Measure useful work as concurrency changes (2 hours)
- Batching and engine scheduling: Continuous batching, paged key/value memory, engine scheduling, and queueing.
- Client and engine limits: Engine sequence and token budgets versus supplied client concurrency controls.
- Controlled concurrency sweeps: Same-workload sweeps with fixed model, precision, output budget, and cache policy.
- Latency and useful throughput: Time to first token, completion latency, token throughput, and accepted-answer throughput.
- Queue evidence and variability: Client wait versus observable engine wait, repeated runs, errors, and measurement limits.
- Alternative-engine comparison: Compact vLLM, SGLang, and NVIDIA Triton role/mechanism comparison without new installs.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Run bounded concurrency experiments, retain per-request observations, and explain a latency/throughput trade-off.
Module 4: Establish a bounded operating envelope (2 hours)
- Operating targets and limits: Capacity targets, context and sequence caps, headroom, and stop thresholds.
- Bounded actual overload: Short bounded overload against actual inference with supplied request-driver controls.
- Complete request accounting: Completed, failed, rejected, unfinished, and timed-out request accounting without omissions.
- Restart and checked recovery: Engine/process restart, readiness, cold reload, and independently checked recovery requests.
- Timeout and backend completion: Client timeout versus observed backend completion and released resource evidence.
- Independent infrastructure check: Learner-authored infrastructure check for a new limit or recovery condition.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Choose a safe tested configuration and prove bounded overload/restart recovery without constructing an application gateway.
Module 5: Match compute to the service's constraints (2 hours)
- Google Colab boundaries: Google Colab notebook availability, session limits, persistence, and hosting restrictions.
- Vast.ai host and rental variability: Vast.ai host variability, verification tiers, historical reliability, and rental interruptions.
- Vast.ai storage and access: Vast.ai mapped ports, sensitive-data limits, container storage, volumes, and cleanup ownership.
- Runpod product contrast: Runpod Pods versus Serverless as one specialist contrast with distinct storage lifecycles.
- Major-cloud compute classes: AWS EC2, Azure GPU VM, and Google Compute Engine classes and compatibility boundaries.
- Same-workload compute matrix: Same-workload profile matrix covering VRAM, startup, quotas, security, support, and persistence.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Use sourced compute profiles to reject an unsuitable placement and distinguish experiments from long-running services.
Module 6: Price useful capacity and defend a placement (2 hours)
- Dated price identities: Dated region, currency, resource, rental-mode, billing-unit, and source identities.
- Full operating costs: Compute, startup, idle time, storage, transfer, interruption loss, and operations costs.
- Utilization and accepted-answer cost: Low/high utilization sensitivity and cost per accepted answer with stated denominators.
- Measurements and projections: Measured capacity versus unmeasured hardware projections and unknown price components.
- Independent changed-workload decision: Independent changed-workload sizing and placement decision before explanatory feedback.
- Cleanup and evidence handoff: Capacity limits, unresolved evidence, stop ownership, and verified cleanup handoff.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Submit a reproducible capacity/cost recommendation and revise it for an unanswered workload change without claiming unmeasured provider performance.
Practical exercise
- Operate one vLLM service for a small synthetic support-answer workload. The prepared host supplies supported hardware, downloaded licensed model files, a tested runtime, resource sampler, and a request driver. Teach what vLLM does, how to access the host, how its configuration names the model and limits, and how to start and inspect it. Learners author the serving configuration, change a safe setting, and send actual requests. Installation and downloads occur before class; first-use operation does not. Record exact runtime/image identity, model revision, tokenizer/template, precision, host GPU and VRAM, driver, context/sequence/token limits, cache policy, and timestamps.
- Use the same inputs, output cap, minimum answer rubric, and synthetic data rules throughout. The rubric checks task completion and known source facts. It is not a new evaluator, judge system, or benchmark-building course. Reject HTTP-only success. Change one resource setting at a time. Estimate weight and context-memory demand, then compare with reported GPU memory and runtime diagnostics. Reserved cache can stay flat when active context changes. Mark active-memory fields unavailable if the selected runtime does not expose them. Teach precision/quantization effects through estimates; actually reload another precision only if a compatible prepared variant fits the pilot. Never claim a precision effect from a different model or workload.
- Run a short fixed-workload concurrency sweep and repeat it under the same declared cold/warm conditions. Measure first-token and completion latency, generated-token throughput, useful completions, errors, and available memory/queue observations. State the arrival schedule, observation window, sample size, units and exclusions. Do not add overlapping queue and service durations. Do not infer server queue time from end-to-end latency alone. A supplied client limit is not an engine admission guarantee. In Module 3 use one compact role/mechanism card: vLLM paged cache and continuous batching, SGLang RadixAttention/prefix reuse, and NVIDIA Triton model repository/backend scheduling. These are comparison-only alternatives, not extra serving tracks or identical API/performance claims. TGI's maintenance-only status is a brief support note, not another required comparison lab.
- Apply safe engine/context/concurrency limits, then run a bounded overload and controlled process restart on the same actual service. Keep the driver capped by duration, request count, and concurrency. Record unfinished and timed-out requests instead of dropping them. Distinguish client timeout from engine cancellation or completion using only exposed evidence. Check readiness and a fresh correct response after restart. Confirm the stated resource condition independently. Do not force GPU exhaustion or infer durable application recovery from a running process. Learners author a new infrastructure assertion and retest it. The driver or a thin gateway can be supplied and tested in advance. Learners do not implement routing, retry/fallback policy, authentication frameworks, or generic application admission.
- Use one dated profile matrix for Google Colab, Vast.ai, one Runpod contrast, and AWS/Azure/GCP GPU compute classes. Work one notebook-versus-rental contrast, then use the remaining rows to test constraints, not conduct six product demonstrations. For Colab distinguish saved notebooks from VM files, variable GPU/session access, and restrictions on unrelated web hosting. It is not guaranteed 24/7 serving. For Vast.ai compare GPU/VRAM, CPU/RAM, disk, network, rental end, interruptibility, host verification and support. Historical reliability is not an SLA, and verified hardware is not confidentiality from a host operator. Secure Cloud criteria need provider-specific confirmation because the recorded official descriptions differ. Map public external ports to internal ports. Require restricted authenticated access or a verified tunnel, not an unauthenticated public inference port. Stopped container storage can still bill; destruction deletes it. Separate host-local volumes need their own billing, deletion, and off-host backup decisions. Use nonsensitive data. Runpod Pods/Serverless differ; do not transfer a Pod storage rule to Serverless.
- Compare AWS EC2 GPU instance profiles, Azure GPU VM families/sizes, and Google Compute Engine accelerator-optimized/compatible GPU-attached VMs. GPU labels alone do not establish a supported runtime, sufficient VRAM, current regional stock, quota, or price. Keep VM operation, storage, access, recovery and cleanup ownership visible. A model API is not a rented GPU VM. These are compute-selection profiles, not cloud provisioning or managed-service SDK lessons. No vendor login or rental is required for the comparison. Missing price or capacity evidence remains unknown.
- Use the measured workload and its minimum accepted-answer rule for cost arithmetic. Prepare dated authorized rate rows with explicit region, currency, SKU, rental mode, units, source, validity, and inclusions. Include billed idle/cold-start time, retained disks/volumes, transfer direction, interruption/restart work, and declared operations effort. Do not count one interval or charge twice. Compare low/high utilization and mark estimates separately from measured charges. Unknown prices are not zero; zero accepted answers produce no finite cost per accepted answer. Provider throughput is a projection unless measured on that exact supported route. No cross-provider speed or cheapest-provider ranking is implied by this one-host experiment.
- Reserve a changed context/arrival pattern or persistence/data restriction until the final independent task. The learner recalculates capacity, states where measurement no longer supports extrapolation, and accepts, rejects, or conditions a placement. Feedback follows the attempt. A second cloud endpoint or local-versus-hosted model API benchmark is not required in this revision. Check cleanup only for authorized lab-owned resources; distinguish retained model files/storage from released compute.
Delivery and timing
- Guided hours include teaching, demonstrations, learner practice, feedback, questions, and contingency as allocated above. Breaks, bulk downloads, account approval, and prerequisite setup are additional. Allocations need a timed learner pilot before delivery; they are not measured completion times.
What you will produce
Operate vLLM, measure useful inference capacity, and choose compute from workload constraints and effective cost.
How the work is checked
- Learner-authored vLLM configuration starts and produces independently checked actual responses.
- Weight/context estimates reconcile with memory observations and explicit unavailable fields.
- Same-workload sweeps retain configuration identities, repeated observations, units, and failures.
- Engine limits and bounded overload/restart tests support a stated operating envelope.
- An independent infrastructure assertion checks actual readiness, results, or resource behavior.
- Compute decisions use documented access, persistence, security, compatibility, and interruption limits.
- Effective costs state utilization, sources, unknowns, and accepted-answer denominators without double counting.
- A changed-workload placement decision states uncertainty before explanatory feedback.
12 guided hours. vLLM is the only required serving engine, taught from basics on prepared supported GPU hardware. Learners configure and measure inference, not rebuild a generic gateway. Local-model onboarding, application retry/fallback policy, Kubernetes installation, and broad managed-service tours belong to other courses. SGLang and NVIDIA Triton Inference Server have a compact sourced comparison. Colab, Vast.ai, Runpod, AWS, Azure, and Google Cloud are comparison-only. No universal provider ranking, speedup, cross-provider equivalence, production SLA, or multi-GPU capacity claim is made.
When another approach is better
- Use a ready inference API when operating a serving engine adds more burden than useful control.
- Do not choose self-managed inference when measured capacity, task quality, data restrictions, or full operating cost miss the workload requirements.
- A bounded single-host experiment cannot establish multi-GPU capacity, production reliability, or equivalent performance across providers.
How we train
Contact us for full course details, including duration, delivery options and lab requirements.
Ask about exercises, instructor feedback and the prior knowledge you need. Tool-specific courses marked provisional may become modules in a broader course.
Discuss your learning goals