AI curriculum

Course outline for Vector and AI Data Infrastructure

Build a searchable document collection with actual embeddings and PostgreSQL vector queries. Choose relational queries, full-text search, or vectors for the same questions. Combine eligible candidates and run actual reranking when useful. Use independently judged questions to measure relevance rather than treating a short distance as a correct answer. Maintain compatible embedding generations through correction, deletion, and cutover.

Vector and AI Data Infrastructure Course Objectives

  • Generate document and query embeddings with an explicit compatibility contract.
  • Create pgvector storage and execute checked similarity queries.
  • Compare exact and approximate retrieval using measured work and recall.
  • Combine metadata restrictions and lexical evidence without leaking excluded documents.
  • Benchmark relevance against independent judgments rather than the retriever's own scores.
  • Test incompatible queries, correction, deletion, rebuild, activation, and rollback.
  • Explain equivalent Weaviate operations without claiming hands-on Weaviate deployment.
  • Choose exact relational answers, full-text matching, or semantic candidates for the same corpus.
  • Run actual cross-encoder reranking after restriction-aware candidate fusion.
  • Defend a retrieval route with held-out quality, latency, and cost ablations.

Pre-requisites

  • Basic SQL queries, tables, keys, and filtering, from PostgreSQL Development or equivalent practice.
  • Basic Python, files, dependencies, and running a local test.
  • Distinguish a document, a search query, and a retrieved passage.
  • No vector mathematics, embedding runtime, or Weaviate experience is required.

Before this course

Complete the required courses or bring equivalent experience. Recommended courses add useful depth; they are not enrolment gates.

Required preparation

Recommended preparation

Readiness checks

  • Basic SQL queries, tables, keys, and filtering, from PostgreSQL Development or equivalent practice.
  • Basic Python, files, dependencies, and running a local test.
  • Distinguish a document, a search query, and a retrieved passage.
  • No vector mathematics, embedding runtime, or Weaviate experience is required.

Basic SQL queries, tables, keys, and filtering, from PostgreSQL Development or equivalent practice.

Find your place in the learning map

Lab Setup

Delivery requirements: the environment, access, and exercise materials must be prepared and verified before delivery. This outline does not certify that a lab package is ready.

Disposable compatible PostgreSQL/pgvector, Python, supported Sentence Transformers, licensed small embedding and cross-encoder model revisions, synthetic documents and linked product records, independent development and sealed test judgments, SQL files, lifecycle adapter, benchmark starter, and cleanup inventory. Actual CPU operation, input truncation, memory/disk limits, compatible releases, restriction propagation, independent checks, safe activation/rollback, and timing require verification. No GPU, hosted model account, or second database backend is required.

Detailed Course Outline

Module 1: Choose a retrieval route for the question (2 hours)

  • Shared retrieval workload: One support corpus, linked product records, and shared question families.
  • Relational query fit: Relational queries for exact IDs, joins, filters, and computed answers.
  • Full-text query fit: Full-text words, phrases, language configuration, and identifier tokenization limits.
  • Vector query fit: Vector paraphrase matching versus exact facts and relational constraints.
  • Restriction pushdown: Mandatory restriction pushdown before candidate text leaves each retrieval branch.
  • Independent judgments and simpler routes: Independent judgments, no-match cases, and the simplest sufficient route.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Execute relational and full-text baselines, then justify which questions need semantic candidates rather than exact answers.

Module 2: Turn documents into compatible vectors (2 hours)

  • Vector foundations: Vectors, dimensions, distance measures, and a small numerical example.
  • Embedding limits: Embeddings as learned representations rather than facts or permissions.
  • Actual local encoding: Sentence Transformers basics, model loading, and actual local encoding.
  • Chunk and query compatibility: Document chunks, query encoding, normalization, and preprocessing identity.
  • Generation manifests: Corpus identifiers, metadata, model revision, and generation manifests.
  • Judgments and excluded documents: Independent relevance judgments and excluded-document test cases.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Generate real document/query vectors and record the model, dimensions, preprocessing, and expected matches.

Module 3: Retrieve candidates through actual SQL (2 hours)

  • pgvector integration: Existing PostgreSQL connection, pgvector extension delta, and vector columns.
  • Parameterized vector storage: Insertion, stable document keys, metadata, and parameterized queries.
  • Exact similarity queries: Distance operators, ordering, limits, and exact nearest neighbors.
  • Compatible query encoding: Query-vector generation through the same declared encoding contract.
  • Candidates versus answers: Nearest candidates versus relevant answers and exact identifier lookup.
  • Independent result checks: Result identity checks, missing matches, and invalid vector rejection.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Run the complete text-to-vector-to-SQL path and inspect actual document results.

Module 4: Measure the price of approximation (2 hours)

  • Exact-neighbor reference: Exact nearest neighbors as the index-recall reference, not relevance truth.
  • Actual HNSW operation: HNSW basics, distance operator classes, and index creation.
  • Approximation measurements: Search effort, candidate recall, latency, build time, and index size.
  • Execution-plan evidence: Execution-plan checks that establish which retrieval path actually ran.
  • Repeatable comparisons: Repeated bounded measurements with fixed queries and corpus versions.
  • Small-corpus limits: Small-data limits and conditions favoring exact retrieval.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Compare actual exact/HNSW runs and explain quality, latency, and storage trade-offs without promising speedup.

Module 5: Respect restrictions and combine search evidence (2 hours)

  • Joined eligibility restrictions: Restriction pushdown for collection, visibility, language, freshness, and joined eligibility.
  • Filtered candidate shortfall: Filtered approximate searches, insufficient candidates, and explicit fallback.
  • Lexical and identifier baselines: Reuse full-text baseline, word/phrase behavior, and exact identifier fallback.
  • Deduplicated rank fusion: Deduplicated rank fusion over separately inspected eligible lexical/vector candidates.
  • Judged retrieval measures: Fixed relevance judgments, top-k precision, recall, and excluded hits.
  • Weaviate comparison: Sourced Weaviate collection, vector, filter, and hybrid-query comparison.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Compare filtered vector, keyword, and hybrid result sets against the same independently judged questions.

Module 6: Rerank candidates and test whether it helps (2 hours)

  • Cross-encoder and fusion roles: Cross-encoder query/passage scoring versus embedding retrieval and rank fusion.
  • Actual CPU pair scoring: Sentence Transformers CrossEncoder first load and actual CPU pair scoring.
  • Eligible candidate union: Bounded eligible candidate union, deduplication, truncation, and stable identity.
  • Actual reranking: Actual fused-candidate reranking with retained scores and changed ordering.
  • Baseline ablations: Fixed baseline ablations with judged quality, latency, and compute cost.
  • Missing candidates and held-out checks: Missing-candidate limits, regressions, no-match decisions, and held-out evaluation.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Run a real small cross-encoder on fused candidates and justify retaining or rejecting it with measured evidence.

Module 7: Change an index without mixing its meaning (2 hours)

  • Encoding compatibility: Model revision, dimensions, normalization, and preprocessing compatibility checks.
  • Same-dimension mismatch: Same-dimension incompatible models and explicit manifest rejection.
  • Correction and deletion: Source corrections, re-embedding, deletion, and retained-copy inventories.
  • Separate generation rebuild: Building a separate generation and validating before activation.
  • Activation and rollback: Atomic active-generation selection, failed activation, and rollback.
  • Retained-generation cleanup: Deleted-document exclusion after rebuild and retired-generation cleanup.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Reject an incompatible query and verify correction, deletion, safe generation cutover, and rollback.

Practical exercise

  • Build a small support-document search service. Planned supplied materials are synthetic documents, visibility labels, manually judged development questions, held-out questions, database startup, and benchmark/report scaffolding. Learners extend encoding/loading code, author relational joins, SQL restrictions, rank fusion, and actual cross-encoder invocation with ID mapping, add compatibility checks, and implement the active-generation change. A complete solved retriever or supplied list of vectors cannot substitute for these changes.
  • Generate actual embeddings locally with Sentence Transformers and one selected, licensed small model. Persist them in actual pgvector. Generate new query vectors at runtime and retrieve by SQL. Compare exact search and HNSW using execution plans, candidate recall against exact neighbors, repeated latency observations, build duration, and size. Score semantic relevance separately against independent human judgments. Do not tune on held-out questions or use distance as its own key.
  • Include a successful semantic query, a keyword-sensitive failure, a similar but forbidden document, and a no-match question. Check every returned identifier and visibility field. Combine SQL restrictions with lexical/vector rank fusion and report regressions as well as improvements. Apply the same mandatory restrictions to every branch before fusion or model scoring. Execute Module 6's actual CPU reranking and baseline ablations with fixed development and held-out partitions. An exact-ID or join answer must remain available when semantic ranking is needless. Evaluate an unseen phrase/identifier/paraphrase question and a missing-candidate case before explanatory feedback. These are application retrieval checks, not proof of complete authorization or resistance to every attack.
  • Test wrong dimensions and a same-dimension wrong model or preprocessing manifest. Correct a source, re-embed it, delete another source, and rebuild separately. Prevent an incomplete generation from activation. Activate a checked generation, roll back a deliberately bad activation, and verify that deletion is not undone by rollback. Apply deletion to every queryable retained generation or make it unavailable. Check cached results and document references within the lab scope. State backup/log retention separately rather than claiming complete erasure.
  • Reserve an unseen query and lifecycle fault for independent prediction and repair before explanatory feedback. Recommend exact SQL or keyword search when it meets the workload without embedding/index complexity. Weaviate receives sourced equivalent-operation examples only, not a second running backend.

Delivery and timing

  • Guided hours include teaching, demonstrations, learner practice, feedback, questions, and contingency as allocated above. Breaks, bulk downloads, account approval, and prerequisite setup are additional. Allocations need a timed learner pilot before delivery; they are not measured completion times.

What you will produce

Choose, implement, and evaluate relational, lexical, vector, hybrid, and reranked retrieval while maintaining compatible data generations.

How the work is checked

  • Actual generated vectors, stored records, queries, and results share an explicit contract.
  • Index recall and judged relevance use different independent references.
  • Actual plans, timings, build observations, and sizes support the comparison.
  • Filter tests exclude disallowed identifiers in every required retrieval path.
  • Hybrid results are checked against the fixed judgment set, including regressions.
  • Relational joins, exact IDs, full-text words/phrases, and paraphrases receive justified route choices.
  • Restricted candidates never reach fusion or reranker text input, with final eligibility checks.
  • Actual cross-encoder pair scores reorder only the supplied eligible candidates.
  • Missing-candidate tests distinguish retrieval loss from reranking mistakes.
  • Frozen baseline ablations report held-out quality, latency, and cost without tuning.
  • Incompatible generations fail before retrieval, including same-dimension mismatches.
  • Correction, deletion, failed cutover, activation, and rollback produce checked results.
  • The independent task remains unanswered until the learner submits evidence.

14 guided hours. General SQL and indexing are preparation, not a repeated database course. Actual local embeddings, pgvector, HNSW, lexical queries, rank fusion, and CPU cross-encoder reranking are required. Relational or keyword answers can be preferable to vectors. Index recall differs from independently judged relevance. Fusion and reranking are different steps; reranking cannot restore candidates retrieval missed. IVFFlat and Weaviate are sourced comparisons. No embedding/reranker training, production administration, guaranteed relevance, or universal deletion certification is included.

When another approach is better

  • Use exact SQL or keyword retrieval when identifiers and structured filters answer the questions reliably.
  • Avoid unmeasured embedding projection or backend migration when relevance and compatibility tests are unavailable.

How we train

Contact us for full course details, including duration, delivery options and lab requirements.

Ask about exercises, instructor feedback and the prior knowledge you need. Tool-specific courses marked provisional may become modules in a broader course.

Discuss your learning goals