Learning outcomes
After this lesson, you will be able to:
- Distinguish generative and discriminative task formulations.
- Explain autoregression, VAEs, GANs, and diffusion models.
- Understand self-supervised learning and foundation models.
- Distinguish a generative model from RAG and tool-using systems.
- Evaluate output quality, diversity, factuality, and risks.
- Run an experiment with a simple autoregressive model.
Place in the curriculum. Lessons 1.3.1–1.3.5 introduced learning paradigms, prediction tasks, and structure discovery. This lesson turns to generation; model-training details remain for 1.4.1–1.4.2.
1. What does it mean to generate?
Previous lessons covered models that classify objects, predict values, find groups, or select actions. A generative approach asks a different question: what regularities describe data, and how can we create a new example resembling the training data? A classifier may identify a technical document type; a generative model may draft a description. It does not know a specific object — it models data regularities.Generative modela model that represents a data distribution or generation process and can create new examples.
Discriminative modela model that estimates a class or relationship between input and target, such as a class probability from features.
Resemblance to training data does not imply truth, usefulness, or safety.2. How a model builds a sequence
Text is usually split into tokens — words, subwords, characters, or other units. An autoregressive model estimates the next token given the current context, appends the selected token, and repeats.Tokena unit of text after tokenization: a word, subword, character, or another fragment.
Autoregressive modela model that predicts the next element sequentially from the context already formed.
Autoregressive factorizationP(x₁,…,xₜ) = ∏ₜ P(xₜ | x₁,…,xₜ₋₁)
The probability of a sequence is built from conditional probabilities of each next token. The model does not retrieve a complete answer from a library. A likely token is not necessarily true, and an early error changes later context.3. Four generative model families
Autoregressive models
They continue a sequence one element at a time. They suit text and other sequences, but generation is sequential and early errors can accumulate.Variational autoencoders (VAEs)
Latent spacea compact internal representation of important data characteristics.
A VAE encodes an example into a probabilistic latent representation and decodes it. Sampling a new point can create a new example. It is useful for representations and variations, though outputs can be smoothed.Generative adversarial networks (GANs)
GANa system with a generator that creates samples and a discriminator that distinguishes real from generated samples; they train adversarially.
The generator tries to fool the discriminator. GANs can create sharp images, but training may be unstable and mode collapse can reduce diversity.Diffusion models
Diffusion modela model trained to reverse gradual noising and create a sample through successive denoising steps.
Training adds noise to data and teaches a model to estimate it. Generation moves from noise toward an image or another object. Quality can be high, but computation and multiple steps are often required.4. Foundation models and adaptation
Foundation modela large model pretrained on broad data and suitable for adaptation to many downstream tasks.
Self-supervised learningtraining in which the signal is derived from the data itself, such as predicting a hidden part or next element, without manual labels for every example.
After pretraining, a model can be used through prompting (instructions and context), fine-tuning (further parameter training), or parameter-efficient tuning (updating a small set of parameters or adapters).Multimodal modela model that processes or connects multiple data types, such as text, images, audio, or video.
Multimodality can connect a question to an image or speech to text, but does not guarantee equal accuracy across input types.5. RAG and tools: the model is not the whole system
RAG (Retrieval-Augmented Generation)an architecture in which relevant external documents are retrieved and added to context before generation.
A modern AI service has layers: the model; context (instructions, history, retrieved documents, images); and orchestration (retrieval, tools, validation, and presentation). RAG is not a generative-model family; it supplies external information. Retrieval may miss a document, and a model may misinterpret it. An API call does not mean the model itself performed an action: software must validate arguments, permissions, and results.6. Python practice: a character bigram model
Build a small model that counts which characters follow one another in sample phrases and generates new sequences. It is not a neural network or an LLM; it demonstrates conditional distributions.Environment: Python 3.10+, standard library only, no network access.
from collections import defaultdict, Counter
import random
corpus = [
"water is heated",
"water circulates",
"a pump creates flow",
"a sensor measures temperature",
"the system maintains a mode",
]
next_chars = defaultdict(Counter)
for text in corpus:
padded = "^" + text + "$"
for current, following in zip(padded, padded[1:]):
next_chars[current][following] += 1
def generate(max_length=80):
current, output = "^", []
for _ in range(max_length):
choices = next_chars.get(current)
if not choices:
break
chars, weights = zip(*choices.items())
current = random.choices(chars, weights=weights, k=1)[0]
if current == "$":
break
output.append(current)
return "".join(output)
random.seed(7)
for _ in range(5):
print(generate())7. Practical assignment: choose an architecture
An engineering organization needs an AI service for four tasks: (1) classify requests; (2) draft technical descriptions from documents; (3) generate synthetic defect images; (4) answer questions using equipment diagrams and procedures. For each task, prepare a decision card: is generation needed; which approach fits (discriminative model, autoregression, diffusion, multimodality, RAG, or a combination); what data and rights are needed; two quality criteria; how to detect an error; and where a human is required. Deliver a four-row table with rationale. Choose the simplest sufficient approach, not the most complex one.Independent assignment
Submit a four-row decision table with rationale and limitations.
8. Quality, limitations, and risks
Evaluate task fit, factuality, diversity, consistency, controllability, cost, and latency. One good sample is not enough.Hallucinationa plausible-sounding but unsupported or incorrect claim produced by a generative system.
Risks include incomplete or biased training data, outdated facts, prompt sensitivity, leakage of confidential fragments, and intellectual-property issues. Synthetic data are not automatically anonymous. In engineering, generated text does not replace a normative document, measurement, or expert review. Check primary sources, dates, versions, units, and applicability. Important use cases need test sets, human evaluation, logging, data protection, monitoring, and safe escalation.9. Summary and boundaries of later lessons
Autoregression continues sequences; VAEs use probabilistic latent representations; GANs train generator and discriminator adversarially; diffusion models restore data from noise. Foundation models are reused across tasks, and multimodal models connect input types. RAG and tools can strengthen a system but are not synonyms for a generative model. Lessons 1.4.1–1.4.2 will examine models, parameters, training, and loss in depth; Theme 2 supplies mathematical foundations. Here the goal is to understand model families and their boundaries, not derive transformer architecture or tune an LLM.Sources and further reading
- Goodfellow et al. (2014), Generative Adversarial Nets
- Kingma & Welling (2013), Auto-Encoding Variational Bayes
- Ho et al. (2020), Denoising Diffusion Probabilistic Models
- Vaswani et al. (2017), Attention Is All You Need
- Brown et al. (2020), Language Models are Few-Shot Learners
- Bommasani et al. (2021), On the Opportunities and Risks of Foundation Models
- Lewis et al. (2020), Retrieval-Augmented Generation
Lesson glossary
Represents a data distribution and creates new examples.
Estimates a class or input-target relationship.
A unit of text after tokenization.
Step-by-step prediction of the next element from context.
A compact internal representation of data.
A probabilistic autoencoder with a latent representation.
An adversarial generator-discriminator system.
Generation by successive denoising.
A broadly pretrained model for many tasks.
Adding retrieved documents to context before generation.
Self-check
Question 1
What defines a generative model?
Question 2
How does autoregressive generation work?
Question 3
What is the VAE idea?
Question 4
What distinguishes a GAN?
Question 5
How does diffusion generation begin?
Question 6
What is a foundation model?
Question 7
What does RAG do?
Question 8
Which checks help evaluate a generative system? Select all.