Reinforcement Learning

Learning outcomes

After this lesson, you will be able to:

• Explain how RL differs from supervised and unsupervised learning.
• Distinguish an agent, environment, state, action, reward, and episode.
• Explain policies, value functions, and discounted return.
• Distinguish exploration from exploitation.
• Describe Q-learning, policy-based, and actor–critic methods conceptually.
• Formulate a teaching RL task and identify reward-design and unsafe-exploration risks.

1. How RL differs from other learning settings

In supervised learning, each training example has a known target. In unsupervised learning, predefined labels are absent and an algorithm searches for structure. In reinforcement learning (RL), an agent interacts with an environment: it selects an action and receives a new state and reward.

The key feature is a sequence of decisions. An action affects not only the immediate reward but also future states and available choices. Long-term outcomes therefore matter.

Examples include a robot learning to move, a game agent choosing moves, or a controller learning in simulation to maintain comfortable temperature while limiting energy use.

2. Elements of an RL task

A common formalism is a Markov Decision Process (MDP): states S, actions A, reward function R, transition probabilities P, and an initial-state distribution. An episode is a bounded sequence of interactions ending at a terminal condition.

3. Reward, return, and discounting

A reward is a numeric signal returned after an action or transition. It can be positive, zero, or negative. The sum of rewards over a sequence is called the return.

The discount factor γ between 0 and 1 sets the relative weight of future rewards. A smaller γ emphasises near-term outcomes; a value closer to 1 gives more weight to distant results.

A reward is not the same as a human goal. If a system is rewarded only for saving energy, it may do so by reducing comfort unacceptably. Safety constraints, permitted ranges, and emergency modes must be defined separately.

4. Policy, value, exploration, and exploitation

• Policy (π) is a rule for selecting an action from a state or observation.
• A value function estimates expected future return from a state.
• Q(s, a) estimates expected return when action a is taken in state s and the policy is followed afterwards.

Exploration tests actions whose outcomes are not yet well known. Exploitation chooses the action currently estimated to be best. Only exploring wastes experience; exploiting too early can prevent discovery of a better strategy.

5. Main algorithm families

• Value-based: estimate states or state–action pairs. Q-learning is an example.
• Policy-based: directly optimise a policy, or a probability distribution over actions. Policy gradient is an example.
• Actor–critic: combine an actor that selects actions with a critic that evaluates states or actions.
• Model-based RL: use a model of environment dynamics for planning.
• Deep RL: use neural networks to represent policies or value functions for complex state spaces.

These categories are not mutually exclusive: deep RL can be value-based, policy-based, or actor–critic. Selection depends on action type, interaction cost, model availability, and stability requirements.

6. Task formulation: a building in simulation

Imagine an agent controlling heating only in a safe simulation. Its state includes indoor and outdoor temperature, time of day, and weather forecast. Actions slightly increase power, decrease it, or hold it. A reward can account for comfort deviation and energy use.

7. Practice: minimal Q-learning in Python

This one-dimensional toy environment has five states. The agent starts at position 0 and must reach position 4. Actions move left or right. Reaching the goal gives +1; each intermediate step costs −0.02. No external libraries are needed; Python 3.10+ is required.

import random

random.seed(7)
N_STATES = 5
ACTIONS = (-1, 1)  # left, right
Q = [[0.0, 0.0] for _ in range(N_STATES)]
alpha, gamma, epsilon = 0.2, 0.95, 0.2

def step(state, action):
    next_state = max(0, min(N_STATES - 1, state + action))
    done = next_state == N_STATES - 1
    reward = 1.0 if done else -0.02
    return next_state, reward, done

for episode in range(400):
    state = 0
    for _ in range(10):
        if random.random() < epsilon:
            action_index = random.randrange(len(ACTIONS))
        else:
            action_index = max(range(2), key=lambda i: Q[state][i])
        action = ACTIONS[action_index]
        next_state, reward, done = step(state, action)
        best_next = 0.0 if done else max(Q[next_state])
        target = reward + gamma * best_next
        Q[state][action_index] += alpha * (target - Q[state][action_index])
        state = next_state
        if done:
            break

state, path = 0, [0]
for _ in range(10):
    action_index = max(range(2), key=lambda i: Q[state][i])
    state, _, done = step(state, ACTIONS[action_index])
    path.append(state)
    if done:
        break
print(" -> ".join(map(str, path)))

How the code works

• Q[state][action_index] stores the action estimate for a state.
• epsilon sets the probability of random exploration.
• target combines immediate reward with the best next-action estimate.
• alpha controls the update step and gamma the weight of future reward.

Expected output: 0 -> 1 -> 2 -> 3 -> 4. This illustrates Q-learning mechanics, not suitability for real equipment.

8. Limitations and common mistakes

• Incorrect reward: the agent optimises a formal signal that does not match the real objective.
• Unsafe exploration: random actions may be unacceptable in a real environment.
• High experience cost: training may require many interactions or a simulator.
• Simulation-to-reality gap: behaviour learned in simulation may not transfer.
• Partial observability and instability: important factors may be hidden, and results depend on seeds and settings.

RL does not guarantee safe or optimal control. Testing, independent constraints, monitoring, and the ability to stop or override actions are required.

9. Key takeaways

• RL trains an agent through environment interaction and reward.
• Actions have sequential consequences, so long-term return matters.
• A policy selects actions; value functions estimate outcomes.
• Exploration and exploitation must be balanced.
• Quality depends on task formulation, reward, constraints, and environment.