Why Reinforcement Learning Is Both Brilliant and Brutal
In this article, we explore Q-Learning Unity — Reinforcement learning is the closest thing we have to how biological intelligence actually develops — through trial, error, and feedback from the environment. It's also, as Andrej Karpathy famously put it, terrible — just less terrible than everything that came before it. After building production ML systems for years, I'd add my own corollary: RL is uniquely humbling because bugs don't crash your program, they just silently produce a dumb agent.
In this tutorial, I'm going to walk you through Q-Learning from mathematical first principles all the way to a working Unity implementation. We'll build a navigating robot that learns to reach a goal while avoiding hazards — no ML library magic, just raw Bellman equations and C# code. If you've ever wondered what's actually happening inside reinforcement learning algorithms, this is the article for you.
Full source code is available on GitHub. Basic C# knowledge is assumed. For Unity newcomers: every game object runs Start() once on initialization and Update() every frame — that's all the Unity context you need.
For a production-ready setup, check out VPS Server's cloud hosting plans.
The Reinforcement Learning Loop: States, Actions, and Rewards
Reinforcement learning operates on a deceptively simple feedback loop. An agent observes its current state, selects an action, receives a reward signal, transitions to a new state, and repeats. The goal is to learn a policy — a mapping from states to actions — that maximizes cumulative reward over time.
Formally, a deterministic policy π maps states to actions:
π: S → A
If you want probabilistic behavior (useful for exploration), you use a stochastic policy that outputs action probabilities:
π: S × A → [0, 1]
The critical insight that separates RL from supervised learning is that there's no labeled training data. The agent must discover what's good through interaction. This makes RL technology guides incredibly powerful for sequential decision problems — and incredibly prone to instability, sparse reward pathologies, and sample inefficiency.
Defining the Environment: A Grid World
Our testbed is a classic grid world: an 8×5 tile map where a robot must navigate from any grass tile to a trophy, without stepping into water. Actions are cardinal directions: A = {Left, Right, Up, Down}.
Tile types double as reward signals, encoded as an enum:
public enum TileEnum { Water = -1, Grass = 0, Award = 1 }
The map itself is a 2D array where -1 is water, 0 is grass, and 1 is the award:
_map = {
{ -1, -1, -1, -1, -1, -1, -1, -1 },
{ -1, 0, 0, 0, -1, 0, 1, -1 }, // 1 = trophy
{ -1, 0, 0, 0, -1, 0, 0, -1 },
{ -1, 0, 0, 0, 0, 0, 0, -1 },
{ -1, -1, -1, -1, -1, -1, -1, -1 },
};
Notice the water column at index 4 — this creates a chokepoint that forces the agent to reason about longer paths rather than just moving greedily toward the goal. It's a simple but effective way to validate that your agent is actually planning, not just reacting.
The TileGrid utility class handles coordinate lookups and action-based transitions:
public T GetTileByCoords<T>(int x, int y);
public T GetTargetTile<T>(T source, ActionEnum action);
public void GenerateTiles();
The generic T here is intentional — we'll reuse this grid structure for both value iteration and Q-learning with different tile types.
The Bellman Equation: The Mathematical Heart of RL
Before we can train an agent, we need a principled way to assign value to states. This is where the Bellman Equation comes in — arguably the most important equation in all of reinforcement learning.
The core insight: the value of a state equals the immediate reward you get from the best action plus the discounted value of wherever that action takes you. Mathematically:
V(s) = max over all actions a of: R(s, a) + γ · V(s')
Where:
s= current states'= next state after taking actionaR(s, a)= immediate rewardγ= discount factor (how much we care about future rewards, typically 0.9–0.99)
The discount factor γ is crucial. Set it too low and the agent becomes myopic, ignoring long-term consequences. Set it too high and learning becomes unstable. For our grid world, γ = 0.9 works well.
Iterative Value Computation in C#
We solve the Bellman equation through value iteration — repeatedly updating state values until they converge. Terminal states (water and trophy) keep their fixed reward values. Grass tiles get updated each iteration:
private float gamma = 0.9f;
private double GetNewValue(VTile tile)
{
return Agent.Actions
.Select(a => tileGrid.GetTargetTile(tile, a))
.Select(t => t.Reward + gamma * t.Value)
.Max();
}
private void CalculateValues()
{
for (var y = 0; y < TileGrid.BOARD_HEIGHT; y++)
{
for (var x = 0; x < TileGrid.BOARD_WIDTH; x++)
{
var tile = tileGrid.GetTileByCoords<VTile>(x, y);
if (tile.TileType == TileEnum.Grass)
tile.NextValue = GetNewValue(tile);
}
}
}
private void Step()
{
for (var y = 0; y < TileGrid.BOARD_HEIGHT; y++)
for (var x = 0; x < TileGrid.BOARD_WIDTH; x++)
tileGrid.GetTileByCoords<VTile>(x, y).Step();
}
The double-buffering pattern here (writing to NextValue, then committing with Step()) is important — it prevents values from propagating more than one step per iteration, which would corrupt the computation.
In practice, this converges in about 10 iterations. The value gradient radiates outward from the trophy tile, decaying by γ with each hop. Tiles adjacent to water get pulled down by the -1 penalty. The chokepoint at column 4 creates an interesting valley in the value landscape that the agent must navigate around.
From State Values to Q-Learning: Teaching the Agent to Act
Value iteration tells us how good each state is, but it doesn't directly tell the agent what to do — especially when the agent can't see the full environment or when the environment changes dynamically. This is where Q-Learning shines.
Instead of assigning values to states, Q-Learning assigns values to (state, action) pairs — called Q-values or action quality values. The Q-value Q(s, a) represents the expected cumulative reward of taking action a in state s and then following the optimal policy thereafter.
The Q-Learning Update Rule
The update rule derives directly from the Bellman equation:
Q(s, a) ← Q(s, a) + α · [R(s,a) + γ · max_a'(Q(s', a')) - Q(s, a)]
The term in brackets is the temporal difference (TD) error — the gap between what we predicted and what we actually got. The learning rate α controls how aggressively we update (typically 0.001–0.01; we use 0.005 here).
Unity Implementation
Each QTile stores a Q-value per action. On every step, we update Q-values for all actions from the current state, then move the agent:
private void Step()
{
if (_agent.State.TileType != TileEnum.Grass)
{
ResetAgentPos();
return;
}
QTile s = _agent.State;
// Update Q-values for ALL actions from current state
foreach (var a in Agent.Actions)
{
double q = s.GetQValue(a);
QTile sPrime = tileGrid.GetTargetTile(s, a);
double r = sPrime.Reward;
double qMax = Agent.Actions.Select(sPrime.GetQValue).Max();
double td = r + gamma * qMax - q;
s.SetQValue(a, q + alpha * td);
}
// Move the agent using the current best action
ActionEnum chosen = PickAction(s);
_agent.State = tileGrid.GetTargetTile(s, chosen);
}
A key advantage over pure value iteration: Q-Learning works even when the agent has partial observability (limited field of vision) or when objects in the environment move. The agent learns from its own trajectory through the world, not a global computation over all states.
With a fixed greedy policy, training typically stabilizes between 500–1000 steps on this grid. When multiple actions tie for best Q-value, we randomly break ties (achieved by shuffling the action list) — this prevents the agent from always defaulting to the same arbitrary action.
Exploration vs. Exploitation: The ε-Greedy Strategy
Here's the dirty secret of naive Q-Learning: if your agent always picks the highest Q-value action, it will often get stuck in a local optimum. It finds a path to the goal and stops exploring, never discovering potentially better routes.
This is the exploration-exploitation dilemma, and it's one of the fundamental unsolved problems in RL research. For tabular Q-Learning, the standard practical solution is ε-greedy:
if random() > epsilon:
take the greedy (best Q-value) action // exploit
else:
take a random action // explore
Decaying Epsilon Schedule
Early in training, we want lots of exploration. As the agent learns, we shift toward exploitation. A simple linear decay handles this:
epsilon = Math.Max(epsilonMin, epsilon - epsilonDecay);
Typical hyperparameter values for this grid world:
epsilon_start = 1.0(fully random at the start)epsilon_min = 0.05(always keep 5% exploration)epsilon_decay = 0.001per step
This gives you roughly 950 steps of meaningful exploration before settling into near-greedy behavior. More sophisticated schedules (exponential decay, cosine annealing) exist, but linear decay is surprisingly effective for tabular problems.
Why This Matters Beyond Grid Worlds
The exploration-exploitation tradeoff doesn't go away with deep RL — it gets harder. Techniques like noisy networks, intrinsic curiosity modules, and entropy regularization (used in algorithms like SAC and PPO) are all sophisticated answers to the same fundamental question: how do you ensure your agent explores enough without wasting compute on random wandering?
If you're planning to scale this up to neural network-based agents, check out resources on Read more about this topic and consider frameworks like Stable-Baselines3 for Python-based environments.
Taking It Further: From Tabular Q-Learning to Production RL
Tabular Q-Learning works beautifully for small, discrete state spaces like our 8×5 grid. But real-world problems have state spaces that are astronomically large or continuous — you can't maintain a table with one entry per (state, action) pair when states are images or sensor readings.
The natural extension is Deep Q-Networks (DQN), where a neural network approximates the Q-function. This is what DeepMind used to achieve superhuman Atari performance. From there, the field branches into actor-critic methods (A3C, PPO, SAC), multi-agent RL, and model-based RL.
For self-hosting RL experiments, you'll want serious compute. I run my RL training jobs on a VPS cluster from vps-server.host with GPU instances — the ability to run parallel environment rollouts dramatically cuts training time compared to a single machine. For more ML engineering patterns and benchmarks, I regularly publish on Data Mammoth.
For more on building AI agents with memory and planning capabilities, see our guide on Read more about this topic.
Conclusion: Reinforcement Learning Agents Start Here
We've gone from the mathematical foundations of reinforcement learning through the Bellman equation, implemented iterative value computation, built a full Q-Learning agent in Unity, and tackled the exploration-exploitation tradeoff with ε-greedy strategies. This is the complete picture of tabular Q-Learning — not a hand-wavy overview, but actual working code grounded in the underlying math.
The reinforcement learning agent we built here is humble — a robot on a grid — but the patterns are universal. The Bellman equation, temporal difference learning, and exploration-exploitation management appear in every serious RL system, from game-playing AIs to robotics controllers to recommendation systems.
Start with this implementation, get it running in Unity, then break it deliberately: remove the epsilon decay and watch it overfit to one path. Crank gamma to 1.0 and observe the instability. Understanding why things fail is how you build intuition for the harder problems ahead. The full code is on GitHub — clone it, modify it, and start experimenting.