Learning from Examples

The timing for learning these techniques has never been better. AI systems are being deployed in increasingly high-stakes, dynamic environments where traditional approaches fall short. Autonomous vehicles need to navigate unpredictable roads. Medical AI systems must make recommendations under genuine uncertainty. Recommendation engines serve billions of users with wildly different preferences. In each of these domains, imitation learning and Thompson sampling offer concrete advantages over older methods.

Consider the challenge of training a robotic arm to perform a delicate assembly task. Writing a reward function that captures every nuance of what "good" performance looks like is extraordinarily difficult. You might reward the robot for completing the task, but how do you encode the smoothness of motion, the avoidance of unnecessary force, or the subtle adjustments needed when parts do not align perfectly? Imitation learning sidesteps this problem entirely. You demonstrate the task yourself, and the system learns from your behavior. Research from groups at Carnegie Mellon and Berkeley has shown that even a small number of expert demonstrations can dramatically accelerate learning in robotic manipulation tasks.^[1]

Thompson sampling has had a similarly outsized impact in industry. When Google and Microsoft began applying it to online advertising and content recommendation in the early 2010s, they found it consistently outperformed simpler approaches like epsilon-greedy exploration. A well-known study from Microsoft Research demonstrated that Thompson sampling achieved lower regret (a measure of suboptimal decisions) than competing bandit algorithms across a wide range of real-world scenarios.^[2] The practical gains were not marginal. They were substantial enough to drive widespread adoption across the tech industry.

Beyond individual applications, these techniques matter because they represent a broader shift in how the field thinks about learning. The old paradigm assumed you could specify what you wanted precisely, either through labeled data or through a reward function. The new paradigm acknowledges that this is often impossible or impractical, and builds methods that work around that limitation. Imitation learning says: show me what you want. Thompson sampling says: let me figure out what works by exploring intelligently. Both are responses to the fundamental messiness of real-world AI deployment.

The research communities around both techniques are also exceptionally active right now. Imitation learning has seen major advances with algorithms like DAgger (Dataset Aggregation), GAIL (Generative Adversarial Imitation Learning), and various inverse reinforcement learning frameworks. Thompson sampling has been extended to deep learning contexts, contextual bandits, and full reinforcement learning settings. Understanding the foundations covered in this book will give you the context to follow and contribute to this ongoing research.

Finally, these techniques are increasingly accessible. Open-source libraries, pre-trained models, and cloud computing platforms have lowered the barrier to experimentation significantly. You do not need a research lab or a supercomputer to start applying these methods. What you do need is a solid conceptual understanding of how they work, why they work, and where they are likely to succeed or struggle. That is exactly what this book provides.

A concrete example of impact: In 2016, OpenAI demonstrated that imitation learning combined with reinforcement learning could teach a simulated robot hand to perform dexterous manipulation tasks far more efficiently than reinforcement learning alone. The key insight was that even imperfect demonstrations provided a useful starting point that dramatically reduced the exploration burden on the RL algorithm. This hybrid approach has since become a standard technique in robotics research.

This book is structured to take you from foundational concepts to practical implementation in a logical, progressive sequence. Each chapter builds on the ones before it, so reading in order is recommended, especially if you are newer to either of these topics. That said, if you already have a strong background in one area, the chapters are written to be reasonably self-contained, and you can navigate to the sections most relevant to your needs.

The first four chapters cover imitation learning in depth. Chapter 1 sets the stage by situating imitation learning within the broader landscape of machine learning paradigms. You will understand what makes it distinct from supervised and reinforcement learning, and why those distinctions matter in practice. Chapter 2 dives into behavioral cloning, the simplest and most widely used form of imitation learning. You will see how it works, how to implement it, and what its limitations are.

Chapter 3 tackles one of the most important and often underappreciated challenges in imitation learning: the distribution shift problem. This is the phenomenon where a model trained on expert demonstrations starts making small errors, and those errors compound over time until the system ends up in situations it has never seen before. Understanding this problem is essential for building imitation learning systems that actually work in deployment. Chapter 4 then introduces inverse reinforcement learning, a more sophisticated approach that tries to infer the underlying reward function from expert behavior rather than simply copying actions.

Chapters 5 and 6 shift focus to Thompson sampling. Chapter 5 introduces the exploration-exploitation dilemma and explains the Bayesian foundations of Thompson sampling from the ground up. Chapter 6 moves into practical territory, covering posterior distributions, common algorithms, and how to build Thompson sampling systems for real applications. These two chapters are designed to give you both the theory and the hands-on knowledge you need.

Chapter 7 is where the two main threads of the book come together. It explores how imitation learning and Thompson sampling can be combined into hybrid systems that leverage the strengths of each approach. This is an area of active research, and the chapter covers both established integration strategies and emerging ideas. Chapter 8 then grounds everything in real-world applications, with detailed case studies in robotics, recommendation systems, and game-playing AI.

The final chapters address evaluation and future directions. Chapter 9 covers how to measure the performance of these systems rigorously, which metrics to use, and how to compare against baselines fairly. Chapter 10 looks ahead at advanced topics including multi-task learning, scalability challenges, and open research problems. The book closes with a conclusion that synthesizes the key lessons and points you toward next steps in your learning journey.

Throughout the book, you will find code examples, worked scenarios, and practical frameworks alongside the theoretical explanations. The goal is always to connect abstract concepts to concrete implementations. By the end, you should be able to not just explain these techniques but actually use them.

This book is written for intermediate learners and practitioners. It assumes you have some foundational knowledge of machine learning, but it does not assume you are an expert in either imitation learning or Thompson sampling. If you are coming in fresh to both topics, you are in exactly the right place. If you already know one of them well, you will still find substantial new material in the chapters covering the other.

Here is a clear picture of what background knowledge will help you get the most out of this book:

• Basic machine learning concepts: You should be comfortable with the ideas of training data, model parameters, loss functions, and gradient descent. You do not need to be an expert, but these terms should not be unfamiliar.
• Probability and statistics fundamentals: Thompson sampling is rooted in Bayesian probability, so comfort with concepts like probability distributions, conditional probability, and Bayes' theorem will be important. Chapter 5 reviews the key ideas, but having prior exposure will help.
• Reinforcement learning basics: Imitation learning is closely related to reinforcement learning. Familiarity with concepts like states, actions, policies, and reward functions will make the early chapters much easier to follow. A basic understanding of the Markov decision process (MDP) framework is particularly useful.
• Python programming: Code examples throughout the book are written in Python. You should be comfortable reading and writing Python code, including working with libraries like NumPy. Familiarity with PyTorch or TensorFlow is helpful but not strictly required.
• Linear algebra basics: Matrix operations, vectors, and basic linear algebra appear in several places, particularly in the chapters on inverse reinforcement learning and posterior distributions.

If you feel uncertain about any of these prerequisites, do not let that stop you from reading. The book is written to be as accessible as possible, and key concepts are explained when they first appear. However, if you find yourself struggling with the probability material in Chapter 5, it may be worth spending a few hours reviewing Bayesian inference before continuing. A resource like the first few chapters of Pattern Recognition and Machine Learning by Christopher Bishop, or even a good online tutorial on Bayes' theorem, can fill that gap quickly.^[3]

What you do not need is a deep background in advanced mathematics, formal proofs, or cutting-edge research. This book prioritizes intuition and practical understanding over mathematical rigor. Where proofs or derivations are relevant, they are presented in an accessible way with the emphasis on understanding the result rather than following every algebraic step. The goal is to make you a more effective practitioner, not to prepare you for a PhD qualifying exam.

It is also worth noting what this book is not. It is not a comprehensive survey of all imitation learning or bandit algorithms. The field is vast, and covering everything would require multiple volumes. Instead, the focus is on the most important, most widely used, and most practically relevant techniques. The goal is depth over breadth, giving you a thorough understanding of the core ideas rather than a shallow overview of everything that exists.

One final note on tools and environment: the code examples in this book are designed to run on a standard laptop or desktop computer. You do not need access to a GPU cluster or cloud computing resources to follow along, though having a GPU will speed things up for the more computationally intensive examples. Setting up a Python environment with standard scientific computing libraries (NumPy, SciPy, Matplotlib) and a deep learning framework (PyTorch is used in most examples) will prepare you for everything in the book.

One of the defining choices in writing this book was to prioritize practical understanding over theoretical completeness. There are excellent academic papers and textbooks that cover the mathematical foundations of imitation learning and Thompson sampling in exhaustive detail. This book takes a different approach. Every concept is introduced with the question: how does this actually work in practice, and what do you need to know to use it effectively?

This means the book is full of concrete examples. When behavioral cloning is introduced, you will see not just the algorithm but a worked example of training a simple cloning model on a navigation task. When Thompson sampling is explained, you will walk through a specific scenario, such as choosing between different content recommendations for a user, and see exactly how the algorithm makes its decisions step by step. Abstract concepts are always grounded in something tangible.

The practical focus also means the book addresses failure modes honestly. Many textbooks and tutorials present algorithms in their best-case scenarios. Real-world practitioners know that things go wrong in predictable ways, and understanding those failure modes is just as important as understanding the algorithm itself. Each major technique is accompanied by a discussion of when it tends to fail, what warning signs to look for, and what to do when it does not work as expected.

Example of a common failure mode: A team building a behavioral cloning system for a self-driving car trains their model on thousands of hours of expert driving data. In testing, the car performs well on straight roads but struggles badly at intersections. The problem is distribution shift: the expert rarely encountered certain unusual intersection configurations, so the model has almost no training data for those situations. Recognizing this pattern early, and knowing the standard mitigation strategies, can save weeks of debugging.

Throughout the book, you will also find structured frameworks and checklists that you can apply directly to your own projects. For example, Chapter 9 includes a step-by-step evaluation framework for imitation learning systems that covers metric selection, test set design, and comparison against baselines. These are not just theoretical guidelines. They are practical tools developed from real experience with these systems.

The code examples deserve special mention. Rather than presenting polished, production-ready code that obscures the underlying logic, the examples in this book are written to be readable and educational. Comments explain what each section is doing and why. Variable names are chosen for clarity rather than brevity. The goal is for you to understand the code well enough to modify it, extend it, and adapt it to your own use cases, not just copy and paste it.

Here is a simple illustration of the kind of code style used throughout the book. This snippet shows the core loop of a basic Thompson sampling implementation for a two-armed bandit problem:

import numpy as np

# Prior parameters for Beta distribution (alpha, beta) for each arm
alpha = np.ones(2)   # successes + 1
beta_params = np.ones(2)   # failures + 1

def select_arm(alpha, beta_params):
    # Sample from the posterior Beta distribution for each arm
    samples = np.random.beta(alpha, beta_params)
    return np.argmax(samples)

def update(arm, reward, alpha, beta_params):
    # Update posterior based on observed reward (1 = success, 0 = failure)
    alpha[arm] += reward
    beta_params[arm] += (1 - reward)
    return alpha, beta_params

Even in this short example, the structure of Thompson sampling becomes visible. You maintain a probability distribution over each option, sample from those distributions to make a decision, and update the distributions based on what you observe. The elegance of the approach is apparent even at this small scale, and the code makes the connection between the mathematical concept and the implementation concrete and immediate.

The hands-on approach extends to the application chapters as well. Rather than describing robotics or recommendation systems at a high level, Chapter 8 walks through specific scenarios with enough detail to understand the design decisions involved. Why did the robotics team choose behavioral cloning over inverse reinforcement learning for their initial prototype? What made Thompson sampling a better fit than a simpler epsilon-greedy approach for the recommendation engine? These are the kinds of questions that matter when you are building real systems, and they are the questions this book is designed to help you answer.

By the time you finish this book, the goal is not just that you understand imitation learning and Thompson sampling intellectually. The goal is that you feel equipped to open a code editor, start a new project, and apply these techniques with confidence. You will know what questions to ask, what pitfalls to watch for, and how to evaluate whether your system is actually working. That practical readiness is what this book is ultimately about.