GPU Memory Hierarchies Mastery

The numbers tell a clear story. NVIDIA's H100 SXM5 delivers approximately 3,958 TFLOPS of FP8 tensor compute throughput, but its HBM3 memory subsystem provides roughly 3.35 TB/s of bandwidth.^[1] At first glance, 3.35 TB/s sounds enormous. In practice, many transformer inference workloads are so memory-bound that the GPU's arithmetic units sit idle for a substantial portion of each kernel's execution. The compute-to-bandwidth ratio, often called arithmetic intensity, determines whether a workload is compute-bound or memory-bound. For large language model inference, arithmetic intensity frequently falls well below the hardware's roofline threshold, meaning bandwidth is the binding constraint, not FLOPS.

This situation has a name in the research community: the memory wall. The concept was first described in the mid-1990s, but it has become dramatically more acute in the GPU era.^[2] The core problem is that transistor density and clock speeds have enabled massive parallelism in compute units, but DRAM physics impose hard limits on how fast data can be moved. High Bandwidth Memory (HBM) stacking has helped, but it has not closed the gap. Each new GPU generation adds more tensor cores faster than it adds memory bandwidth, making the ratio worse over time, not better.

The practical consequences show up in utilization metrics. In a typical large-scale training cluster, GPU utilization as reported by nvidia-smi may look healthy at 80-90%. But that metric measures whether the GPU is executing any kernel, not whether its compute units are actually busy. A more revealing metric is SM (Streaming Multiprocessor) active cycles versus memory stall cycles, which can be measured using NVIDIA's Nsight Compute profiler. In memory-bound workloads, it is common to find that 40-60% of SM cycles are stalled waiting for memory transactions to complete. That is compute capacity you are paying for but not using.

The crisis is compounded by the scale of modern deployments. A data center running 1,000 H100 GPUs for LLM training or inference represents a capital investment in the range of $30-40 million in hardware alone, plus substantial ongoing power and cooling costs.^[3] If 40% of that compute capacity is being wasted on memory stalls, the financial impact is not abstract. It translates directly into longer training runs, higher inference latency, and more hardware required to meet throughput targets. Closing even a fraction of that gap through systematic memory optimization has a measurable return on investment.

The good news is that the memory wall is not a fixed barrier. It is a set of engineering constraints that can be navigated with the right techniques. The GPU memory hierarchy provides multiple levels of storage with very different bandwidth and latency characteristics. Registers offer the highest bandwidth and lowest latency, followed by shared memory, L1 cache, L2 cache, and finally global HBM. The difference in effective bandwidth between registers and global memory spans roughly two orders of magnitude. Optimization strategies that keep data in faster memory levels, reduce redundant global memory accesses, and maximize the reuse of cached data can dramatically shift the effective arithmetic intensity of a workload.

Consider a concrete example. A naive implementation of a multi-layer perceptron forward pass might read input activations from global memory, compute a matrix multiplication, write the result back to global memory, then read it again for the activation function, and write it back again. Each round trip to HBM costs bandwidth. A fused kernel that performs the matrix multiplication and activation function in a single pass, keeping intermediate results in registers or shared memory, can reduce global memory traffic by 50% or more for that operation. Multiplied across thousands of layers and billions of training steps, that reduction compounds into a significant performance advantage. This is the core logic that drives the optimization strategies throughout this book.

Understanding the bottleneck is the first step. The chapters that follow provide the tools to address it systematically, starting with the cache coherency protocols that govern how data moves between GPUs, through the register and shared memory management techniques that keep data close to compute, and out to the pipeline-level design decisions that determine how efficiently your entire system uses the memory bandwidth available to it.

Every optimization decision in this book connects back to a clear understanding of the GPU memory hierarchy. Unlike CPU memory hierarchies, which are designed around sequential access patterns and cache-friendly pointer chasing, GPU memory hierarchies are optimized for massive parallelism and high-throughput streaming access. The design priorities are different, and so are the failure modes. Getting the most out of a GPU requires understanding not just what each level of the hierarchy does, but how the hardware uses it under different access patterns.

The hierarchy, from fastest to slowest, looks like this:

• Registers: Per-thread storage, zero-latency access, total capacity of roughly 256 KB per SM on Ampere and Hopper architectures. Registers are the fastest storage on the chip, but they are a finite resource shared among all active warps on an SM.
• Shared Memory / L1 Cache: Per-SM storage, configurable split between shared memory and L1 cache, typically 128-256 KB total per SM on modern architectures. Latency is approximately 20-30 cycles. Shared memory is explicitly managed by the programmer; L1 cache is hardware-managed.
• L2 Cache: Chip-wide cache shared across all SMs, typically 40-80 MB on current high-end GPUs. Latency is approximately 200-300 cycles. The L2 cache is critical for inter-SM data sharing and for absorbing irregular access patterns that would otherwise hit HBM.
• Global Memory (HBM): Off-chip DRAM, the largest and slowest tier. Capacity ranges from 40 GB to 192 GB on current data center GPUs. Latency is approximately 600-800 cycles, but high bandwidth (3+ TB/s on HBM3) partially compensates when access patterns are coalesced.
• NVLink / PCIe (Inter-GPU): For multi-GPU systems, data movement between GPUs adds another layer of latency and bandwidth constraints. NVLink 4.0 provides up to 900 GB/s bidirectional bandwidth per GPU in NVSwitch-connected systems, but this is still far below intra-GPU HBM bandwidth.

The key insight is that effective bandwidth at each level differs dramatically from peak bandwidth. Registers deliver their full bandwidth only when threads access distinct register locations without bank conflicts. Shared memory has 32 banks on current NVIDIA architectures, and accessing the same bank from multiple threads in the same warp serializes those accesses, reducing effective bandwidth by up to 32x in the worst case.^[4] Global memory delivers peak bandwidth only when warp accesses are fully coalesced into 128-byte aligned transactions. Uncoalesced access patterns can reduce effective HBM bandwidth by 10x or more.

The concept of occupancy ties the hierarchy together. Occupancy refers to the ratio of active warps on an SM to the maximum number of warps the SM can support. Higher occupancy generally helps hide memory latency by allowing the warp scheduler to switch to a ready warp while another warp waits for a memory transaction. However, higher occupancy requires more warps, which means more registers and more shared memory per SM. This creates a direct tension: using more registers per thread reduces occupancy, which reduces the scheduler's ability to hide latency, but using fewer registers may force spilling to local memory (which maps to global memory), which is far worse. Managing this tradeoff is one of the central skills in GPU kernel optimization, and it is covered in depth in Chapter 3.

Modern GPU architectures have introduced features that blur some of these hierarchy boundaries. NVIDIA's Hopper architecture introduced the Tensor Memory Accelerator (TMA), a hardware unit that can perform asynchronous bulk data transfers between global memory and shared memory without involving the SM's compute units.^[5] This allows kernels to overlap computation with data loading more effectively than was possible with software-managed prefetching in earlier architectures. AMD's CDNA3 architecture similarly introduced enhanced prefetch mechanisms and larger L2 caches to address the same bandwidth challenge from a different angle.

Understanding the hierarchy also means understanding what happens when you get it wrong. A common mistake in production ML code is using cudaMalloc and cudaMemcpy for every intermediate tensor without considering whether those tensors could be kept in shared memory or registers across multiple operations. Another common mistake is launching kernels with thread block sizes that are not multiples of 32 (the warp size), which wastes compute resources and reduces occupancy. A third mistake is using atomic operations on global memory for reductions when shared memory atomics or warp-level primitives like __reduce_add_sync would be far more efficient. These are not edge cases. They appear regularly in production codebases, and they have measurable performance costs.

The framework in this book treats the memory hierarchy as the primary design constraint for every optimization decision. Before asking "how do I make this kernel faster," the right question is "where is the data, where does it need to go, and how many times does it need to move?" Answering those questions precisely, using profiling data rather than intuition, is the foundation of the approach developed in the chapters ahead.

One more concept worth establishing here is the distinction between bandwidth-bound and latency-bound kernels. Bandwidth-bound kernels are limited by the total volume of data that can be transferred per second. Latency-bound kernels are limited by the time required to complete individual memory transactions, even if the total data volume is small. The optimization strategies for these two cases are different. Bandwidth-bound kernels benefit from access pattern optimization, kernel fusion, and data reuse. Latency-bound kernels benefit from occupancy increases, prefetching, and warp-level parallelism. Correctly diagnosing which regime your kernel is in is the first step in choosing the right optimization approach.

Optimization without a framework is guesswork. It is easy to spend days tuning a kernel that is not actually the bottleneck, or to apply a technique that helps in microbenchmarks but hurts in production due to interactions with other parts of the system. The framework presented in this book is built around four principles: measure first, model the bottleneck, apply targeted interventions, and validate at scale. These principles apply whether you are optimizing a single CUDA kernel or redesigning the memory architecture of a thousand-GPU training cluster.

The measure first principle sounds obvious, but it is violated constantly in practice. Engineers often apply optimizations based on intuition or prior experience with different hardware, different workloads, or different scales. The GPU performance landscape changes significantly across architectures. A technique that was critical on Volta may be less important on Hopper due to architectural changes in the memory subsystem. The only reliable way to know what is actually limiting performance is to measure it with the right tools. Chapter 8 covers the full toolkit for GPU performance measurement, including NVIDIA Nsight Compute, Nsight Systems, and the CUPTI performance counter API. The key metrics to establish before any optimization work are:

• Memory bandwidth utilization (achieved vs. peak HBM bandwidth)
• SM active cycles vs. memory stall cycles
• L1 and L2 cache hit rates
• Warp occupancy and warp stall reasons
• Tensor core utilization (for matrix-heavy workloads)

The model the bottleneck principle means building a quantitative understanding of why performance is limited, not just observing that it is. The roofline model is the standard tool for this analysis.^[6] A roofline model plots achieved performance (in FLOPS) against arithmetic intensity (in FLOPS per byte of memory traffic) and overlays the hardware's peak compute and peak bandwidth limits. Any kernel that falls below the roofline is leaving performance on the table. The model tells you whether the gap is on the compute side or the memory side, which determines which class of optimizations to pursue. For most ML workloads in current data centers, the gap is on the memory side, and the roofline model makes this visible in a way that raw utilization numbers do not.

The apply targeted interventions principle means matching the optimization technique to the specific bottleneck identified by measurement and modeling. This book organizes its optimization strategies into a hierarchy that mirrors the memory hierarchy itself:

1. Register and shared memory optimization (Chapters 2-3): Reduce register pressure, eliminate shared memory bank conflicts, maximize data reuse within an SM.
2. Kernel-level optimization (Chapters 3-4): Tune occupancy, apply kernel fusion, eliminate redundant global memory traffic.
3. System-level optimization (Chapters 5-7): Optimize unified memory usage, manage CPU-GPU data movement, apply power-aware scheduling.
4. Infrastructure-level optimization (Chapters 8-10): Design monitoring systems, evaluate architectures, build end-to-end pipelines that sustain high throughput at scale.

The validate at scale principle addresses a failure mode that is specific to data center environments. An optimization that improves single-GPU performance by 20% may deliver less than 5% improvement in a distributed training job if it introduces synchronization overhead or changes the communication pattern between GPUs. Validation must happen at the scale at which the workload actually runs. This means having benchmarking infrastructure that can reproduce production conditions, including realistic batch sizes, realistic model sizes, and realistic inter-GPU communication patterns. Chapter 8 provides a methodology for building this infrastructure and using it to validate optimization results before deploying them to production.

A practical example illustrates how the framework works end to end. Consider a transformer training job running on 64 H100 GPUs that is achieving 45% of theoretical peak throughput. The first step is measurement: Nsight Compute reveals that the attention kernel has a 38% L2 cache hit rate and that 52% of SM cycles are memory stall cycles. The roofline model confirms the kernel is memory-bound, operating at roughly 0.8 FLOPS per byte of memory traffic against a hardware roofline threshold of approximately 65 FLOPS per byte for this architecture. The bottleneck is clearly memory bandwidth, not compute.

The targeted intervention in this scenario is FlashAttention, a kernel fusion technique that restructures the attention computation to tile the query, key, and value matrices through shared memory, dramatically reducing global memory traffic.^[7] After applying FlashAttention, the same measurement process reveals that the attention kernel's memory stall cycles drop to 18%, L2 hit rate improves to 71%, and overall throughput increases to 68% of theoretical peak. Validation at scale confirms that the 64-GPU job sees a 41% end-to-end speedup, with no regression in model convergence.

This example is not hypothetical. FlashAttention and its successors have delivered exactly these kinds of improvements in production deployments, and the measurement-driven process described above is how those improvements were identified, implemented, and validated. The framework works because it is grounded in physics and measurement rather than heuristics and intuition. Every chapter in this book applies the same framework to a different layer of the optimization stack.

One important caveat: optimization is not free. Every technique in this book involves tradeoffs. Kernel fusion reduces memory traffic but increases kernel complexity and can make debugging harder. Higher occupancy reduces register pressure but may increase shared memory contention. Aggressive prefetching improves latency hiding but consumes memory bandwidth for speculative loads. The framework helps you make these tradeoffs explicitly and quantitatively, rather than discovering them accidentally in production. Understanding the cost of each optimization is as important as understanding its benefit.

This book is designed to be read in sequence, but it is also structured so that you can navigate directly to the chapters most relevant to your current challenges. The chapters build on each other conceptually, with later chapters assuming familiarity with the techniques introduced in earlier ones. However, each chapter is also self-contained enough to serve as a reference when you are working on a specific problem. The following guide maps common data center challenges to the chapters that address them most directly.

If your primary concern is multi-GPU training efficiency, start with Chapter 1 on cache coherency protocols and multi-GPU synchronization. This chapter explains how data consistency is maintained across GPU memory spaces, how different synchronization strategies affect communication overhead, and how to select the right protocol for your workload's communication pattern. From there, Chapter 5 on unified virtual addressing covers the memory management layer that sits above the hardware protocols, and Chapter 10 on ML pipeline design ties everything together at the system level.

If your primary concern is single-kernel performance, Chapters 2, 3, and 4 form the core sequence. Chapter 2 covers L1 cache architecture and register file optimization, providing the foundation for understanding how data moves within an SM. Chapter 3 covers tensor core utilization and occupancy optimization, which is where most of the performance leverage exists for matrix-heavy workloads. Chapter 4 covers kernel fusion, which is the highest-impact single technique for reducing memory bandwidth pressure in production ML code.

If your primary concern is infrastructure design and scaling, Chapters 7, 8, and 9 are the most relevant starting points. Chapter 7 covers power efficiency and thermal management, which become critical constraints at scale. Chapter 8 covers monitoring and benchmarking, which are the operational foundation for any optimization program. Chapter 9 covers architecture evaluation, which is essential for making informed decisions about hardware procurement and upgrade cycles.

For those working on sparse model architectures, including mixture-of-experts models and pruned networks, Chapter 6 on sparse tensor operations is essential reading. Sparse computations have fundamentally different memory access patterns than dense computations, and the optimization strategies that work for dense matrix multiplication can actually hurt performance for sparse workloads. Chapter 6 covers the storage formats, access pattern optimizations, and synchronization mechanisms specific to sparse computation.

Regardless of your starting point, the following chapters are recommended reading for everyone:

• Chapter 2 (L1 Cache and Register Optimization): The register file is the foundation of all GPU performance. Understanding register pressure and its relationship to occupancy is prerequisite knowledge for every other optimization technique in the book.
• Chapter 8 (Monitoring and Benchmarking): You cannot optimize what you cannot measure. The profiling and benchmarking methodology in this chapter is the operational backbone of the entire optimization framework.
• Chapter 10 (ML Pipeline Design): System-level design decisions often have more impact than kernel-level optimizations. This chapter shows how to architect pipelines that sustain high throughput across the full training or inference workflow.

Each chapter follows a consistent structure. It opens with a clear statement of the problem being addressed and the performance impact of solving it. It then covers the underlying technical concepts with enough depth to support the optimization techniques that follow. The core of each chapter is a set of specific, actionable optimization strategies, each accompanied by concrete examples, measurable outcomes, and implementation guidance. Chapters close with a summary of key tradeoffs and a set of diagnostic questions you can use to determine whether the techniques in that chapter apply to your specific workload.

Throughout the book, code examples are provided in CUDA C++ unless otherwise noted. The examples are written to be illustrative rather than production-ready, prioritizing clarity over completeness. Where relevant, equivalent approaches in ROCm HIP are noted, and the architectural concepts apply to both NVIDIA and AMD GPU platforms unless a specific feature is hardware-specific. Performance numbers cited in examples are based on measurements on H100 SXM5 hardware unless otherwise specified, and results will vary on other architectures.

A note on the pace of change in this field: GPU architectures evolve rapidly, and some specific numbers and features will change with new hardware generations. The principles, however, are durable. The roofline model, the memory hierarchy tradeoffs, the occupancy-latency relationship, and the kernel fusion logic have all remained valid across multiple GPU generations because they are grounded in fundamental physics and computer architecture principles. Where specific architectural details are likely to change, the text notes this explicitly and focuses on the underlying principle rather than the specific implementation. The goal is to give you a mental model that remains useful as the hardware evolves, not just a set of recipes that work on today's silicon.

Finally, this book assumes that you are working in a production environment where performance improvements have real consequences. The techniques here are not academic exercises. They are the result of hard-won experience optimizing real workloads on real hardware at real scale. The examples are drawn from scenarios that appear regularly in large-scale ML infrastructure: transformer training, large-scale inference serving, scientific simulation, and recommendation system training. If you bring the same rigor to your optimization work that this book applies to its analysis, you will find measurable, sustainable performance improvements that compound over time into a significant competitive advantage.