Pitfalls of Unified Memory Models in GPUs

Introduction and Motivation

• The presenter starts by asking the audience about their knowledge of GPUs and then shifts to a tangent about two cars, an MG MGB GT and a modern Volvo, to illustrate the difference in complexity and repairability between old and new technology (19s).
• The presenter notes that despite the advancements in technology, the way humans interact with these vehicles has not changed much, and this idea will be relevant throughout the talk (1m9s).
• The presenter shares their personal experience of starting a new job and being tasked with making a program faster, despite having no prior knowledge of GPUs or CUDA (1m45s).
• The presenter's initial attempt at writing a CUDA program resulted in a simple function that sometimes worked and sometimes didn't, depending on the hardware and software (2m31s).

Initial GPU Programming Challenges

• The presenter is puzzled by the inconsistent behavior of the program and sets out to answer two questions: why the program sometimes works and why the otherwise performant program is slow (3m12s).
• The presenter briefly mentions the differences between CPUs and GPUs, noting that CPUs typically have many threads that run, but does not fully elaborate on this point (3m32s).

Understanding GPUs and Concurrency

• Writing code for GPUs is declarative, where the programmer specifies what they want to achieve, and the hardware decides how to execute it, often implicitly making the code concurrent (3m54s).
• A CPU is like an office where multiple people work on relatively independent tasks, whereas a GPU is like a factory with specialized equipment for specific tasks, making one thing very well (4m23s).
• In a factory analogy, storage space contains raw materials, similar to memory in computing, and dedicated space for each person or team can be thought of as shared workspaces (4m49s).
• Implicit concurrency in GPUs is achieved by specifying a small program that divides up a range and expects it to run in parallel, with each thread processing a portion of the range (5m41s).

Streams and Data Processing

• The concept of a moving assembly line, introduced by Henry Ford, revolutionized manufacturing by having workers stay fixed while the pieces moved, and this idea is preferred in computing when possible (6m37s).
• In computing, the moving assembly line concept is preferred when hardware can perform specific tasks with no dependency between each other, and this is encapsulated via the notion of a stream (7m7s).
• A stream is an ordered sequence, and this concept is important in understanding how GPUs process data (7m9s).
• GPUs can handle tasks in a consistent way through the use of streams, which allow for the execution of logically consistent sets of operations, enabling the hardware to be utilized more efficiently when tasks spend most of their time reading and writing from memory (7m12s).

GPU Performance and Big O Notation

• A high computation to data ratio is ideal for GPUs, as seen in the example of matrix multiplication, where the computation time is O(n^3) and the data is O(n^2), resulting in a significant speedup when using GPUs (8m15s).
• However, the Big O notation can be misleading, as asymptotically better algorithms may not be faster on GPUs due to the importance of constants in these settings (8m57s).

Memory Management in GPUs

• The goal is to write a mem copy function that can handle any type of copy efficiently, regardless of whether the memory is on the CPU or GPU (9m17s).
• There are three types of memory to consider on a GPU: memory allocated using Cuda malloc, which is prioritized and remains in the same location; memory allocated using Cuda malloc, which is allocated in system RAM; and memory that comes from other sources (9m51s).
• Cuda malloc is a special type of memory that is constantly prioritized and will never migrate, requiring manual decision to move it to a different location (10m9s).
• Unified memory models in GPUs allow for direct access to system memory by the graphics card, making it a desirable feature (10m44s).

Unified Memory Model

• Managed memory or unified memory is a type of memory allocation where the physical storage may be either on a hardware device (GPU) or in system memory, and its location can change depending on the program's needs (11m4s).
• This type of memory allocation is achieved through a series of system calls, including opening a file descriptor for a special device provided by Nvidia, allocating a large slab of memory via mmap, and deallocating memory (12m5s).
• The Nvidia-provided device serves as an interface between system memory and device memory, allowing for the allocation and deallocation of memory (12m11s).
• When allocating memory, the program specifies the address where the memory should exist, ensuring that the pointers are the same across both the system and the device (13m10s).
• Deallocating memory involves remapping the memory and freeing what's left over, which can be seen in the Nvidia System Profiler (13m38s).
• The Nvidia System Profiler is a useful tool for understanding what happens when using unified memory models, showing the program's memory allocation and deallocation calls (14m3s).

System Calls and Page Faults

• Using tools like estrace can help understand the underlying system calls and memory allocation processes involved in unified memory models (11m49s).
• When a program runs, it immediately encounters a page fault due to the way the driver enforces memory allocation, resulting in the allocation of physical storage and handling of the page fault, which can take a significant amount of time (14m29s).
• The time it takes to handle the page fault can be substantial, and in some cases, it can take up a great deal of the program's running time, especially when allocating more memory (15m12s).
• The hardware tries to help by allocating more memory each time a page fault occurs, resulting in an irregular pattern of page faults with increasing spacing between them (15m32s).
• This pattern repeats, but after a certain point, the hardware seems to forget that the program is running into page faults and stops helping, resulting in a continued pattern of page faults (15m59s).

Data Movement and Coherency

• The physical location of data in unified memory models is invisible to a program and may be changed at any time, even if the program is accessing the data through a shared pointer (16m27s).
• This is a rare occurrence in programming, where memory can move in a way that is entirely opaque and matters significantly, unlike caches where data is just copied and stored (16m57s).
• The access to data through a virtual address will remain valid and coherent from any processor, regardless of locality, which is necessary due to the complexity of the system (17m23s).
• The requirement for accesses to remain valid has significant implications for the performance of programs using unified memory models (17m45s).

Memory Copy Performance Issues

• Existing functions, such as cudaMemcpy, can be used to copy data in CUDA, which may be a solution to the issues encountered with unified memory models (17m57s).
• A memory copy operation is performed on the CPU, which should be fast since the pointers are accessible via the CPU, but it falls into a pitfall, resulting in poor performance (18m5s).
• The performance issue is illustrated in a graph, showing a significant decrease in throughput as the size of the data being copied increases, with a drop from over 600 megabytes per second to around 0.085 megabytes per second (18m25s).
• The reason for this poor performance is due to CPU page faults, which occur when the CPU tries to access a page of memory that is not currently available, resulting in a request to the GPU to allocate space (19m44s).
• The page faults are caused by the interaction between the device (graphics card) and the CPU, which is limited by the page size of the CPU, resulting in a large number of page faults (around 37,000) (20m32s).

Page Faults and Performance Impact

• Each page fault results in a CIS call, which is expensive and leads to poor performance, with the page faults being 4 kilobyte in size, which may be improved by increasing the page size (21m15s).
• The page faults occur regularly and repeatedly, resulting in a significant impact on performance, as shown in a zoomed-in view of the profile (22m0s).
• The GPU's hardware attempts to help with memory management by transferring more memory than requested, resulting in a pattern of page faults and speculative prefetches that repeat and are unequally spaced (22m17s).
• Each red line represents a page fault for 4K, while purple lines represent a speculative prefetch, with the width of each block growing as the number of page faults increases (22m53s).
• The hardware handles this process automatically, without the need for user intervention, and the amount of memory transferred can range from 64 kilobytes to 2 GB (23m2s).

CUDA Managed Memory and Prefetching

• Using CUDA's managed memory with specified copying from the device to the CPU can result in slightly faster performance, but still exhibits a stepwise curve due to prefetches (23m41s).
• Copying from the device to the host results in a faster line, reaching up to 10 GB per second, due to the GPU's capacity for dealing with larger pages up to 2 megabytes (24m15s).

Using CUDA Functions for Memory Management

• It is recommended to use CUDA functions instead of standard functions when writing code that needs to deal with memory management generically, as they can provide faster performance (24m40s).
• Using CUDA functions can result in a speedup of roughly twice as fast as standard functions, even in the slowest case, with a simple one-line change (24m50s).

Memory Migration and Performance

• The cost of memory management comes from handling page faults and physical memory mapping, leading to the question of whether moving memory to the same location before copying would be more efficient (25m21s).
• CUDA allows for memory migration, which can potentially improve performance by eliminating the need for copying between GPU and CPU (25m41s).
• Unified memory models in GPUs allow for memory to be moved between devices, and the size of the memory to be specified, enabling the splitting of arrays across multiple devices, including the CPU and GPU, which can lead to a different type of parallelism (26m13s).
• This parallelism allows for updates to be made on an array on the GPU and CPU simultaneously, thanks to the guarantees provided by the unified memory model (26m45s).

Prefetching and Performance Bottlenecks

• Prefetching the source pointer to the device can result in a significant improvement in performance, with a flat line of around 16 GB per second, but this performance drops drastically when the destination array can no longer fit in memory (27m1s).
• The performance drop is due to the occurrence of page faults, which happen when the system tries to write to or read something that is not already in memory (27m38s).
• Prefetching both pointers ahead of time can result in a huge amount of performance, but this also leads to a significant drop in performance when neither pointer fits in memory anymore (28m3s).
• The system's ability to manage memory proactively is compromised when both pointers are prefetched, leading to poor performance (28m34s).
• The system's decision on what to evict from memory can be flawed, leading to poor performance, as it may evict important data (29m19s).

Managing Memory Accesses

• A significant issue with unified memory models in GPUs is the constant reading of data from memory that is not necessary, resulting in continued page faults for reads and writes, indicating the need for more careful management of memory copies (29m38s).
• A solution to this issue is to manage memory accesses more efficiently, as shown in a profile from a running program where memory accesses are optimized to reduce bandwidth consumption and page faults (30m3s).
• If device memory is full, it will never get evicted, and using the Cuda malloc API can cause the graph to shift arbitrarily to the left by allocating more card memory (30m45s).
• Moving large amounts of data, such as 512 megabytes, repeatedly can cause slow bandwidth due to constant eviction of necessary data (30m56s).

Optimizing Memory Copies

• To manage memory copies more carefully, it is necessary to define a prefetch size, create multiple streams to cue operations properly, and calculate the number of prefetches needed (31m23s).
• A loop can be used to start the copy process, prefetching the previous data and sending it back to the CPU, and then prefetching the blocks needed from system memory back onto the card (31m49s).
• The CPU needs to be involved in remapping data back into its own memory, and this can be done by explicitly sending the data back to the CPU and using a different stream for prefetching (32m30s).
• Using a different stream for prefetching may not significantly improve performance, but it is still a better approach (33m1s).
• By combining various prefetches, sending data back to its original location, and managing memory, performance can be significantly improved, with the custom code potentially beating standard CUDA functions for certain tasks (33m56s).

Complexity of Unified Memory

• The use of a single pointer across devices and system memory was initially thought to simplify things, but it has proven to be more complicated than expected (34m43s).
• The complexity of managing memory and pointers is due in part to the fact that computers are still being programmed with concepts and tools designed 50 years ago, such as those used for the PDP1 (35m27s).
• The idea of changing programming languages to express hardware more explicitly was discussed in the 1990s, but it was met with resistance, and the tools used today are still largely the same (35m46s).
• It is impossible for a compiler to statically determine whether a function or code will work due to the inability to succinctly express the properties of a pointer, making it difficult to ensure safe and reliable operation (36m7s).
• The hardware is capable of handling complex memory operations, but the lack of explicit expression of pointer properties makes it difficult to write safe and reliable code (36m30s).
• The resistance to change in programming tools and concepts has not been beneficial, and it is time to reconsider the way computers are programmed to better match the capabilities of modern hardware (36m43s).

Profiling and Code Simplicity

• To understand performance issues in code, it's essential to profile the code using different tools and try to understand why things are happening the way they are, as there is always a reason for it, no matter how complex it may seem (36m50s).
• When writing code, it's crucial to consider simplifying things to make it easier for others to understand, and to prioritize performance when in doubt (37m39s).

Unified Memory: Benefits and Drawbacks

• Unified memory models in GPUs, sold by companies like Nvidia and AMD, can make it easier to port old CPU applications to GPUs, but they may not be worth using in performance-critical environments (38m22s).
• Unified memory models can be useful for migration or compatibility reasons, or when more memory is needed than is available on the GPU, but they may not be better than manually handling memory copies (38m48s).
• In some cases, the overhead of unified memory models may not be related to page faults, but rather to coherency issues, where the device hardware needs to ensure a consistent view of memory across the system (39m40s).
• Even if the same page is requested multiple times on the same device, the hardware may still incur overhead to ensure coherency, and counting page faults may not be enough to overcome these issues (40m11s).