Solving AI's Hardware Problem with Compute in Memory

Table of Contents:

1. Introduction
2. The Memory Wall Problem
   • The Von Neumann Computer
   • The Memory Wall
3. Challenges with Adding More Memory
   • Practical and Technological Limits
   • Energy Limitations
   • Economic Factors
4. Memory Scaling Issues
   • Historical Context
   • Transistor Density Optimization
   • Challenges in DRAM Scaling
   • Security Vulnerabilities
5. New Hardware Solutions
   • Compute in Memory
   • An Idea Decades Old
   • Practical Shortcomings
     • Levels of Implementation: Device
     • Circuits
     • System
6. Conclusion

The Memory Wall Problem and Innovative Solutions

In the fast-growing field of deep learning, the size and complexity of models have increased exponentially over the past decade. However, this rapid growth is posing significant challenges to the existing hardware infrastructure. The industry's biggest models, such as OpenAI's Dall-E 2 and Google's Imagen and GPT-3, now have billions of parameters. The demand for even larger models, with trillions of parameters, is on the horizon. However, the ability of Current hardware to accommodate these models is being strained, particularly in terms of memory usage.

The Von Neumann Computer

Most modern computers adopt a Von Neumann architecture, where both instructions and data are stored in the same memory bank. Processing units, such as CPUs or GPUs, access memory to execute instructions and process data. While this architecture has been instrumental in advancing software capabilities, it operates quite differently from the human brain. The brain's compute ability integrates tightly with memory and input/output communication, while computers separate compute from memory and communication, resulting in memory-related challenges.

The Memory Wall

The AI hardware industry is attempting to Scale up memory and processing unit performance to keep up with the growing demands of deep learning models. For example, the latest Nvidia data center GPUs now offer up to 80GB of memory. However, the performance of hardware is not keeping pace with the rapid growth of models, especially when it comes to memory. Leading-edge models can easily require hundreds of gigabytes of memory, and a trillion-parameter model may necessitate the use of hundreds of GPUs. As a result, processing units often waste cycles waiting for data to travel between memory and compute units, leading to a memory capacity bottleneck commonly referred to as the Memory Wall.

Adding More Memory: Practical Limitations

While increasing memory capacity seems like an obvious solution, practical and technological limits restrict the amount of extra memory that can be added. Challenges such as connection and wiring issues, energy limitations, and the cost of both capital investment and ongoing operation hinder straightforward memory expansion. Widening a highway does not necessarily alleviate traffic congestion, and adding more memory to GPUs faces similar limitations. Energy consumption is a significant concern, as accessing memory off-chip uses significantly more energy than computing operations. Moreover, the increasing cost of AI hardware poses economic challenges, potentially limiting the benefits of advanced AI to only the wealthiest tech giants or governments.

Memory Scaling: Historical Challenges

The memory scaling problem can be traced back to the adoption of Dynamic Random Access Memory (DRAM) in the 1960s and 70s. While DRAM offered low latency and cost-effective manufacturing, it struggled to keep up with compute scaling. The CPU and GPU industries focused on optimizing transistor density, while memory had to address scaling capacity, bandwidth, and latency simultaneously. Over time, memory capacity and bandwidth improved, but latency enhancements lagged behind, resulting in a widening performance gap.

Another challenge arose as transistor sizes reduced. Smaller DRAM cells, with capacitors storing 1 bit of data, became more susceptible to electrical noise and security vulnerabilities. As a result, memory industry innovation faced significant roadblocks, limiting progress in memory scaling.

New Hardware Solutions: Compute in Memory

To overcome the limitations of current hardware architecture, researchers propose adopting a memory-centric paradigm known as "Compute in Memory." This concept refers to integrating processing elements into Random Access Memory (RAM), allowing computations to occur within memory cells. While the idea has been explored for decades, recent advancements in technologies like Resistive Random Access Memory (ReRAM) and spin-transfer torque magnetoresistive Random Access Memory (STT-MRAM) offer promising possibilities.

ReRAM, for example, stores information by adjusting the electrical resistance of a material, enabling direct computation within the memory cells. However, challenges remain in commercializing these emerging memory technologies, as substantial hurdles need to be addressed.

Implementing Compute in Memory

Implementing Compute in Memory can occur at three levels: the device, circuit, and system. At the device level, alternative memory hardware, such as ReRAM and STT-MRAM, offers new possibilities beyond traditional DRAM and SRAM. Circuit-level implementations modify peripheral circuits to enable calculations within memory arrays themselves, exploiting the memory's internal bandwidth. Prominent examples like Ambit propose in-memory accelerators that perform logic operations by activating multiple rows of memory to execute AND/OR functions. However, challenges remain in supporting more complex logic operations.

At the system level, innovative packaging technologies like 2.5D or 3D memory die stacking enable closer integration of processing units and memory. By stacking multiple RAM dies on top of a CPU die and utilizing Through-Silicon Vias (TSVs), internal bandwidth can be greatly increased. These advancements, such as AMD's 3D V-Cache, offer the potential to significantly boost memory capacity, enhancing memory and logic integration without the need for a single die.

Conclusion

As deep learning models Continue to grow in complexity, addressing the memory wall problem becomes crucial for advancing AI capabilities. While the current hardware paradigm faces limitations in terms of memory capacity and compute efficiency, innovative solutions like Compute in Memory offer a way forward. By integrating processing elements into memory and utilizing emerging memory technologies, such as ReRAM, researchers aim to overcome existing challenges. However, the path to fully realizing these ideas faces practical hurdles, and industry progress will be essential to making substantial improvements over current hardware architectures.