Unleashing Extreme Scale AI Computing with Cerebras

Table of Contents

• Introduction to Cerebrus
• The Vision Behind Cerebrus
• The Growth of Deep Learning and AI
• Challenges with Traditional Processors
• The Solution: Wafer Scale Engine
• Features of the Wafer Scale Engine
• Cerebrus Server: Physical Chassis and Integration
• Software Stack: Compiler and SDK
• Applications of Cerebrus in Commercial Space
• Applications of Cerebrus in Research Labs
• Applications of Cerebrus in HPC
• What's Next: New Execution Mode

Introduction to Cerebrus

Cerebrus is an AI computer systems company founded in 2016. With offices in Silicon Valley, San Diego, Toronto, and Japan, Cerebrus is dedicated to developing a fundamentally new compute solution for deep learning and artificial intelligence. Their goal is to accelerate the compute power of AI and transform the landscape of computing.

The Vision Behind Cerebrus

The team at Cerebrus, comprised of experienced chip architects, system engineers, and business leaders, came together to Create a machine that could accelerate deep learning workloads by a significant margin. They recognized the tremendous opportunity in AI and saw the limitations of existing processors to keep up with the demands of deep learning.

The Growth of Deep Learning and AI

Deep learning and AI have experienced explosive growth in recent years. The models used for natural language processing, such as BERT and GPT-3, have grown in complexity and size, requiring massive amounts of compute power and memory. The compute requirements for state-of-the-art deep learning models Continue to increase, posing a challenge for traditional architectures.

Challenges with Traditional Processors

Traditional processors were not designed to handle the demands of deep learning and AI workloads. While progress has been made in scaling out large neural network workloads over clusters of traditional machines, this approach has limitations in terms of efficiency and scalability. Programming large clusters of CPUs or GPUs can be challenging, and traditional architectures struggle to meet the memory bandwidth and compute requirements of deep learning.

The Solution: Wafer Scale Engine

Cerebrus developed the Wafer Scale Engine (WSE) as a new compute solution for deep learning. The WSE is a revolutionary processor architecture that offers extreme-scale computation and flexible compute capabilities. With wafer-level integration, the WSE provides a massively Parallel array of individually programmable compute cores, connected by a high-bandwidth, low-latency interconnect mesh.

Features of the Wafer Scale Engine

The Wafer Scale Engine boasts several key features that set it apart from traditional processors. It has AI-optimized compute elements, each with a fully programmable Core and a set of ML-optimized extensions. The memory architecture is optimized for deep learning, with a high-performance on-chip memory that allows for quick access and efficient data flow. The communication between cores is directly connected on the wafer, enabling high-bandwidth, low-latency cluster-scale networking.

Cerebrus Server: Physical Chassis and Integration

To house and power the Wafer Scale Engine, Cerebrus developed the Cerebrus CS2, a server that fits into a standard data center rack. The CS2 server features standard power connections, 100-gigabit Ethernet links, and an aggregate system-level I/O of 1.2 terabits per Second. With the CS2 server, Cerebrus offers a compact and high-performance solution for deploying the Wafer Scale Engine.

Software Stack: Compiler and SDK

Cerebrus recognizes that a powerful hardware platform is only as useful as the software that allows users to program it. Therefore, they have invested significantly in developing a comprehensive software platform. The Cerebrus software stack includes a compiler that translates the user's compute graph into an intermediate representation for execution on the Wafer Scale Engine. They also offer a software development kit (SDK) for lower-level programming and customization.

Applications of Cerebrus in Commercial Space

Cerebrus has collaborated with companies in the commercial space to accelerate their deep learning workloads. One notable example is GlaxoSmithKline, which uses Cerebrus' machine to train large language models for applications like biomedical text mining and graph-Based neural networks. The speed and performance of the Wafer Scale Engine have enabled faster training times and improved accuracy for commercial applications.

Applications of Cerebrus in Research Labs

Cerebrus machines have also found applications in research labs, particularly in the fields of science and supercomputing. For instance, Cerebrus collaborates with the Argonne National Laboratory to accelerate cancer drug response prediction, X-ray data processing, and classification of gravitational waves. The high-performance compute capabilities of the Wafer Scale Engine have been instrumental in advancing research in these domains.

Applications of Cerebrus in HPC

Cerebrus has observed a growing interest in using Novel computing architectures for high-performance computing (HPC) applications. The Wafer Scale Engine's ability to handle sparse linear algebra computations makes it suitable for various HPC tasks. For instance, Cerebrus collaborated with the National Energy Technology Laboratory to accelerate simulations for energy-efficient combustion. They also integrated their machine with Lawrence Livermore National Laboratory's Lassen supercomputer for cognitive simulations.

What's Next: New Execution Mode

Cerebrus is continuously working on advancing their technology. They are set to unveil a new execution mode in their software stack that will handle extremely large inputs and models. This new mode will enable researchers to train billion to trillion-parameter scale models, leverage sparsity techniques for greater acceleration, and cluster multiple machines together for higher throughput. The future holds exciting possibilities for Cerebrus and their customers.

Highlights

• Cerebrus is an AI computer systems company with a revolutionary solution for deep learning and AI workloads.
• The Wafer Scale Engine offers extreme-scale computation and flexible compute capabilities.
• Cerebrus collaborates with commercial companies and research labs to accelerate deep learning applications.
• The Wafer Scale Engine is also suitable for HPC tasks, with applications in energy-efficient combustion and cognitive simulations.
• Cerebrus is constantly innovating, with a new execution mode in development to handle larger inputs and models.

FAQ

Q: Where is Cerebrus based? A: Cerebrus is based in Silicon Valley, with offices in San Diego, Toronto, and Japan.

Q: What is the Wafer Scale Engine? A: The Wafer Scale Engine is a revolutionary processor architecture developed by Cerebrus for deep learning and AI.

Q: How does the Cerebrus software stack work? A: The Cerebrus software stack includes a compiler that translates compute graphs into an intermediate representation for execution on the Wafer Scale Engine. It also offers an SDK for lower-level programming and customization.

Q: What are the applications of Cerebrus in commercial space? A: Cerebrus collaborates with companies to accelerate deep learning workloads in applications like biomedical text mining and drug discovery.

Q: Can the Wafer Scale Engine be used for high-performance computing (HPC)? A: Yes, the Wafer Scale Engine is suitable for HPC tasks and can be used in simulations for energy-efficient combustion and cognitive simulations.

Q: What is next for Cerebrus? A: Cerebrus is developing a new execution mode that will handle extremely large inputs and models, enabling researchers to train trillion-parameter scale models and cluster multiple machines together for higher throughput.