Boost PyTorch Transformers Performance with Intel Sapphire Rapids

Table of Contents

1. Introduction
2. Benchmarking NLP Models
3. Test Servers
4. Code Setup and Installation
5. Ice Lake Instance Benchmark Results
6. Sapphire Rapids Instance Benchmark Results
7. Optimum Intel Library and Pipeline
8. AMX and BF16 Support
9. Running the Benchmark
10. Speed Up and Latency Reduction
11. Conclusion

Introduction

In this article, we will explore how to run distributed training on the latest generation of Intel Xeon CPUs for inference on NLP models. We will compare the performance of the Ice Lake architecture with the new Sapphire Rapids architecture and measure the speedup achieved using the Optimum Intel library. Get ready to dive into the world of CPU-based inference and witness some serious performance boosts!

Benchmarking NLP Models

Before diving into the benchmark results, let's take a moment to understand the NLP models we will be working with. We will be using the Hugging Face Transformers library, which provides a wide range of pre-trained models for natural language processing tasks. Specifically, we will focus on the Auntie Stillberg, Bird Base, and Roberto Base models for sentiment analysis.

Test Servers

To conduct our benchmark tests, we will be using two different test servers. The first server is based on the Ice Lake architecture, offering a baseline benchmark for comparison. The Second server runs on the Sapphire Rapids architecture, which is a newer generation CPU. Both servers have the same specs, with a size of 16 XL.

Code Setup and Installation

Setting up the code and installing the necessary libraries is a relatively straightforward process. We will be installing the Torch library, along with the Intel extension for PyTorch. This extension brings hardware acceleration capabilities to different chips. Additionally, we will need the Transformers library for working with the NLP models. For the Sapphire Rapids instance, we will also install the Optimum Intel library, which further enhances performance.

Ice Lake Instance Benchmark Results

Once the setup is complete, we can run the benchmark tests on the Ice Lake instance. The benchmarking process involves creating a sentiment analysis pipeline using the chosen NLP models. We will run predictions on both short and long customer reviews to measure the model's performance. The benchmarking results will provide us with mean and percentile values, giving us a comprehensive view of the model's inference capabilities.

Sapphire Rapids Instance Benchmark Results

Next, we will replicate the same benchmark tests on the Sapphire Rapids instance. However, this time, we will run the tests twice. First, using the vanilla Transformers pipeline, and then utilizing the Optimum Intel pipeline. The Optimum Intel pipeline leverages the hardware acceleration features of the Sapphire Rapids architecture, such as AMX and BF16 support. This allows us to achieve greater speed and performance.

Optimum Intel Library and Pipeline

The Optimum Intel library plays a crucial role in enhancing the performance of our NLP models. By creating an Optimum Intel pipeline and enabling BF16 data type, we can take advantage of the advanced Matrix Extensions (AMX). AMX brings new hardware registers known as tile registers, which greatly improve matrix operations. The BF16 data type offers similar range coverage as FP32 but with faster performance.

AMX and BF16 Support

AMX and BF16 support are key features of the Sapphire Rapids architecture. AMX introduces advanced matrix multiplication and accumulation operations, available for INT8 and BF16 data types. BF16 is particularly useful as it overcomes overflow issues faced with FP16, while still delivering fast performance. By checking the CPU flags, we can ensure that our instance supports these features.

Running the Benchmark

Running the benchmark tests is a straightforward process. We warm up the models for 100 iterations and then predict a thousand times to Gather the prediction times. The mean and 99th percentile values are then calculated and analyzed. This approach allows us to evaluate the performance of the NLP models with different sequences, both short and long.

Speed Up and Latency Reduction

After running the benchmark tests on both instances, we analyze the results. Switching from Ice Lake to Sapphire Rapids reveals noticeable speed improvements, with a 20-30% improvement in the performance of larger models. However, the real breakthrough comes when running the Optimum Intel pipeline with BF16. This combination provides significantly reduced latencies, even achieving single-digit milliseconds for distilled Bird and staying within a 10-millisecond range for longer sequences. Overall, we observe a 3X speedup, making CPU-based inference a viable option for achieving low-latency predictions.

Conclusion

In conclusion, this article has demonstrated the performance improvements achieved by using the latest generation of Intel Xeon CPUs for inference on NLP models. By benchmarking the Ice Lake and Sapphire Rapids instances, we witnessed significant speedups, especially when leveraging the Optimum Intel library and BF16 support. CPU-based inference now offers comparable performance to GPU-based inference, allowing for cost-effective and easily manageable solutions. With NLP models becoming increasingly complex, the advancements in CPU technology provide a promising future for inference tasks.

Highlights

• Explore the performance of Intel Xeon CPUs in NLP model inference
• Benchmark Ice Lake and Sapphire Rapids architectures
• Utilize the Optimum Intel library for enhanced performance
• Leverage AMX and BF16 support for greater speed
• Achieve low-latency predictions with CPU-based inference

FAQs

Q: Which CPUs are used for benchmarking? A: The benchmark tests were conducted on the Ice Lake and Sapphire Rapids architectures.

Q: How does the Optimum Intel library enhance performance? A: The Optimum Intel library utilizes hardware acceleration features, such as AMX and BF16 support, to significantly improve model performance.

Q: What benefits does BF16 offer over FP16? A: BF16 offers similar range coverage as FP32 but with faster performance. It overcomes overflow issues typically faced with FP16.

Q: Can CPU-based inference achieve low-latency predictions? A: Yes, utilizing the Optimum Intel library and BF16 support can result in single-digit millisecond predictions for certain models and stay within a 10-millisecond range for longer sequences.

Q: Are GPU servers still necessary for inference tasks? A: With the advancements in CPU technology, CPU-based inference offers comparable performance to GPU-based inference, making GPU servers less essential for certain use cases.