Accelerate TensorFlow on Intel CPUs

Table of Contents

1. Introduction
2. Overview of TensorFlow
3. Optimization of TensorFlow for Intel CPUs
4. Maximizing Performance in Existing TensorFlow Implementations
5. Summary and Call to Action
6. Deep Learning and AI at Intel
7. TensorFlow as a Mathematical Framework
8. The Core of TensorFlow: Computational Graphs
9. Intel's Optimization of TensorFlow
10. Tools for Performance Optimization
11. Distributed Multi-node Training in TensorFlow
12. Performance Scaling Studies
13. Tuning Parameters when Transitioning from GPUs to Intel CPUs
14. Compatibility Issues with Intel and Cray Runtimes
15. Performance Differences between MKL and LibXSMM
16. Comparing Performance with Intel Compiler and Goon Compiler
17. Conclusion

Introduction

In this article, we will discuss the optimization of TensorFlow for Intel CPUs, as well as how to maximize performance in existing TensorFlow implementations. We will explore the benefits of using TensorFlow for deep learning and AI, and how Intel has worked with Google to optimize the framework. We will also provide guidance on tuning parameters when transitioning from GPUs to Intel CPUs and address compatibility issues with Intel and Cray runtimes. Additionally, we will compare performance differences between MKL and LibXSMM, as well as the Intel and Goon compilers.

Overview of TensorFlow

TensorFlow is a widely used framework for deep learning and AI, both within Google and among external users. It is a versatile mathematical framework designed for deep neural networks, but can also be utilized for other machine learning applications. TensorFlow utilizes a computational graph, where nodes represent different computations and edges represent data flow. The framework can be easily extended to add additional user operations. TensorFlow's core code system is written in C++ with a Python front-end wrapper. It also supports multi-node support using the open source G RPC protocol.

Optimization of TensorFlow for Intel CPUs

Intel has worked closely with Google to optimize TensorFlow for Intel CPUs. This optimization includes modifications to the Intel MKL libraries to support deep learning, as well as adjustments to TensorFlow itself for improved performance. By using the Intel optimized version of TensorFlow, significant improvements in deep learning workloads have been observed, with speedups of up to 70-80% in certain cases. Changes have been made to the layout of data to ensure maximum performance and reduce memory bottlenecks. Intel has also contributed to the optimization of other major machine learning frameworks such as Cafe, MXNet, and CNTK.

Maximizing Performance in Existing TensorFlow Implementations

While the Intel optimized version of TensorFlow provides significant performance improvements, developers can further enhance performance by paying attention to certain areas. One significant factor is the layout of the data, with the NCHW format often providing the best performance for image-related benchmarks. Thread settings also play a crucial role, with the need to set the appropriate values for interrupt parallelism and intra-op parallelism based on the number of cores and the workload. Additionally, experimenting with batch size and adjusting OpenMP settings can further improve performance. Tools like Intel VTune and TensorBoard's timeline feature can be utilized for profiling and performance analysis.

Summary and Call to Action

In summary, TensorFlow is a powerful framework for deep learning and AI, and Intel has made significant efforts to optimize it for Intel CPUs. By using the Intel optimized version of TensorFlow, developers can achieve substantial performance improvements. However, further tuning and adjustments to settings may be required to maximize performance for specific workloads. Developers are encouraged to experiment with different parameters and utilize performance profiling tools to identify areas for improvement. By following these guidelines, optimal performance can be achieved in TensorFlow implementations on Intel CPUs.

Deep Learning and AI at Intel

Intel recognizes the growing importance of deep learning and AI and has made it a key initiative. In addition to building platforms specifically tuned for AI workloads, Intel has also focused on optimizing libraries and collaborating with open source communities to enhance machine learning frameworks. The acquisition of Nirvana, a company specializing in deep learning hardware and software, has further augmented Intel's capabilities in this domain. Through these efforts, Intel aims to provide developers with the tools and technologies they need to accelerate AI innovation.

TensorFlow as a Mathematical Framework

TensorFlow is a versatile mathematical framework that serves as the foundation for deep neural networks. Its generic design allows it to be used for a wide range of applications beyond deep learning. At its core, TensorFlow consists of a set of kernels that can be easily extended to support additional user operations. The framework's code system is primarily written in C++, with a Python front-end wrapper that makes it accessible and easy to use.

The Core of TensorFlow: Computational Graphs

At the core of TensorFlow is a computational graph, where nodes represent different computations and edges represent the flow of data. TensorFlow's computational graph allows for efficient execution and optimization of operations. The essence of TensorFlow can be boiled down to executing and optimizing this computational graph. By representing computations and data flow as a graph, TensorFlow enables developers to easily define and manipulate complex models for deep learning and AI.

Intel's Optimization of TensorFlow

Intel has collaborated with Google to optimize TensorFlow for Intel CPUs. This optimization effort involved enhancing the Intel MKL libraries to include deep learning support, as well as making changes within TensorFlow itself to improve performance on Intel CPUs. Intel CPUs feature advanced vectorization capabilities such as AVX2 and AVX-512, and TensorFlow has been optimized to make full use of these instructions. Additionally, Intel has worked on memory allocation issues within TensorFlow to reduce bottlenecks and improve overall performance.

Tools for Performance Optimization

Intel provides a range of tools that can aid in the optimization of TensorFlow for Intel CPUs. The Intel VTune Performance Analyzer is a powerful profiling tool that allows developers to analyze the performance of their TensorFlow implementations. It provides insights into CPU usage, memory access Patterns, and other performance metrics. Additionally, TensorFlow's timeline feature can be enabled to Visualize the execution of operations and identify potential performance bottlenecks. These tools are invaluable for fine-tuning TensorFlow implementations and achieving optimal performance on Intel CPUs.

Distributed Multi-node Training in TensorFlow

TensorFlow supports distributed training across multiple nodes, allowing for improved performance and scalability. The most common approach for distributed training is data parallelism, where multiple workers run the full model with their own partitioned data. These workers communicate with one or more parameter servers responsible for applying gradients and distributing weights. When using multiple parameter servers, it is recommended to have one parameter server for every four workers for optimal performance. Scaling studies have shown that TensorFlow's distributed training can achieve significant performance gains even with the G RPC protocol over TCP/IP.

Performance Scaling Studies

Intel has conducted extensive scaling studies to evaluate the performance of TensorFlow in a distributed multi-node environment. These studies have shown that TensorFlow's performance scales well with increasing numbers of workers. For example, using the Inception V3 benchmark, performance reached 95-96% with four workers compared to a single worker. Further scaling to eight or sixteen workers still yielded good performance, achieving around 90% of the maximum performance. These results demonstrate the ability of TensorFlow to efficiently utilize resources in a distributed training setting.

Tuning Parameters when Transitioning from GPUs to Intel CPUs

When transitioning from GPUs to Intel CPUs, it is essential to pay attention to certain tuning parameters to ensure optimal performance. One key factor is the data layout, with the NCHW format often delivering the best performance for image-related benchmarks. Additionally, thread settings play a crucial role, and developers should experiment with interrupt parallelism and intra-operator parallelism to find the optimal settings for their specific workloads. Batch size can also be adjusted to improve performance, with certain workloads benefiting from larger batch sizes. By fine-tuning these parameters, developers can maximize performance when transitioning from GPUs to Intel CPUs.

Compatibility Issues with Intel and Cray Runtimes

Some compatibility issues may arise when using Intel and Cray runtimes together in a mixed environment. For example, the Intel OpenMP runtime and Cray runtime's affinities can conflict with each other, leading to issues in performance and resource utilization. To address these issues, developers may need to either disable one of the runtimes' affinities or rely solely on the Cray affinity settings. It is important to carefully configure the runtime environments to avoid conflicts and achieve optimal performance.

Performance Differences between MKL and LibXSMM

There may be performance differences between the Intel Math Kernel Library (MKL) and the LibXSMM library. While MKL has been highly optimized for Intel CPUs and offers excellent performance, some users have reported seeing performance improvements when using LibXSMM instead. Further investigation is required to determine the exact performance differences between the two libraries and identify the specific use cases where each library excels. Developers are encouraged to experiment with both libraries and evaluate their performance in their particular workloads.

Comparing Performance with Intel Compiler and Goon Compiler

When compiling TensorFlow, developers have the option to choose between the Intel Compiler (ICC) and the Goon Compiler (GCC). Performance differences may exist between the two compilers, and it is worth exploring the specific settings and optimizations available with each compiler. Users have reported seeing performance benefits when using ICC with certain models, but further testing and parameter tuning are necessary to fully understand the performance implications. Developers can experiment with different compiler options and settings to determine the best compiler for their TensorFlow implementation.

Conclusion

In conclusion, optimizing TensorFlow for Intel CPUs can significantly improve performance and scalability in deep learning and AI applications. Intel has collaborated with Google to enhance TensorFlow's performance on Intel CPUs, and developers can readily access the Intel optimized version of TensorFlow. By tuning parameters, utilizing performance profiling tools, and leveraging Intel's capabilities, developers can achieve optimal performance when implementing TensorFlow on Intel CPUs. Additionally, addressing compatibility issues and exploring alternative libraries and compilers can further enhance performance. With Intel's commitment to advancing AI and deep learning technologies, TensorFlow users can expect continued optimizations and improvements for Intel CPUs.

(Resources: Intel Deep Learning, TensorFlow)