Accelerate Your PyTorch Training with Mosaic ML Composer

Table of Contents

1. Introduction to Mosaic ML Composer
2. The Training API
3. Training Optimizations
4. Streaming Data Loading
5. Achieving Faster Training Speeds with Composer
6. Concrete Results with Composer Optimizations
7. Learn More About Mosaic ML Composer

Introduction to Mosaic ML Composer

Mosaic ML Composer is a Python library built on top of PyTorch that enables efficient and accelerated training of neural networks. In this article, we will explore the various features and functionalities of Composer and how it can help supercharge your PyTorch training.

But first, let's take a moment to appreciate the significant impact that AI has had on our lives. Thanks to the AI community and powerful tools like PyTorch, AI is now ubiquitous, powering everything from personalized recommendations to autonomous vehicles and even AI-powered drug discovery.

As AI continues to support a growing number of use cases, there is a need for models to become more intelligent. This growth in intelligence is achieved through an increase in both the complexity of the models and the Scale of the training data. The exponential growth in model sizes over the past few years is a clear indication of this trend.

The Training API

Composer simplifies the training process by encapsulating PyTorch's training loop and providing support for multi-GPU and multi-node training. With Composer, you can customize your training using callbacks or hooks that are called at different stages of the training process.

To demonstrate the simplicity of using the Composer trainer API, let's consider the following example. To train a model using Composer, you first need to create a model class that implements the Composer model interface. This involves implementing three functions: init, forward, and loss. Once you have defined your model, you can initialize the Composer trainer object with the model, data loaders, optimizer, number of epochs, and the device to run on. Then, simply call the fit function to let Composer handle the training loop.

By leveraging Composer's trainer API, you can sit back and let Composer and your GPUs do the heavy lifting. You can easily track metrics during training using tools like Weights & Biases or Comet ML. Once training is complete, you can access the torch metrics in the trainer state object.

Training Optimizations

Composer offers more than 20 built-in optimizations that can significantly speed up your training process. These optimizations, which are fairly complex algorithmically, can be easily applied and even composed together to achieve improved performance.

For example, let's consider the following code snippet that demonstrates the application of some Composer optimizations. By instantiating the desired optimizations, such as Progressive Resize Blur Pull and Label Smoothing, and passing them to the Trainer object, you can benefit from significant speed ups, especially with models like ResNet. Composer provides dedicated documentation for each optimization, including research Papers and insights into how the algorithms work.

Streaming Data Loading

Training large-scale language models often involves the painstaking and time-consuming process of downloading and managing data on local storage. However, Composer's streaming data loading feature enables you to stream training data directly from the cloud, eliminating the need for local data storage.

To utilize this feature, you need to convert your dataset into a supported format that allows the streaming engine to index and stream the data. The MDS Writer class can be used for this purpose. Once the dataset is uploaded to your preferred cloud storage (such as AWS S3), you can instantiate the standard Python data loader, providing an instance of the streaming dataset. The trainer can then be instantiated, and the training process can be initiated with a simple call to the fit function. Composer takes care of streaming the data from the cloud while you focus on model training.

Achieving Faster Training Speeds with Composer

With Composer's built-in optimizations, you can achieve significant speed improvements in your training process. The optimizations encapsulated by Composer, such as Progressive Resize Blur Pull and Label Smoothing, can be easily applied without needing to worry about the underlying complexity. By leveraging these optimizations, you can ensure that your models train faster and more efficiently.

Consider the results achieved with Composer when training a ResNet-50 model on eight A100 GPUs. The use of Composer optimizations delivered a 4.4x speedup in training speed compared to the baseline. Similarly, when applying Composer optimizations to large-scale NLP training, a 2.7x speedup was achieved on the same GPU configuration. These results, which were submitted to MLPerf, outperformed other submissions and even surpassed Nvidia's highly optimized training.

Concrete Results with Composer Optimizations

The impressive results achieved with Composer's optimizations highlight the potential for rapid iterations and exploration of new models and ideas. By using Composer, machine learning developers can easily take advantage of state-of-the-art optimizations without the complexity of implementation. This allows for faster and more efficient training of models across various domains.

Learn More About Mosaic ML Composer

If you're interested in learning more about Mosaic ML Composer and how it can enhance your PyTorch training, make sure to visit the Composer GitHub repository and explore the documentation. Join the Composer community to connect with other machine learning developers, share your experiences, and provide feedback. The Mosaic ML team is dedicated to helping users adopt Composer and reap the benefits of efficient and accelerated training.

Thank you for tuning in! We hope you found this introduction to Mosaic ML Composer informative and inspiring. Stay tuned for more exciting developments in the field of machine learning and AI!

Highlights

• Mosaic ML Composer: Supercharge your PyTorch training with efficient and accelerated neural network training.
• Composer's Trainer API: Simplified training process with support for multi-GPU and multi-node training.
• Training Optimizations: Over 20 built-in optimizations to improve training speed and performance.
• Streaming Data Loading: Stream training data directly from the cloud, eliminating the need for local storage.
• Achieving Faster Training Speeds: Concrete results show significant speed improvements with Composer.
• Learn More: Visit the Composer GitHub repository, explore the documentation, and join the Composer community.

FAQ

Q: Can Composer be used with other deep learning frameworks besides PyTorch? A: No, Composer is specifically designed to work on top of PyTorch and utilizes its functionalities.

Q: Are the Composer optimizations applicable to all types of models? A: While some optimizations may have a more significant impact on specific models, most optimizations can be applied to various types of models.

Q: Can I use Composer for inference or just training? A: Composer focuses primarily on the training aspect of machine learning and neural networks. For inference, PyTorch or other frameworks can be used.