Supercharge ChatGPT Training with DeepSpeed-Chat

Table of Contents

1. Overview
2. Easy to Use Training and Inference Experience
3. Customizing your own RLHF Training Pipeline using DeepSpeed Chat's RLHF APIs
4. DeepSpeed Hybrid Engine: Unified Infrastructure to Power and Optimize RLHF Training
5. Effective Throughput Analysis
6. Scalability Analysis
7. Conclusion
8. FAQ

Overview

We're excited about the impact that AI models similar to chat GPT have had on the digital world. These models are incredibly versatile, able to perform tasks like summarizing, coding, and translating with results that often match or even surpass human experts. There are many ongoing efforts in the open-source AI community to make these chat GPT-style models more accessible, such as Chat Llama, Alpaca, Bikuna, Databricks Dali, and others. However, there's still a need for a complete RLHF (Reinforcement Learning from Human Feedback) pipeline that can train powerful chat GPT-like models and is easily accessible to the AI community.

Easy to Use Training and Inference Experience

In this section, we'll discuss how to use the DeepSpeed RLHF (Reinforcement Learning from Human Feedback) system to train opt-13b and opt-66b models, making the process of training these models straightforward and user-friendly. With DeepSpeed Chat, training your first chat GPT-style model is a breeze. A single script takes care of all three stages of RLHF training and generates your first chat GPT model.

We'll use a pre-trained opt-13b model as the actor model and opt-350m as the reward model in a single script to Create a final 13B chat GPT-style model. The best part is that your model will be ready to use in just half a day. Once your model is trained, you can test it using DeepSpeed Chat's inference API, allowing you to Interact with your model in a conversational manner, asking questions and receiving responses.

If You want to experiment with different model sizes and configurations, DeepSpeed Chat makes it easy. Whether you're trying to meet specific training time, resource, or quality requirements, you can adjust the model size and GPU count to suit your needs. For instance, if you want to train a larger, higher-quality model on your GPU cluster, you can use the same script with your desired model size, such as 66b, and GPU count, such as 64 GPUs. In just nine hours, you'll have a 66 billion parameter chat GPT model ready to use. And if you only have a short amount of time, you can train a smaller model with DeepSpeed Chat. We've prepared a training example for a 1.3B model that you can test on your consumer-grade GPUs. The best part is that your model will be ready to use by the time you're back from your lunch break.

Customizing your own RLHF Training Pipeline using DeepSpeed Chat's RLHF APIs

With DeepSpeed Chat, you can customize your own Reinforcement Learning from Human Feedback (RLHF) training pipeline using the APIs provided. This tool allows you to create your own RLHF training strategy, which can be used to develop a variety of RLHF algorithms for research purposes.

To customize your RLHF training pipeline, you first create an engine using the DeepSpeed RLHF engine. This engine takes several parameters, including the actor and critic model paths, the tokenizer, the total number of iterations, and other arguments. Next, you create a trainer using the DeepSpeed PPO (Proximal Policy Optimization) trainer, which takes the engine and arguments as parameters. Then, for each batch of Prompts in your training data loader, you generate an experience using the trainer and train the RLHF using the output.

The full RLHF training pipeline in DeepSpeed Chat is designed to provide a smooth training experience. It is Based on the Instruct GPT model and includes three main steps: Supervised fine-tuning (SFT), reward model fine-tuning, and RLHF training using the Proximal Policy Optimization (PPO) algorithm. Additional features, such as exponential moving average (EMA) collection and mixture training, are provided to improve model quality.

DeepSpeed Chat also offers features to support researchers and practitioners in training their own RLHF models with multiple data resources. These include data abstraction and blending capabilities, which allow the model to be trained with multiple datasets for better quality.

DeepSpeed Hybrid Engine: Unified Infrastructure to Power and Optimize RLHF Training

DeepSpeed Chat introduces the DeepSpeed Hybrid Engine, a unified infrastructure designed to power and optimize Reinforcement Learning from Human Feedback (RLHF) training. The first two steps of the RLHF pipeline are similar to the standard fine-tuning of large models. These steps are enhanced by zero-based optimizations and a flexible mix of parallelism strategies in DeepSpeed Training, which helps increase Scale and speed.

The third step of the pipeline is the most challenging in terms of performance. It involves two phases: the inference phase, which generates tokens or experiences and provides inputs for training, and the training phase, which updates the weights of the actor and reward models. This step also involves managing the interaction and scheduling between these models.

Two main costs are associated with this step. First, there's the memory cost, as multiple copies of the SFT and RW (Reward) models need to be maintained throughout this stage. Second, there's the cost of the generation phase, which, if not properly accelerated, can significantly slow down the entire stage. Additionally, two features we added in Stage 3, exponential moving average (EMA) collection and mixture training, can increase memory and training costs.

To address these challenges, we've combined the full system capabilities of DeepSpeed Training and Inference into the Hybrid Engine. This engine uses the original DeepSpeed engines for fast training mode and applies the DeepSpeed inference engine for generation evaluation mode. This results in a significantly faster training system for RLHF training at Stage 3.

The transition between DeepSpeed Training and Inference engines is seamless. The engine can switch between evaluation and training modes for the actor model. When running the inference and training pipeline, DeepSpeed selects different optimizations to run the model faster and improve the overall system throughput.

During the experience generation phase of RLHF training, the Hybrid Engine uses a lightweight memory management system to handle the KV cache and intermediate results. It also uses highly optimized inference-adapted kernels and tensor parallelism implementation to significantly increase throughput (tokens per second) compared to existing solutions.

During the training execution, the Hybrid Engine uses memory optimization techniques, such as DeepSpeed Zero, to optimize memory allocation and usage. These system optimizations are compatible with each other and can be combined to deliver the highest training efficiency under the Hybrid Engine.

The Hybrid Engine can seamlessly change model partitioning across training and inference to support tensor parallelism-based inferencing and zero-based sharding mechanism for training. It can also reconfigure the memory system to maximize memory availability during each of these modes. This allows for improved performance by avoiding memory allocation bottlenecks and supporting large batch sizes.

Effective Throughput Analysis

To ensure optimal efficiency, DeepSpeed Heat focuses on achieving high throughput during both the generation and RL training phases of the RLHF training process. The generation phase, although comprising a small proportion of the total computation, can take up a large portion of the end-to-end time due to the need to run the actor model multiple times. DeepSpeed Heat optimizes both phases by using the largest possible batch size to increase efficiency.

During the generation phase, DeepSpeed Heat uses high-performance Transformer kernels to maximize GPU memory bandwidth utilization when the model fits in a single GPU memory. It also leverages tensor parallelism (TP) when the model doesn't fit in a single GPU memory, reducing inter-GPU communication and maintaining high GPU memory bandwidth utilization. These optimizations allow DeepSpeed Heat to achieve up to a nine times throughput improvement during the generation phase compared to existing solutions.

The majority of the time in an RLHF training iteration for a 1.3 billion parameter model is spent on the generation phase. By using high-performance inference kernels from DeepSpeed, DeepSpeed Heat can achieve unparalleled throughput improvement, up to 19 times, during this phase compared to Hugging Face and Colossal AI.

Scalability Analysis

DeepSpeed Heat demonstrates excellent scalability, with the best effective throughput varying depending on the model size and the number of GPUs used. Our data shows that DeepSpeed RLHF scales well with up to 64 GPUs. However, at larger scales, the scalability becomes almost linear or less than linear due to the interplay between available memory and the maximum global batch size.

DeepSpeed RLHF's scalability properties are a result of its ability to reduce memory consumption per GPU, allowing it to support larger batches per GPU and achieve more than proportional scaling at smaller scales. However, as the number of GPUs increases, the maximum global batch size and the batch size per GPU become limiting factors. As a result, the scalability becomes almost linear or less than linear at larger scales.

While DeepSpeed Heat may not achieve super-linear scaling at larger scales, it still performs significantly better than existing systems. DeepSpeed Heat's efficiency is 19 times higher than existing systems, highlighting the effectiveness of our system in optimizing RLHF workloads.

Conclusion

DeepSpeed Chat is a powerful system that empowers data scientists and researchers to train powerful chat GPT-like models quickly, affordably, and with excellent scalability. By addressing the limitations of existing systems and optimizing RLHF training pipelines, DeepSpeed Chat offers an easy-to-use training and inference experience, customization options for RLHF training strategies, a unified infrastructure for RLHF training optimization, and efficient throughput and scalability.

With DeepSpeed Chat, data scientists can train models with over 13 billion parameters on a single GPU, achieve fast training times, and generate high-quality chat GPT-like models that can be used in real-world scenarios. DeepSpeed Chat has the potential to democratize RLHF training and make it accessible to the AI community. We invite you to explore DeepSpeed Chat and contribute to its development.

FAQ

Q1: What is DeepSpeed Chat?

DeepSpeed Chat is a system that enables easy and efficient training of chat GPT-like models using Reinforcement Learning from Human Feedback (RLHF) techniques. It provides an intuitive training and inference experience, customizable RLHF training pipelines, and a unified infrastructure for optimizing RLHF training.

Q2: How does DeepSpeed Chat optimize RLHF training?

DeepSpeed Chat optimizes RLHF training by combining the training and inference capabilities of DeepSpeed into a unified Hybrid engine. This engine seamlessly transitions between training and inference modes, leverages optimized memory management, and utilizes high-performance inference kernels and tensor parallelism to maximize throughput and efficiency.

Q3: Can DeepSpeed Chat scale to large models and GPU clusters?

Yes, DeepSpeed Chat is designed to scale to large models and GPU clusters. It supports models with over 13 billion parameters on a single GPU and achieves excellent scalability on multi-node, multi-GPU systems. With DeepSpeed Chat, data scientists can train models of different sizes and configurations to meet their specific needs.

Q4: How does DeepSpeed Chat compare to existing RLHF systems?

DeepSpeed Chat outperforms existing RLHF systems in terms of throughput, efficiency, and scalability. It provides over a 10 times improvement in throughput on a single GPU and enables the training of significantly larger models compared to other systems. DeepSpeed Chat is a cost-effective solution that democratizes RLHF training and makes it accessible to the AI community.

Q5: Is DeepSpeed Chat open source?

Yes, DeepSpeed Chat is open source and available for the AI community to use and contribute to. You can find the code and documentation on the DeepSpeed GitHub page. We welcome your feedback, contributions, and collaborations to further improve DeepSpeed Chat.

Q6: What are the advantages of using DeepSpeed Chat for RLHF training?

DeepSpeed Chat offers several advantages for RLHF training. It provides an easy-to-use training and inference experience, customization options for RLHF training pipelines, efficient throughput and scalability, and a unified infrastructure for optimizing RLHF training. DeepSpeed Chat enables data scientists to train powerful chat GPT-like models quickly, affordably, and with excellent scalability.

Q7: Can DeepSpeed Chat be used for other types of language models?

Yes, DeepSpeed Chat can be used for other types of language models. While it is specifically designed for chat GPT-like models, the customizable RLHF training pipeline and efficient infrastructure of DeepSpeed Chat can be adapted for training other types of language models that use RLHF techniques.

Q8: How can I get started with DeepSpeed Chat?

To get started with DeepSpeed Chat, visit the DeepSpeed GitHub page and follow the instructions for installation and usage. The GitHub page contains tutorials, examples, and documentation to help you explore and use DeepSpeed Chat effectively. Join the DeepSpeed community and contribute to the development of this powerful system.