Mastering Cart-Pole with Deep Reinforcement Learning

Table of Contents:

1. Introduction
2. Problem Statement
3. The Cart Pole Problem
4. Initial Setup and Challenges
5. Sensors and Controls Used
6. Training Models - Q Network and Deep Q Network
7. Results and Analysis
8. Identified Issues and Solutions
9. Conclusion
10. Future Improvements

Introduction

Welcome to our exploration of implementing reinforcement learning on the cart pole problem. In this article, we will Delve into the details of this problem, discuss the challenges faced during the initial setup, explore the sensors and controls used, examine the training models - Q Network and Deep Q Network, analyze the results obtained, and address the issues identified during the project. We will conclude with some potential future improvements to make the model more successful.

Problem Statement

The goal of the cart pole problem is to balance a pole on top of a cart in a reverse pendulum manner. The cart has the ability to move left or right to maintain the balance. Our objective is to train a machine learning model that can take appropriate actions to keep the pole balanced. The environment considers an episode as a failure if the pole falls, the cart moves too far from the center, or a certain time limit is reached while the pole remains balanced.

The Cart Pole Problem

In the cart pole problem, we aim to find a solution to maintain the balance of a pole on top of a cart. The longer the pole remains balanced, the greater the reward we use to train our reinforcement learning model. Our initial setup involved a pole placed in the middle of the cart, but due to wiring issues, it did not fall over as intended. To overcome this, we modified the setup to treat the top part of the cart as the pole, allowing for more accurate balancing.

Initial Setup and Challenges

The initial setup posed challenges due to wiring limitations that prevented the pole from falling over. We pivoted the design to have the cart balance on its back two wheels instead. To track the cart's position, we used an HC-SR04 ultrasonic sensor, which measured the distance from the edge of the environment. The environment was printed using polylactic acid (PLA) and powered by an L298N motor driver controlled by an Arduino R3 Uno microcontroller. However, some design flaws, such as improper cable management, affected the cart's balance and performance.

Sensors and Controls Used

To accurately monitor and control the cart's movements, we utilized an MPU 6050 3-axis accelerometer and gyroscope mounted on the cart. This allowed us to track the cart's angle and angular velocity. The various sensors and controls were connected to an Arduino microcontroller, which communicated with a laptop running a Python script.

Training Models - Q Network and Deep Q Network

We attempted to train two different models: the Q Network and the Deep Q Network. In the Q Network, we defined the actions as an array of two options: left or right. We implemented 18 different bucket combinations Based on the states of angular velocity and pole angle. However, this adaptation did not yield better results compared to the Deep Q Network. The Deep Q Network had a four-layer architecture with two Hidden layers to increase its potential for success. The action space was expanded to include specific speeds and directions for better control.

Results and Analysis

The Deep Q Network outperformed the Q Network, although some issues were identified during training that affected its performance. The overall loss converged after approximately 600 episodes, indicating a shift from exploration to exploitation. The reward graph demonstrated a Sense of balance trend at around 600 to 700 episodes. However, after 2,500 episodes, the average episode duration plateaued at 10 cycles, suggesting limited progress. Unfortunately, an accidental data overwrite prevented the analysis of the Q Network's results.

Identified Issues and Solutions

Several issues were identified that contributed to the lack of success in improving the balance of the cart. Inconsistent sensor readings, caused by the accelerometer's fluctuations when the cart was stationary, affected the accuracy of the model. Battery voltage drop also impacted the motor speed and balance. Additionally, wiring falling into the cart's path resulted in unexpected spikes in the data. To address these issues, improvements such as better cable management, more consistent training environments, and code adjustments are recommended.

Conclusion

While the project did not achieve the desired level of success, the Deep Q Network showed promise compared to the Q Network. With modifications to the cart design, improved cable management, a more consistent training environment, and further code adjustments, this project has the potential to succeed in balancing the cart pole problem using reinforcement learning.

Future Improvements

To enhance the model's performance, we suggest focusing on improving cable management to prevent interference during movements. Additionally, providing a more consistent training environment and incorporating minor alterations to the code base should yield better results. Combining these changes with extended training time will likely lead to increased success in balancing the cart pole problem.

Highlights:

• The cart pole problem aims to balance a pole on a cart using reinforcement learning.
• Initial setup challenges led to modifications, treating the top part as the pole.
• Sensors such as an ultrasonic sensor and accelerometer were used for tracking.
• The Q Network and Deep Q Network models were trained, with the latter showing more promise.
• Identified issues include inconsistent sensor readings and wiring interference.
• Future improvements involve better cable management, consistent training environments, and code adjustments.

FAQ:

Q: What is the cart pole problem? A: The cart pole problem involves balancing a pole on top of a cart using reinforcement learning techniques.

Q: What were the challenges faced during the initial setup? A: The initial setup had wiring limitations that prevented the pole from falling, leading to the modification of treating the top part of the cart as the pole.

Q: What sensors and controls were used? A: An HC-SR04 ultrasonic sensor and an MPU 6050 accelerometer and gyroscope were used to track the cart's position, angle, and angular velocity. The controls were managed through an Arduino microcontroller.

Q: Which training models were implemented, and which one showed more promise? A: Two models were trained: the Q Network and the Deep Q Network. The Deep Q Network showed more promise compared to the Q Network.

Q: What were the identified issues during the project? A: Issues included inconsistent sensor readings, battery voltage drop affecting motor speed, and wiring interference altering the cart's balance and trajectory.

Q: How can the project be improved in the future? A: Future improvements include better cable management, creating a more consistent training environment, and implementing minor code adjustments. These changes, along with extended training time, should enhance the model's performance.