AI Roasting: Generating Insults from Reddit's r/RoastMe

Table of Contents

1. Introduction to Machine Learning
2. Long Short-Term Memory (LSTM)
3. The Role of Feedback Connections in LSTMs
4. Understanding Feed Forward Neural Networks
5. Generating Sequential Data with LSTMs
6. Using Reddit Comments for Training
7. Technical Details of the Project
8. Challenges with Scraping Data from Reddit
9. Preprocessing the Data for Training
10. Encoding Characters as Numbers with One-Hot Encoding
11. Splitting the Data into Training Instances
12. Using Numpy for Efficient Data Processing
13. Creating the LSTM Model
14. Training the Model on the Data
15. Analyzing the Results of the Trained Model
16. Filtering Characters and Reducing Data Size
17. Scraping Data from r/RoastMe Subreddit
18. Modifying the Scraper Script for RoastMe Comments
19. testing the Model on a Smaller Dataset
20. Improving the Model and Increasing Training Speed
21. Analyzing the Results of the Enhanced Model
22. Closing Remarks and Future Ideas for Improvement

Introduction to Machine Learning

Machine learning is a field of study that focuses on developing algorithms and models that allow computers to learn from and make predictions or decisions based on data. In this article, we will specifically explore an architecture called Long Short-Term Memory (LSTM) in machine learning.

Long Short-Term Memory (LSTM)

LSTM is a type of neural network architecture that incorporates feedback connections to better analyze sequences of data. Unlike regular feed forward neural networks, which process data and forget about it when new data is introduced, LSTMs are designed to remember past information to generate Meaningful sequential data, such as text.

The Role of Feedback Connections in LSTMs

The inclusion of feedback connections in LSTMs enables the neural network to retain and utilize information from previous instances, even when generating new data. This is especially useful when working with sequential data where the context is crucial, such as classifying pictures into categories or generating text that makes sense.

Understanding Feed Forward Neural Networks

A feed forward neural network is a standard neural network architecture that processes data by passing it through Hidden layers and outputting a result. It is commonly used for tasks where each input is independent of the others, such as image classification. The previous inputs do not influence the output for the current input.

Generating Sequential Data with LSTMs

In contrast, LSTMs excel at generating sequential data, like text, where the previous inputs have a direct influence on the next ones. To demonstrate this capability, we will be using Reddit comments, specifically from the r/RoastMe subreddit, to train our LSTM model. The goal is to generate auto-generated roasts based on the Patterns and structures the model has learned from the training data.

Using Reddit Comments for Training

At first, the intention was to use comments from multiple subreddits, but the focus shifted to using comments from the r/RoastMe subreddit. This subreddit is ideal for training an LSTM model because the comments are mainly insults, which serve as excellent training material. Additionally, we only consider top-level comments and ignore replies to maintain a coherent and insult-oriented dataset.

Technical Details of the Project

While this article is not a full Tutorial with code, I will explain the technical aspects of the project and provide links to the corresponding GitHub repositories for those interested in examining or replicating the project. The primary libraries and tools used in this project include PRAW (Python Reddit API Wrapper), numpy for efficient data processing, and Keras for building and training the LSTM model.

Challenges with Scraping Data from Reddit

Scraping data from Reddit posed some challenges, including intermittent internet connection issues and slow scraping speeds. These challenges were Partly due to Reddit's server limitations to prevent excessive requests from users. However, with patience and modifications to the scraper script, we managed to overcome these obstacles and obtain a substantial amount of data for training the LSTM model.

Preprocessing the Data for Training

After securing the dataset from Reddit comments, the next step involved preprocessing the data to prepare it for training. This process included concatenating and filtering the comments, assigning numerical values to each unique character using one-hot encoding, and splitting the data into instances that would serve as input-output pairs for the LSTM model.

Encoding Characters as Numbers with One-Hot Encoding

To process text data with an LSTM model, we need to convert characters into numerical representations. To achieve this, we utilized one-hot encoding, where each character is assigned a specific number. This encoding scheme allows the neural network to work with categorical data, even though there is no inherent order among the characters.

Splitting the Data into Training Instances

As Mentioned earlier, LSTMs benefit from analyzing the previous outputs to generate meaningful sequential data. To facilitate this learning, we split the data into sentences, where each sentence consists of 30 characters. For each sentence, the corresponding output is the subsequent character in the original text. This setup enables the LSTM model to learn the patterns and relationships between characters.

Using Numpy for Efficient Data Processing

Numpy, a powerful library in Python, was employed to handle the manipulation and processing of the dataset. Numpy provides efficient, high-performance numerical operations ideal for working with large amounts of data. By utilizing numpy, the workflow of the project was accelerated, and computations were performed more efficiently.

Creating the LSTM Model

The LSTM model was constructed using Keras, a user-friendly deep learning library. The model includes a single LSTM layer, a dense layer, and a softmax activation layer. The LSTM layer allows the model to learn from sequential data, while the dense layer and softmax activation enable the generation of appropriate predictions based on the learned patterns.

Training the Model on the Data

Once the LSTM model was defined, it was trained on the Reddit comments dataset. The training process involved setting the learning rate, specifying a callback function, and fitting the data to the model. Although the training time was longer than anticipated, the model performed reasonably well even after a single epoch of training.

Analyzing the Results of the Trained Model

After training the model, we evaluated the results by generating auto-generated roasts based on the patterns learned from the Reddit comments. While not all the insults generated by the model made complete sense, many of them exhibited coherent structures and included insults commonly found on the r/RoastMe subreddit. For a first attempt at working with LSTMs, the results were quite promising.

Filtering Characters and Reducing Data Size

To address the issue of excessive data size, which posed challenges during training, we filtered out unnecessary characters and reduced the dataset's Dimensions. By focusing on English alphabet letters, space, new lines, dots, question marks, exclamation marks, dashes, and a few additional characters, we significantly reduced the data size without compromising the model's performance.

Scraping Data from r/RoastMe Subreddit

To make the project more interesting and engaging, we decided to scrape data specifically from the r/RoastMe subreddit. This subreddit provides an abundance of insults in the comments section, making it an ideal source for training the LSTM model. By extracting the top-level comments and excluding automated bot messages, we obtained a suitable dataset for our purposes.

Modifying the Scraper Script for RoastMe Comments

To adapt the existing scraper script for the r/RoastMe subreddit, a few modifications were made. The script was updated to Collect all comments into a single file, exclude header lines that contain post information, and ensure the focus was solely on insult-oriented comments. These adjustments streamlined the data collection process and allowed for further analysis and training.

Testing the Model on a Smaller Dataset

To test the improved model and its training speed, we worked with a smaller dataset containing approximately 3.5 million characters. This sample dataset proved sufficient for testing purposes, as it yielded reasonable results with significantly reduced training time. The model successfully generated auto-generated roasts, although coherence and structure were not always perfect.

Improving the Model and Increasing Training Speed

To further enhance the model's performance, we made additional adjustments, including increasing the model's complexity and introducing dropout layers. However, these modifications significantly increased training time, leading us to decrease the training data size to only 1 million characters. This compromise allowed us to achieve faster training times while still yielding satisfactory results.

Analyzing the Results of the Enhanced Model

After training the enhanced model for a considerable number of epochs, we evaluated the generated insults based on their coherence and creativity. While some insults lacked logical sense, many of them were amusing and even displayed a degree of accuracy. The enhanced model demonstrated an improved ability to generate insults that resembled those commonly found on r/RoastMe.

Closing Remarks and Future Ideas for Improvement

In conclusion, this project provides an introduction to machine learning, specifically focusing on the application of LSTM models in generating auto-generated roasts from Reddit comments. The results achieved with the LSTM model demonstrate its potential for sequential data generation. Future improvements may involve incorporating ideas from other renowned machine learning researchers and exploring different avenues for training data collection and preprocessing.

Overall, this project serves as a stepping stone into the vast field of machine learning and opens up possibilities for further innovation and experimentation.