Build and Deploy Recommendation System with BigQuery ML
Table of Contents
- Introduction
- The Importance of Personalization and Recommendation Systems
- Building Recommendation Systems with BigQuery ML
- Understanding Explicit and Implicit Feedback
- Using Google Merchandise Store Data to Train a Recommendation System
- Creating a Model with Matrix Factorization Algorithm in BigQuery ML
- Evaluating the Model and Making Predictions
- Exporting Recommendations for Use in Production
- Tips and Best Practices for Improving Recommendation Systems
- Dealing with the Cold-Start Problem
- Finding Lookalikes for New Users
- Hosting Models for Real-Time Predictions with AI Platform
Building Recommendation Systems with BigQuery ML
In today's world, consumers expect personalization and recommendations of products and services tailored to their interests. To better serve customers, businesses need to be able to train and deploy recommendation systems quickly and effectively to improve conversions, click-through rates, and build customer satisfaction, loyalty, and brand affinity. But to build recommendation systems, You need a place for your data. Google BigQuery is a fully managed, highly scalable data warehouse solution on Google Cloud that comes with lots of great features, especially around storage and analytics. With BigQuery ML, you can train and deploy machine learning models directly using SQL, which means you've got data storage, analytics, and machine learning all within BigQuery.
The Importance of Personalization and Recommendation Systems
Personalization and recommendation systems have become increasingly important in today's digital age. Consumers are bombarded with an overwhelming amount of information and choices, and they expect businesses to provide them with personalized recommendations that are tailored to their interests and needs. By providing personalized recommendations, businesses can improve customer satisfaction, loyalty, and brand affinity, as well as increase conversions and click-through rates.
Understanding Explicit and Implicit Feedback
To recommend something to someone, you need to know what they like already. In general, there are two ways to do this: explicit feedback and implicit feedback. If your users directly give you ratings of 4 and 1/2 stars out of 5 on a particular product or piece of content, then this is called explicit feedback. The Second way is to infer interest indirectly. For example, maybe the more time a user spends viewing a product, the more they're interested in it. Since you're inferring interest indirectly or implicitly, this is called implicit feedback. As implicit feedback tends to be more common in retail businesses, we'll build a recommendation system using some sample implicit feedback data from a real online store.
Using Google Merchandise Store Data to Train a Recommendation System
The Google Merchandise Store is an actual e-commerce store where people can shop for Google-branded shirts, hats, and other products. A sample of the anonymized Google Analytics data is available as one of the BigQuery public data sets. Based on the store's Google Analytics data, we've cleaned the data so that we can see each anonymized visitor ID, what product they were looking at, and how long they spent viewing the product, which is in milliseconds here. We need these three columns to train a recommendation system in BigQuery ML -- the user, the product, and the ratings. Essentially, the more time they spend viewing the product, the higher the implicit rating.
Creating a Model with Matrix Factorization Algorithm in BigQuery ML
With the data ready, we can now Create the model by specifying a model name like retail recommender in our data set BQML. Then we specify the options, including the model Type, which in this case is matrix factorization for recommendation systems, the user column, the item column, and the rating column. We're setting feedback type to implicit because the model is trained differently for explicit and implicit feedback. Then within AS, we select the training data. After training is done, we can evaluate our model using select star from ML dot evaluate. BigQuery ML automatically splits the training data into its own training and testing data behind the scenes for you, so this allows you to evaluate your model immediately after the model has finished training.
Evaluating the Model and Making Predictions
Of these evaluation metrics, average rank, also known as mean percentile rank, is perhaps the most widely used metric for implicit matrix factorization. In general, the lower the average rank, the more closely the predicted recommendations match the behavior in the test set, with 0.5 being random chance and 0 being a perfect prediction. If you're happy with the evaluation metrics, you can now make predictions. To make predictions with BigQuery ML on a single user or a set of users to start, you can run select star from ML dot recommend, specifying the model you created and the user ID or IDs that you want to make predictions for. To get the top N recommendations, you can order by the predicted confidence descending and limit N. You can also make predictions on all the users, so you have the top N recommendations for every single user.
Exporting Recommendations for Use in Production
Now that we have our predicted recommendations, what can we do to use the recommendations in production? One way is to export recommendations for ad retargeting campaigns with Google Analytics. By focusing on specific products at a time, you can create a new column for likelihood to purchase this product based on the predicted recommendations. Then you can import your predictions back into Google Analytics to create new campaigns for those products. Another way is by connecting the predicted recommendations with your CRM, like Salesforce. And by doing so, you can create targeted email campaigns to provide Relevant products directly to your customers' inboxes. There are even integrations between Google Analytics and Salesforce that you can leverage. You can also export the recommendations to Cloud Storage, a separate database, or simply Read from BigQuery to use in a separate service to serve the recommendations.
Tips and Best Practices for Improving Recommendation Systems
As you might be thinking, how you can potentially improve your recommendation systems in production, here are some tips and best practices:
Dealing with the Cold-Start Problem
One question you might have is what happens to new users. In other words, since matrix factorization can only make predictions on users who belong to the training data, how do you make predictions on users who you don't have any data for? Well, this is actually a common problem called the cold-start problem. There are many ways to deal with this, which is outside the scope of this video. But as a simple suggestion, the easiest way would be to Show users items that are sorted by popularity or what products have been trending for the past timed interval.
Finding Lookalikes for New Users
Another question you may ask is how do you recommend items to new users if you know some of their demographic data? One way is to find lookalikes to those new users based on your pool of existing users. For example, given some demographic data for User X, first compare User X to all of the existing users that you already have recommendations for. Then find the five most similar users, we'll call these lookalikes, to User X based on whatever data you may have -- demographic data or otherwise. Then make recommendations to User X based on what these lookalikes would themselves be recommended. To find lookalikes, one way is to use Cosine similarity between User X and the other users. Another way that you can find lookalikes is to find the nearest cluster of users and use them as your lookalikes. And you can do this using a clustering algorithm like k-means, which is also available with BigQuery ML.
Hosting Models for Real-Time Predictions with AI Platform
BigQuery ML does batch predictions very well, but your business may need predictions on a single user at a time, and you need the predictions to be fast. In these situations, you can export your matrix factorization models from BigQuery ML and host it for real-time predictions using AI Platform on Google Cloud.
Conclusion
In this video, we've shown you how to build and deploy a recommendation system with BigQuery ML. We've covered the importance of personalization and recommendation systems, understanding explicit and implicit feedback, using Google Merchandise Store data to train a recommendation system, creating a model with matrix factorization algorithm in BigQuery ML, evaluating the model and making predictions, exporting recommendations for use in production, and tips and best practices for improving recommendation systems. With these tools and techniques, you can create powerful recommendation systems that improve customer satisfaction, loyalty, and brand affinity, as well as increase conversions and click-through rates.
