Monitoring Machine Learning Models in Production
Table of Contents
- Introduction
- Understanding Machine Learning Monitoring for Production
- Use of Weights and Biases in Medical Imaging Code Base
- Industrial-Grade Monitoring Systems
- Grafana Cloud and Prometheus Gateway
- Understanding DevOps
- Local Version of Python
- ML Monitor
- Docker Image
- Listening Port
- Jupiter Python Notebook
- Incorporating Metrics into Graphs
- Composite Graph
- UCI Data Sets
- Statistical Machine Learning
- Cloning the Repository
- Production Pipeline Dashboards
- Boston Housing Prices Data Set
- ML Monitor Software
- Normalized RMSE
- Visualization
- Bars and Points
- Min Max Average Current Values Set as Table
- Failure Time
- Mean Squared Error
- Monitoring in Grafana
Understanding Machine Learning Monitoring for Production
Machine learning monitoring for production is a crucial aspect of the machine learning process. It involves monitoring the performance of machine learning models in real-time to ensure that they are functioning as expected. In the last episode, we looked at the use of weights and biases in a medical imaging code base. In this episode, we will be looking at more industrial-grade monitoring systems, namely Grafana Cloud and Prometheus Gateway.
Industrial-Grade Monitoring Systems
Grafana Cloud and Prometheus Gateway are well-understood and well-documented software and systems that are used by most existing companies that use DevOps. They provide Timely outputs, and the graphs are well-built. In this episode, we will be using a local version of Python that is installed on your computer. We will be releasing the code along with this video, and in that code, you will be able to use this ML monitor, which then uses a Docker image to bring your Grafana and Prometheus conversations up.
Local Version of Python
To get started, You will need Python greater than 3.6. This is working for a local version, so it does not support Collab at the moment. First, clone the repository, and then do pip install dot. All of the setup is given in the repository, so it will automatically set everything up. Then, you need to set up Docker. If you don't have Docker on your computer, you might need to install it and run these commands that will install Docker and get it up and running.
ML Monitor
The ML monitor is like a listening port that is constantly listening to your machine learning algorithm. In this case, we will be showing it in terms of a Jupiter Python notebook. The ML monitor is constantly listening at the Jupiter notebook to look for metrics that you are sending to the Grafana Cloud. We will be showing you how to incorporate these metrics into graphs, and you will be finally able to see a composite graph.
Docker Image
Today's version is a local host, but there are also ways to send it to the Grafana Cloud itself or even to expose it as an external URL through some domain name. Today's experiment or today's episode is just to keep things local, to have a local version of Grafana running so that you can have a dockerized version and use the Prometheus gateways in order to send machine learning metrics in you know into the cloud.
Jupiter Python Notebook
The example that we will be seeing today is one of the UCI data sets. Let's get straight into how to use the Grafana Cloud in order for doing statistical machine learning. The first step will be to clone this repository that I've made. This repository is actually heavily Based on this other ML monitoring GitHub, but in this case, this GitHub actually looked at just sending a value. There was no real machine learning model in this case, so that's the reason why I want to mention that this is the source.
Incorporating Metrics into Graphs
Now, in this case, you will actually see an extension, which is this ML monitoring with Grafana, where we actually build a machine learning model and then monitor it. After that, as soon as the monitoring starts, you will have thread logs such as this. With that out of the way, let's get started.
Visualization
Now that we have installed the Docker, the Docker is the only component that you need in order to utilize the Grafana and the Prometheus gateways. I will Show you how to work it, especially for the IPython notebook, which is present in this GitHub, which is called the ML monitoring Grafana. I'm going to show you how to use that and how to Create your own production pipeline dashboards.
Bars and Points
For this task, what we will do is create three different terminals. In the first one, We Are going to invoke the Docker container that has the Grafana dashboards inside. In the Second one, we are going to open Jupiter notebook because that is the notebook in which we will be passing the terms. In the third one, we will import the ML monitor, and once that's there, we are going to say ML monitor.control.start. This is the command that is going to now start sensing, so it puts this sensor or sniffer if you may that's called them on ML monitor.logging so that whatever is coming from your IPython notebook, which is being pushed by this term called RMSE, that will now be detected.
Min Max Average Current Values Set as Table
Now you need to go to localhost 3000, and this is your dashboard of interest. Now you go to home and say new dashboard and choose visualization. First of all, you'll have to pick a metric, so in this case, I'm calling it NRMSE. For the visualization, I'm going to say bars give me points, and also what I can do is I can say min max average current values set as a table to the right.
Failure Time
In the third terminal, here you are going to say Python, and inside Python, you will import the ML monitor. Once that's there, we are going to say ML monitor.control.start. This is the command that is going to now start sensing, so it puts this sensor or sniffer if you may that's called them on ML monitor.logging so that whatever is coming from your IPython notebook, which is being pushed by this term called RMSE, that will now be detected.
Mean Squared Error
Now you see that things have become fixed, and this is probably because it is a stable build. But if the values are changing too much, you know that the values are sort of off. This is essentially how you monitor things in Grafana, and you can add many more metrics, and then it'll keep running.
Monitoring in Grafana
In conclusion, Grafana Cloud and Prometheus Gateway are well-understood and well-documented software and systems that are used by most existing companies that use DevOps. They provide timely outputs, and the graphs are well-built. By using a local version of Python and the ML monitor, you can incorporate metrics into graphs and create your own production pipeline dashboards.
