Explore the Responsible AI Tracker

Table of Contents

1. Introduction
2. What is Responsible AI Tracker?
3. Using Responsible AI Tracker for Model Comparison and Validation
   1. Importing the UCI Income Dataset
   2. Training and Registering a Model
   3. Analyzing Model Performance with Responsible AI Dashboard
   4. Mitigating Issues with Data Balancing
      1. Balancing Both Cohorts
      2. Targeted Mitigation Approach for the Married Cohort
4. Comparing Model Performance and Metrics
5. Creating Custom Cohorts
6. Conclusion

Introduction

Welcome to this tour on Responsible AI tracker. In this article, we will explore the functionality and features of Responsible AI tracker, an open-source extension to the Jupyter Lab framework. Responsible AI tracker helps data scientists track and compare different iterations or experiments on model improvement. We will learn how to use tracker to compare and validate different model improvement experiments, and how to use the extension in combination with other tools in the Responsible AI toolbox.

What is Responsible AI Tracker?

Responsible AI tracker is an extension to the Jupyter Lab framework that allows data scientists to track and compare different iterations or experiments on model improvement. It brings together notebooks, models, and visualization reports on model comparison within the same interface. Responsible AI tracker is part of the Responsible AI toolbox, a larger open-source effort at Microsoft for bringing together tools for accelerating and operationalizing Responsible AI.

Using Responsible AI Tracker for Model Comparison and Validation

Importing the UCI Income Dataset

To demonstrate how to use Responsible AI tracker, we will Create a project that uses the UCI Income dataset. This dataset is a classification task for predicting whether an individual earns more or less than 50K. We can also import a notebook that contains code for building a model or cleaning up data.

Training and Registering a Model

Once the project is set up, we can train a model on a split of the UCI Income dataset. We will build a model with five estimators using a graded boosted model algorithm and perform basic feature imputation and encoding. After training the model, we can register it to the notebook for future reference.

Analyzing Model Performance with Responsible AI Dashboard

To gain insights into the model's performance, we can use the Responsible AI dashboard, another tool in the Responsible AI toolbox. The dashboard consists of visual components for error analysis, interpretability data exploration, and fairness assessment. For example, the error analysis component shows the overall error rate and identifies cohorts where the error rate is higher, such as married individuals.

Mitigating Issues with Data Balancing

To address performance issues identified through the Responsible AI dashboard, we can Apply data balancing techniques using the raimitigations library. We explore two approaches: balancing both cohorts separately and targeting data balancing only for the married cohort. These approaches aim to improve model performance and mitigate issues related to imbalanced data.

Balancing Both Cohorts

By balancing both cohorts separately, we can achieve improvement in model accuracy. However, this approach may introduce performance drops for the not married cohort, which raises concerns about backward compatibility issues and user trust.

Targeted Mitigation Approach for the Married Cohort

Alternatively, we can balance only the married cohort while leaving the rest of the data untouched. This approach results in overall improvement and avoids performance drops for the not married cohort. Precision for the married cohort is also higher compared to the blanket approach.

Comparing Model Performance and Metrics

After applying mitigation techniques, we can compare the performance of different models across various metrics. By using the model comparison table, we can assess the improvement achieved and identify which models perform better for specific cohorts. It provides a comprehensive picture of the improvements and variations across different models, cohorts, and metrics.

Creating Custom Cohorts

Responsible AI tracker allows us to create custom cohorts Based on specific criteria or characteristics of the data. By defining cohorts and analyzing their performance separately, we can gain deeper insights and tailor mitigation strategies accordingly. For example, we can create a cohort for married individuals with more than eleven years of education and assess its performance.

Conclusion

Responsible AI tracker is a valuable tool for data scientists working with Jupyter Lab and the Responsible AI toolbox. It enables them to track, compare, and validate model improvement experiments. By leveraging the Responsible AI dashboard and data balancing techniques, data scientists can enhance model performance, identify and mitigate issues, and gain insights into the impact of changes on different cohorts. Start using Responsible AI tracker today to accelerate and operationalize responsible AI practices.

Highlights

• Responsible AI tracker is an open-source extension to the Jupyter Lab framework that helps data scientists track and compare model improvement experiments.
• Responsible AI tracker is part of the Responsible AI toolbox, an open-source effort by Microsoft to provide tools for accelerating and operationalizing responsible AI.
• The Responsible AI dashboard provides visual components for error analysis, interpretability data exploration, and fairness assessment.
• Data balancing techniques, implemented through the raimitigations library, can mitigate issues related to imbalanced data.
• Comparing model performance across different metrics and cohorts allows for Better Insights and decision-making.
• Custom cohorts can be created to analyze and address specific issues within the data.
• Responsible AI tracker helps data scientists enhance model performance, identify backward compatibility issues, and build trust with end users.

FAQ

Q: Can Responsible AI tracker be used with other machine learning platforms besides Sklearn? A: Yes, Responsible AI tracker supports different machine learning platforms and can be integrated into various workflows.

Q: Are there other tools in the Responsible AI toolbox that complement Responsible AI tracker? A: Yes, the Responsible AI toolbox includes other tools such as the Responsible AI Dashboard and the Responsible AI Mitigations Library, which can be used in combination with Responsible AI tracker for a comprehensive AI development and deployment process.

Q: Can I use Responsible AI tracker with my own custom datasets? A: Yes, Responsible AI tracker allows you to import your own datasets and notebooks, enabling you to apply its functionalities to your specific data and models.