Sibyl: Explaining Machine Learning Models for High-Stakes Decisions

Sibyl: Explaining Machine Learning Models for High-Stakes Decisions

Table of Contents

1. Introduction
2. Explaining Machine Learning Predictions in High-Risk, Low-Technical-Expertise Domains
   1. The Importance of Explanation in High-Risk Domains
   2. Context Dependency in Explanation
3. Investigating the Domain of Child Welfare Screening
   1. The Decision-Making Workflow in Child Welfare Screening
   2. Integration of Machine Learning in the Workflow
4. Identifying the Need for Explanations in Child Welfare Screening
   1. Observing Screeners' Trust in the Model
   2. Uncertainty and Confusion Surrounding Risk Scores
   3. Ethical Considerations in Oversimplifying Complex Cases
5. Gathering Feedback from Screeners
   1. Understanding the Information Needs of Screeners
   2. Designing High-Fidelity Mockups for Feedback
   3. Interest in "What-If" Style Explanations
6. Introducing the Sibyl Tool for Explanations
   1. Overview of the Sibyl Tool and Its Interfaces
   2. Feature Contribution Explanations
   3. "What-If" Explanations
   4. Historical Feature Distributions Visualization
   5. Global Feature Importance Explanation
7. User Study with the Sibyl Interfaces
   1. Formal User Study Methodology
   2. Findings on the Effectiveness of Feature Contribution Explanations
   3. Importance of Focusing on Case Explanations
   4. The Importance of Interpretable and Human-Worded Explanations
8. Future Work and Deployment of the Sibyl Tool
   1. Quantitative Evaluation of the Sibyl Tool
   2. Steps Towards Deployment
9. Example of a Sibyl Interface: The Feature Contributions Page
10. Conclusion

Explaining Machine Learning Predictions in High-Risk, Low-Technical-Expertise Domains

Machine learning predictions play a crucial role in high-risk domains where the consequences of decisions are significant. However, understanding and interpreting these predictions can be challenging, especially for users with low technical expertise in machine learning. In this article, we will explore the importance of explaining machine learning predictions in such domains and delve into the specific case of child welfare screening. By investigating the decision-making workflow and the usage of machine learning tools in this domain, we aim to understand the benefits and challenges of providing explanations for risk score predictions.

The Importance of Explanation in High-Risk Domains

In high-risk domains, where decisions can have severe consequences, explanations of machine learning predictions are vital. Obtaining insights into how the model arrives at a particular prediction can help users validate its reliability and build trust. However, the approach to providing explanations may vary depending on the context, the nature of the task, and the expertise of the users. We recognize that the users in high-risk domains may not have prior knowledge of machine learning, necessitating the need for clear and easily understandable explanations tailored to their background and requirements.

Context Dependency in Explanation

Explaining machine learning predictions is not a one-size-fits-all approach. The best way to provide explanations depends on the unique characteristics of each domain. To understand the role of explanations in a specific high-risk domain, we focused on child welfare screening. This domain involves social workers evaluating referrals for potential cases of child abuse. The decision-making process includes reviewing referral details and assessing risk scores generated by a machine learning model. By investigating this domain, we aim to uncover the challenges and potential benefits of explaining risk score predictions in a low-technical-expertise setting.

Investigating the Domain of Child Welfare Screening

Child welfare screening is a critical area where machine learning models can assist social workers in making informed decisions. The decision-making workflow begins when a referral for a potential child abuse case is received by the responsible agency. A group of social workers assesses the referral details and associated information, such as referral history, criminal history, and demographic data of the parties involved. Based on this assessment, they decide whether to screen the case in for further investigation or screen it out.

Integration of Machine Learning in the Workflow

As part of the child welfare screening workflow, social workers are provided with a machine learning-generated risk score prediction. This risk score predicts the likelihood of a child being removed from their home within two years if the case is screened in. To evaluate the potential benefits of explanations in this domain, we closely studied the screeners' interaction with the machine learning model. Our observations revealed instances where the screeners expressed distrust or confusion concerning the model's predictions, highlighting the need for effective explanations.

Identifying the Need for Explanations in Child Welfare Screening

To further understand the requirements and expectations of screeners in the child welfare screening domain, we conducted interviews to Gather their feedback. We first explored the type of information they would be interested in obtaining to better interpret and utilize the risk score predictions. Additionally, we presented high-fidelity mockups of explanation tools to Collect concrete feedback on their design and features.

Understanding the Information Needs of Screeners

The interviews with screeners confirmed their interest in having explanations for the risk scores provided by the machine learning model. Notably, they expressed a desire to receive a list of risk factors and protective factors considered by the model. These insights would aid them in comprehending the underlying basis for the risk predictions. Furthermore, screeners showed keen interest in exploring "what-if" style explanations, allowing them to understand the impact of changing feature values on the risk score predictions.