Enhancing Human-AI Partnerships in Child Welfare: Unveiling Worker Practices and Challenges
Table of Contents
- Introduction
- The Use of AI-Based Decision Support Tools
- 2.1 Limitations of Human Decision-Making
- 2.2 Imperfections and Biases in AI-Based Judgements
- Designing Effective Human-AI Partnerships
- 3.1 The Complementary Strengths of Humans and AI Systems
- 3.2 Achieving Synergy in Human-AI Partnerships
- Adoption of ADS in Child Welfare Agencies
- 4.1 The Allegheny Family Screening Tool (AFST)
- 4.2 Examples of AI-Assisted Decision-Making in Child Welfare
- Existing Research on the AFST in Social Work
- 5.1 Retrospective Quantitative Analyses
- 5.2 Large-Scale Crowd Experiments
- Understanding Workers' Practices and Challenges
- 6.1 The Contextual Inquiries and Interviews
- 6.2 Factors Guiding Workers' Reliance on the ADS
- Workers' Knowledge of Contextual Information
- 7.1 Compensating for Gaps and Limitations in the Model's Risk Scores
- 7.2 Detecting and Overriding Erroneous AFST Recommendations
- Workers' Beliefs about the ADS's Capabilities and Limitations
- 8.1 Limited Authoritative Information on the Model
- 8.2 Workers' Strategies to Learn about the Model
- 8.3 Calibrating Reliance on Algorithmic Recommendations
- Organizational Pressures and Incentive Structures
- 9.1 Monitoring Performance Measures
- 9.2 The Influence of Decision Protocols
- Misalignments between Algorithmic Predictions and Workers' Decision Targets
- 10.1 Immediate Safety Concerns vs. Long-Term Risk Predictions
- The AFST as a Source of Tension for Workers
- 11.1 Perceived Missed Opportunity to Complement Workers' Expertise
- A More Complicated Picture of Reliance on ADS
- Design Implications for Researchers and Public Sector Agencies
- 13.1 Training and Decision-Time Interfaces
- 13.2 Stakeholder Involvement in Model-Level Design Decisions
- 13.3 Collaborative Determination of Decision-Making Power
- Conclusion
- Acknowledgements
- FAQs
💡 Highlights
- AI-based decision support tools (ADS) are being increasingly deployed across complex decision-making contexts.
- Workers in child welfare agencies use their contextual knowledge to compensate for limitations in AI models.
- Limited transparency and organizational pressures influence workers' reliance on ADS.
- Design implications include better training interfaces, stakeholder involvement, and power distribution in decision-making.
Introduction
In recent years, there has been a growing adoption of AI-based decision support tools (ADS) in various high-stakes decision-making contexts, including child welfare agencies. The goal is to leverage the strengths of both humans and AI systems to overcome limitations in human decision-making. However, the use of ADS also raises concerns about the imperfections and biases inherent in AI-based judgements. This article provides an in-depth exploration of workers' practices, challenges, and desires for algorithmic decision support in the field of child welfare. By understanding how workers rely on and perceive these tools, we can identify design implications for improving human-AI partnerships in child welfare.
The Use of AI-Based Decision Support Tools
2.1 Limitations of Human Decision-Making
While humans have expertise and intuition, their decision-making processes are not without limitations. ADS have been introduced in various domains to address these limitations and enhance decision outcomes. However, it is important to recognize that AI-based judgements also come with their own imperfections and biases.
2.2 Imperfections and Biases in AI-Based Judgements
AI models, although capable of processing vast amounts of data, are not perfect. They can introduce biases and make errors, albeit different from those made by humans. The recognition of the duality of AI's potential improvements and risks is increasing in popular discourse. It is essential to understand how to design effective human-AI partnerships that can draw on the complementary strengths of each.
Adoption of ADS in Child Welfare Agencies
Child welfare agencies in the United States have started adopting ADS to assist social workers in making screening decisions about potential child maltreatment cases. One notable example is the Allegheny Family Screening Tool (AFST), which outputs a risk score for future out-of-home placement. The AFST has been studied extensively in research communities, including HCI, ML, and FAccT. However, there is still limited understanding of how workers actually use the tool in their day-to-day work.
Existing Research on the AFST in Social Work
Existing research on the AFST and ADS in social work has primarily relied on retrospective quantitative analyses or large-scale crowd experiments that abstract away from the context in which workers make decisions. This lack of understanding prompted the need for an in-depth qualitative investigation into workers' current practices and challenges when working with the AFST.
Understanding Workers' Practices and Challenges
To gain insights into workers' practices and challenges, the researchers conducted contextual inquiries and semi-structured interviews with call screeners and their supervisors in a child welfare agency. The observations and interviews revealed four high-level themes that guided workers' reliance on the ADS.
Factors Guiding Workers' Reliance on the ADS
Workers demonstrated a strong reliance on their knowledge of rich, contextual information about a given case. They compensated for gaps and limitations in the model's risk scores by leveraging their understanding of case-specific context from allegations. This finding confirmed hypotheses that workers can detect and override erroneous AFST recommendations by cross-checking algorithmic outputs with other case-related information.
Workers' Beliefs about the ADS's Capabilities and Limitations
Due to limited authoritative information on the AFST model, workers improvised strategies to learn about the model on their own. For instance, they played guessing games with colleagues to refine their ability to predict risk scores. These informal beliefs and intuitions about the model's behavior influenced workers' reliance on algorithmic recommendations.
Organizational Pressures and Incentive Structures
Workers' decisions to rely on the algorithm were influenced by organizational pressures and incentive structures related to the use of the ADS. Performance measures were monitored, and workers perceived a need to avoid disagreement with high algorithmic risk scores. Some workers felt that their expertise as human decision-makers was undervalued due to decision protocols that discouraged disagreement with the ADS.
Misalignments between Algorithmic Predictions and Workers' Decision Targets
The model's prediction of the risk of out-of-home placement in two years did not Align with workers' immediate safety concerns for the child. Workers struggled to incorporate long-term risk predictions into their decision-making processes, leading to tension between their own judgement and the ADS.
The AFST as a Source of Tension for Workers
Despite being used for nearly half a decade, the AFST remained a source of tension for many workers. The tool was perceived as a missed opportunity to complement workers' own abilities as human experts. Existing narratives that depict frontline decision-makers as either too skeptical or too reliant on ADS outputs fail to capture the complexities observed in this study.
Design Implications for Researchers and Public Sector Agencies
The findings from this research have important design implications for both researchers and public sector agencies. Training and decision-time interfaces need to be tailored to support workers in understanding the boundaries of an ADS's capabilities. Methods to involve diverse stakeholders in shaping an ADS's model-level design decisions should be explored. It is crucial to communicate clearly and collaboratively determine the distribution of decision-making power between workers and the ADS, especially in situations where disagreement may arise.
Conclusion
This article presented an in-depth qualitative investigation into workers' practices, challenges, and desires for algorithmic decision support in child welfare. It highlighted the importance of considering the limitations and biases of both human decision-making and AI-based judgements. The research findings revealed the factors that guide workers' reliance on the ADS, including their knowledge of contextual information and beliefs about the capabilities and limitations of the model. Organizational pressures and misalignments between algorithmic predictions and workers' decision targets also influenced their use of the ADS. The article concludes with design implications for researchers and public sector agencies to improve human-AI partnerships in child welfare.
Acknowledgements
The researchers would like to express their gratitude to the Allegheny County for their valuable input and commitment to transparency, which shaped this research.
FAQs
Q: What is the Allegheny Family Screening Tool (AFST)?
The AFST is an AI-based decision support tool adopted by child welfare agencies, specifically designed to assist social workers in making screening decisions for potential child maltreatment cases.
Q: How do workers use their contextual knowledge in conjunction with the ADS?
Workers leverage their understanding of case-specific context from allegations to compensate for gaps and limitations in the model's risk scores. They consider cultural misunderstandings, ulterior motives of callers, and other case-related information to inform their decisions about whether to override a high-risk score.
Q: How do workers learn about the AFST model?
Due to limited authoritative information on the model, workers often resort to improvising strategies to learn about it on their own. This can include playing guessing games with colleagues to hone their ability to predict risk scores over time.
Q: How do organizational pressures influence workers' reliance on the ADS?
Organizational pressures, such as the monitoring of performance measures, influence workers' decisions to rely on the algorithm. Some workers feel the need to agree with high algorithmic risk scores to avoid being perceived as disagreeing with the ADS too often.
Q: What are the design implications of the study?
The study suggests designing training and decision-time interfaces that support workers in understanding the capabilities and limitations of an ADS. It also emphasizes the importance of involving diverse stakeholders in shaping the ADS's model-level design decisions. Additionally, the study highlights the need for clear communication and collaborative determination of decision-making power between workers and the ADS.
Q: What are the limitations of existing narratives on workers' reliance on ADS?
Existing narratives often depict frontline decision-makers as either too skeptical or too reliant on ADS outputs. However, this study uncovered a more nuanced picture that is not fully captured by these narratives.
Resources:
