Explainable AI through Automated Rationale Generation

Explainable AI through Automated Rationale Generation

Table of Contents:

1. Introduction
2. The Challenge of Human-AI Communication
3. The Need for Explainable AI
4. Generating Natural Language Explanations 4.1 Rationale Generation: An Approach 4.2 Machine Translation Intuition 4.3 Collecting Rationales from Humans 4.4 Training the Model 4.5 Evaluating the Explanations
5. Perception Study: Comparing Rationale Systems 5.1 Experiment Design 5.2 Quantitative Analysis 5.3 Qualitative Analysis 5.4 Insights into Confidence, Human-Likeness, and Understandability
6. Preference Study: Complete View vs Focused View 6.1 Understanding the Perceived Differences 6.2 Preferences in Confidence and Failure 6.3 Role of Detail in Understandability
7. Limitations, Future Research, and Benefits
8. Summary and Conclusion

Generating Natural Language Explanations: Enhancing Human-AI Communication

Introduction Good afternoon everyone and thank you for joining me today at IUI. In this presentation, I will be discussing our work on explainable AI, specifically focusing on Automated Rationale Generation. My name is Upol, and I am excited to present the collaborative efforts of Georgia Tech, Cornell, and the University of Kentucky in the field of Entertainment Intelligence Lab.

The Challenge of Human-AI Communication In recent years, the emergence of AI systems has raised concerns about their lack of explainability. Unlike humans, AI currently lacks the ability to express its motivations and actions in an interpretable manner. This poses difficulties for both non-technical and technical experts in understanding and trusting these "black box" systems. Collaboration becomes challenging without trust. This presentation aims to address this challenge by exploring the generation of natural language explanations.

The Need for Explainable AI Imagine a scenario where a self-driving car could communicate its decision-making process through natural language explanations. Although this may be aspirational, it highlights the importance of building trust in AI systems. To achieve this, we need to generate plausible natural language explanations from a human-centered perspective. By understanding how these explanations are generated and their impact on user perceptions and preferences, we can enhance the transparency and trustworthiness of AI systems.

Generating Natural Language Explanations Our approach to generating natural language explanations is called Rationale Generation. Just like humans convey their motivations and goals through language, we aim to translate AI system actions and states into explanations. Inspired by various fields, including philosophy, science, mind, and psychology, we treat explanation generation as a translation problem. By translating data structures and numbers into natural language, we bridge the gap in human-AI communication.

Collecting Rationales from Humans To generate the necessary rationales, we collect data from humans performing a task while thinking out loud. We designed a game similar to Frogger, where participants play and provide explanations for their actions. By integrating turn-taking gameplay and automated speech-to-text systems, we reduce participant burden and improve data accuracy. This iterative review process helps minimize errors and ensures a high-quality corpus for training the model.

Training the Model Using a sequence-to-sequence neural network, we set up a straight-forward translation problem where the input is the action and state, and the output is the natural language rationale. We explore two configurations: Complete View and Focused View. Complete View utilizes the entire screen with injected noise to generalize the rationales, while Focused View focuses on a partial window around the game character without noise injection. This distinction enables us to generate either holistic or localized rationales.

Evaluating the Explanations To evaluate the generated explanations, we conducted perception and preference studies. The perception study involved participants watching videos of Frogger gameplay accompanied by different rationales. The goal was to assess whether our system outperformed the randomly generated baseline and how close it was to the exemplary rationale. The preference study compared the two rationale styles, focusing on confidence, failure, and unexpected behavior. Through quantitative and qualitative analysis, we gained insights into how users perceive and prefer different rationale styles.

Limitations, Future Research, and Benefits While our current study provides promising results, it is important to acknowledge the limitations. The Frogger game is not a real-world representative sample, and task-specific data collection is required for different communities of practice. However, the potential benefits of rationale generation in enhancing human-AI communication are significant. By improving user engagement and facilitating better understanding of AI systems, we can foster trust and collaboration.

Summary and Conclusion In conclusion, our research on generating natural language explanations contributes to the field of explainable AI and human-AI communication. By taking a step towards translating machine actions and states into interpretable rationales, we address the challenge of trust and transparency. Our perception and preference studies demonstrate the effectiveness of the rationale generation approach and provide insights into user perceptions. Future research will focus on interactivity, contestability, and team dynamics to further enhance human-AI communication.

Highlights:

• Exploring explainable AI through Automated Rationale Generation
• Generating plausible natural language explanations for AI actions
• Using a translation approach inspired by philosophy, science, mind, and psychology
• Collecting rationales through turn-taking gameplay and automated speech-to-text
• Training a sequence-to-sequence neural network for rationale generation
• Evaluating user perceptions and preferences through perception and preference studies
• Enhancing trust, transparency, and collaboration in human-AI communication

FAQ:

Q: How does rationale generation improve collaboration and trust in AI systems? A: Rationale generation allows AI systems to communicate their motivations and actions in a human-interpretable manner. This bridging of the communication gap enhances understanding, fosters trust, and facilitates collaboration between non-technical and technical experts.

Q: Can rationale generation be applied to complex real-world scenarios? A: While our current study focuses on a simplified game environment, rationale generation can be applied to real-world scenarios. The key is breaking down tasks into actions, states, and explanations, making data collection specific to the community of practice.

Q: What are the potential future directions for this research? A: Future research aims to introduce interactivity and contestability to the explanation system, allowing users to question and contest explanations. Additionally, exploring team dynamics and the role of explanations in multi-agent systems is an exciting avenue for further investigation.