Investigating the applications of model reasoning for human-aware AI systems

Abstract

This dissertation investigates how intelligent agents can reason over their models to better support, explain, and adapt to human users. Traditional AI planning assumes that the underlying model—describing the agent's actions, goals, and environment—is fixed and complete. However, in real-world deployments, these models often diverge from users' expectations, leading to confusion, mistrust, or failure. To address this, I propose a shift from reasoning within a model to reasoning about the model itself, using a framework called model-space search. Through four interconnected works, I demonstrate how model reasoning enables agents to operate more effectively in human-aware settings. First, I show how agents can proactively support users by detecting likely failure due to model misalignment and suggesting minimal corrections. Second, I extend explanation frameworks to include the intentions of system designers, revealing hidden influences on agent behavior. Third, I introduce Actionable Reconciliation Explanations, which combine model reconciliation and excuse generation to help users both understand and influence agent behavior. Finally, I explore how Large Language Models can enhance model-space search by guiding it toward more plausible and interpretable updates. Together, these contributions establish model reasoning as a foundation for building AI systems that are not only autonomous but also transparent, adaptable, and aligned with the people they serve.