PhD defence of Shalaleh Rismani – Built In, Not Bolted On: Operationalizing System Safety for AI

Abstract

AI systems of varying autonomy are increasingly embedded in how we communicate, create, and make decisions across everyday and professional domains. While these technologies offer transformative potential, they also introduce novel sociotechnical harms, posing safety risks that differ from those in traditional safety-critical domains. Building on insights from the field of safety engineering, human-computer interaction, and science and technology studies, this thesis argues that such harms cannot be fully understood or mitigated through model-level interventions alone. They must be addressed at the system level by examining how AI systems are developed and integrated within broader sociotechnical contexts. System safety, as part of a new generation of safety engineering approaches, offers a valuable lens for understanding safety as an emergent property of complex sociotechnical systems. However, its approaches have yet to be meaningfully adapted to the context of AI, where new forms of complexity and risk demand both translation and expansion. This thesis addresses that gap by operationalizing system safety for AI, developing actionable approaches for responsible development, evaluation, and use of AI systems across their lifecycle.

Using a mixed-methods approach, this thesis examines three critical dimensions for establishing AI system safety practices. First, at the organizational and compliance level, we identify that existing practices for managing risks from AI systems are ad hoc and fragmented. To address this, we translate hazard analysis frameworks from system safety, particularly STPA, to support practitioners in identifying and mitigating potential hazards early in the AI system development process. We demonstrate that our adaptation of STPA offers key affordances for risk management by tracing how sociotechnical harms can emerge from the interactions among technical components, organizational processes, and institutional decision-making. Notably, mitigating these hazards and harms requires adequate evaluation and monitoring. Second, at the developer and evaluation level, we find that current measurements of ethical AI principles tend to focus on model-level metrics and a narrow subset of harms. Using a system safety lens, we trace links between these measures, the attributes they assess, and the types of hazards and harms they signal. This mapping clarifies which hazards and harms are addressed or overlooked, supporting the design of evaluation measures that serve as more intentional feedback mechanisms for system-level safety. Regardless of all mitigation strategies, AI systems will have erroneous and inappropriate outputs. To understand how users respond when this happens, the third dimension focuses on the user level. In a controlled experiment involving AI-based writing assistants, we investigate the relationship between different types of mental models and user control over AI systems. We find that while users with more in-depth mental models—those involving how the system works—find the tool easier to use, they do not necessarily demonstrate more effective control over the system, particularly when the AI generates erroneous suggestions. This highlights the importance of designing human-AI interactions that not only support user understanding but also enable users to translate that understanding into effective responses when failures arise.

Together, these contributions establish a foundation for operationalizing AI system safety across key points of intervention, hazard mapping, system evaluations, and end-user interaction, and offer a forward-looking lens for unpacking what safety will require as we move towards increasingly autonomous AI systems embedded in society.