Sociotechnical Sources of Risk and Resilience in Autonomous and Intelligent Systems: Exploring Safety in the Implementation of Clinical Artificial Intelligence

Carl Macrae (University of Nottingham), Sociotechnical Sources of Risk and Resilience in Autonomous and Intelligent Systems: Exploring Safety in the Implementation of Clinical Artificial Intelligence (2023):

Autonomous and intelligent systems (AIS) are being developed and deployed across a wide range of sectors and encompass a variety of technologies designed to engage in different forms of independent reasoning and self-directed behaviour. AIS may bring considerable benefits to society but implementing these technologies poses a range of risk management challenges, particularly in safety-critical sectors where complex interactions between human, social and technical processes underpin safety and resilience. Healthcare is one safety-critical sector that is at the forefront of efforts to implement AIS, primarily through clinical artificial intelligence systems intended to support or replace key aspects of clinical work such as reading medical images to identify signs of pathology and make referral decisions. This paper develops a detailed qualitative analysis of the sociotechnical sources of risk and resilience associated with implementing clinical artificial intelligence in healthcare. The analysis draws on 40 in-depth interviews with participants actively involved in the implementation and regulation of clinical artificial intelligence. Qualitative template analysis is used to examine the sociotechnical sources of risk and resilience in the specific setting of healthcare, drawing on and elaborating Macrae's (2022) SOTEC framework that integrates structural, organisational, technological, epistemic and cultural sources of sociotechnical risk in AIS. This analysis characterises and contextualises the sociotechnical sources of risk associated with implementing AIS in healthcare and identifies an array of sociotechnical patterns of resilience that may counter those risks. Through this analysis, the paper extends the SOTEC framework to encompass new patterns of sociotechnical risk and explores the duality of risk and resilience in the management of safety.