As artificial intelligence (AI) systems become increasingly embedded in essential infrastructure and services, the risks associated with unintended failures rise. Developing comprehensive emergency response protocols could help mitigate these significant risks. This report focuses on understanding and addressing AI loss of control (LOC) scenarios where human oversight fails to adequately constrain an autonomous, general-purpose AI.

Strengthening Emergency Preparedness and Response for AI Loss of Control Incidents

Elika Somani, Anjay Friedman, Henry Wu, Marianne Lu, Christopher Byrd, Henri van Soest, Sana Zakaria

ResearchPublished Jul 30, 2025

As artificial intelligence (AI) systems become increasingly embedded in essential infrastructure and services, the risks associated with unintended failures rise. Future critical failures from advanced AI models could trigger widespread disruptions across essential services and infrastructure networks, potentially amplifying existing vulnerabilities in other domains. Developing comprehensive emergency response protocols could help mitigate these significant risks. This report focuses on understanding and addressing a specific class of such risks: AI loss of control (LOC) scenarios, defined as situations where human oversight fails to adequately constrain an autonomous, general-purpose AI, leading to unintended and potentially catastrophic consequences.

Key Findings

LOC risks are increasingly plausible and remain unaddressed

  • Researchers have identified warning signs of control-undermining capabilities in advanced AI models – including deception, self-preservation and autonomous replication – which could potentially enable increasingly capable models to evade human oversight.

Detection and early warning challenges

  • Governments and other stakeholders lack a common framework to analyse and respond to LOC risks. There is no clear consensus on which AI capabilities could lead to LOC, how safeguards may interact with such capabilities, or the best warning signs of LOC risks. This fragmented understanding hampers the ability of model developers or governments to detect early LOC warnings. Furthermore, current detection methods rely on pre-deployment model evaluations and ongoing monitoring by AI developers, with limited validation by independent third-party evaluators. However, models may operate differently in testing environments, potentially interacting with deployment contexts in unexpected ways. Open-source models present challenges to detection given the potential for unmonitored access and modifications to the model with limited oversight.

Escalation gaps

  • Safety frameworks published by industry have yet to align on a consistent approach to risk escalation. Importantly, there are no clear thresholds for when a LOC incident should trigger an emergency response.

Containment and mitigation limitations

  • Containing a LOC event requires advances in technical AI safety. Traditional cybersecurity safeguards such as endpoint detection, firewalls and malware detection are essential but may be insufficient. Containment measures may be ineffective if AI systems gain significant control over resources before risks are detected.

Recommendations

Detection of LOC threats

  • Governments, with AI developers and other stakeholders, should establish a clear, shared definition of AI LOC and a set of criteria for detection.
  • AI developers and researchers should refine detection by developing standardised benchmarks and improving their reliability and validity.
  • Governments should enhance awareness and information sharing between all stakeholders, including the tracking of compute resources.

Actions for escalation

  • AI developers should establish well-defined escalation protocols and conduct regular training exercises to ensure their effectiveness.
  • Government stakeholders should consider mandatory reporting mechanisms for AI risks and potential incidents.
  • Government stakeholders should establish disclosure channels and whistleblower safeguards for employees of AI developers.
  • AI developers, AISIs and relevant government departments should enhance cross-sector and international coordination.

Actions for containment and mitigation

  • AI developers should prepare containment measures that are rapid and flexible.
  • AI developers and other stakeholders should further explore and advance research on containment methods.
  • AI developers, external researchers and AISIs should prioritise safety and alignment measures, including by building validated safety cases.
  • Government stakeholders should seek to strengthen AI security to protect model weights and algorithmic techniques.
  • Governments and developers should improve safety governance by fostering robust safety cultures and adopting secure-by-design principles.

Topics

Document Details

  • Copyright: Crown Commercial Services
  • Publisher: RAND Corporation
  • Availability: Web-Only
  • Year: 2025
  • Pages: 61
  • DOI: https://doi.org/10.7249/RRA3847-1
  • Document Number: RR-A3847-1

Citation

Chicago Manual of Style

Somani, Elika, Anjay Friedman, Henry Wu, Marianne Lu, Christopher Byrd, Henri van Soest, and Sana Zakaria, Strengthening Emergency Preparedness and Response for AI Loss of Control Incidents. Santa Monica, CA: RAND Corporation, 2025. https://www.rand.org/pubs/research_reports/RRA3847-1.html.
BibTeX RIS

Research conducted by

This publication is part of the RAND research report series. Research reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND research reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.