Why an AI ‘Kill Switch’ isn’t a Catch-all Solution
Recent cybersecurity events involving AI agents, such as the Hugging Face-OpenAI incident, have prompted policymakers to consider whether an AI “kill switch” is necessary to prevent increasingly capable systems from causing significant, imminent harm. But the phrase suggests a simplicity that does not exist. A “kill switch” could mean anything from revoking an agent’s permissions to suspending a model or disabling the infrastructure supporting a service. Before policymakers mandate such mechanisms, they should ask a more fundamental question: What risks would a shutdown requirement actually address and what new risks might it create?
For instance:
- Emergency safeguards could consist of different mechanisms that vary in scope and effect. Safeguards might shut down an individual user’s access, revoke an agent’s permissions, suspend a particular model, service, or deployment, or disable the infrastructure that hosts a model or through which a service is provided. Implementing these controls would combine technical, operational, and administrative mechanisms. Who controls activation is a separate consideration, as a provider may act independently or in response to a government order, or another party may activate the mechanism. How the safeguard is designed and which entity has the ability to invoke it have significantly varied implications for security and service continuity.
- Emergency safeguards may not halt all activities. For example, suspending a particular model does not automatically mean that delegated tasks already initiated will cease. It may not disable copies of the model that are downloaded and run independently.
- Effective implementation of an emergency safeguard depends on timing. An emergency intervention requires a problem to be detected or anticipated, and action to be taken quickly enough to limit harm. In many cases, even swift activation of a safeguard could result in harm if no other risk management measures are in place.
- Introducing emergency safeguard mechanisms can create new risks. A broad suspension could interrupt services for users who have no connection to the activity that prompted it. Businesses relying on an affected service could experience disruption across their own operations. Further, building in the technical capability to remotely disable a product or service could create new pathways for adversarial interference, or otherwise compromise security and resilience. Policy interventions that prescribe specific technical implementations may become less suitable as models and deployments evolve, introducing vulnerabilities and preventing products and services from evolving.
- Whether emergency shutdown requirements would address gaps in existing cybersecurity controls. Given the operational and business risks that a shutdown mechanism may introduce, policymakers considering new requirements should first determine the specific harms they seek to prevent or mitigate and where existing requirements and practices fall short. Proposals for new requirements should assess costs and the potential introduction of disproportionate risk. Policymakers should also understand that conflicting obligations across State or international lines exacerbate these challenges. Where requirements would have extraterritorial impact, customers may become concerned about service continuity, and U.S. products and services may become less trusted and less competitive.
- Emergency safeguards should be viewed as one part of a comprehensive approach to AI risk management. AI developers, integrators, and deployers should maintain effective, proportionate, and secure controls to address misuse and unsafe behavior by AI models and agents. The ability to interrupt or deactivate a system can be an important safeguard, but its effectiveness depends on complementary measures and controls, such as testing, access restrictions, limits on AI agents’ permissions, ongoing monitoring, and incident response and recovery planning. Many of these practices are outlined in existing risk management frameworks and standards, such as the NIST AI Risk Management Framework and ISO/IEC 42001. The availability of a shutdown capability alone is insufficient to assess a system’s overall safety.
Emergency safeguards can play an important role in AI risk management, and in many instances are already factored into established risk management programs. But, they are only one part of broader approach. Before imposing a standalone shutdown mandate, policymakers should identify the gap it would fill, weigh the new risks it could create, and determine who is best positioned to act. AI risk management cannot hinge on a single switch; it requires the implementation of a variety of measures and controls.