Anthropic Pushes Courts and Regulators to Halt High-Risk AI Models

Anthropic chief executive Dario Amodei is pressing governments to impose binding safeguards on advanced artificial intelligence, arguing that voluntary promises cannot keep pace with increasingly capable systems. In an October 15 letter to California Governor Gavin Newsom, disclosed October 28, the company backed court orders that could halt models judged to pose a serious public risk.
- Anthropic sent California its recommendations in an October 15 letter, Reuters reported on October 28.
- Reuters said the proposal would let officials seek court orders blocking models that create a “serious risk.”
- Anthropic plans to spend at least $1 billion on independent evaluation over five years, Reuters Breakingviews reported on November 2.
What regulation is Anthropic asking for?
Anthropic wants governments to move beyond disclosure rules and voluntary industry commitments. The company’s proposal would allow public authorities to ask courts to stop the release of an AI model after a review finds a serious threat. The approach would give developers a chance to challenge government action while creating a legal checkpoint before deployment.
“When a review finds a model poses a serious risk, the government should be able to ask a court to block it,” Anthropic said in the letter to Newsom, as quoted by Reuters.
Anthropic submitted the recommendations after Newsom convened an expert panel in September to develop AI safety proposals. The company’s position is narrower than a general pause on AI development. It focuses on highly capable models that could create acute risks, including cyber or other high-impact misuse.
Why does Anthropic reject broad “kill switches”?
Anthropic does not support requiring companies to install controls capable of shutting down an entire model in every crisis. The company told California officials that such a requirement could reassure regulators without reliably containing a harmful system. Anthropic instead favors interventions aimed at the specific source of harm.
- Anthropic described full-model “kill switches” as a possible “false sense of reassurance,” Reuters reported.
- The company supports disabling a problematic AI agent or cutting off an individual customer’s access.
- Anthropic says targeted controls could work better than shutting down an entire model.
The distinction matters because advanced models can serve many customers and applications at once. A problem tied to one deployed agent or user may not justify removing access for every legitimate customer. Anthropic’s proposal therefore places more emphasis on identifying the dangerous action and stopping it directly.
How would independent evaluators change AI oversight?
Amodei’s regulatory argument is paired with a plan to give outside specialists a permanent role inside the AI-development process. Independent evaluators would work alongside Anthropic’s own safety and strategy teams, rather than reviewing a model only after its public launch. The goal is to test capabilities while developers still have time to change deployment plans.
Reuters Breakingviews reported on November 2 that Anthropic expects to invest at least $1 billion in this effort over the next five years. The company also has a responsible-scaling policy that places limits on development when safety work does not keep pace with model capabilities.
The proposed oversight would add an external check to a sector where companies often evaluate their own systems. It could also give regulators more technical information before deciding whether a model should be released.
What events pushed the debate toward courts and government?
Anthropic’s recommendations arrived amid a dispute over how governments should respond when frontier models acquire powerful capabilities. Reuters reported that the Trump administration ordered Anthropic in June to restrict foreign access to its cyber-capable Mythos 5 and Fable 5 systems, citing national-security concerns. The Commerce Department later lifted that order.
California’s process reflects a broader effort by governments to define enforceable standards rather than rely only on corporate pledges. Reuters reported that a bipartisan group of U.S. senators was debating legislation in September that would give the federal government power to block certain unsafe AI models, while allowing companies to challenge decisions in federal court.
Anthropic’s court-based proposal would place judges between regulators and developers. That could make restrictions more legally durable, but it would also raise questions about evidence, timing and the technical threshold for “serious risk.”
What does Amodei want the industry and governments to do next?
Amodei has argued that AI companies should not decide safety standards alone. A November 1 interview cited by Blumify described his preferred sequence: companies should first improve their own practices, then establish shared industry standards, followed by international cooperation. Government participation would be needed to make those standards credible and to address the pace of releases.
“There needs to be involvement of the government,” Amodei said in the interview, referring to efforts to establish safety standards and potentially regulate the pace at which systems are released, as reported by Blumify.
Amodei also said Anthropic was speaking regularly with officials in the Commerce Department, Treasury Department and intelligence community. The company’s policy push now faces practical tests: lawmakers must define which models qualify as dangerous, agencies must develop credible evaluations, and courts must decide how quickly an injunction could be issued when a model is nearing release.
Who could be affected by stronger AI rules?
AI developers would face new testing, documentation and legal obligations before releasing the most capable systems. Cloud providers and customers could also encounter access limits if a model or a particular agent is judged unsafe. Regulators would gain a formal route to intervene, while courts would need to assess technical evidence that can change quickly.
Businesses using advanced AI could see slower product launches or tighter controls around cyber, biological and autonomous capabilities. At the same time, outside evaluation could make it easier for customers to judge whether a system has been tested independently. The cost would fall first on leading developers, but the rules could eventually shape the wider market.
Anthropic’s position leaves unresolved how regulators should measure risk and who should pay for evaluations. Those decisions will determine whether stronger oversight becomes a targeted safety regime or a broader barrier to deployment.


