Anthropic’s own AI is already writing 80% of the company’s code. In a couple of years, it could be 100%. That is the problem. And that is exactly why Anthropic co-founder Jack Clark says the industry needs a brake pedal — before it is too late to build one.
Anthropic’s warning about recursive AI self-improvement came — the same week the company filed confidentially for its IPO and raised $65 billion. The timing is not accidental. Anthropic co-founder Jack Clark and Marina Favaro, leader of The Anthropic Institute, published a blog post warning the global AI industry that it is rapidly approaching a point of no return. Their core claim is specific. AI systems are becoming capable of improving themselves — a process called “full recursive self-improvement” — without meaningful human involvement. Once that threshold is crossed, the mechanisms humans currently use to monitor, align, and secure AI systems become significantly harder to apply. The industry, Clark told CNN and the BBC, currently has a gas pedal and no brake pedal. Anthropic wants to help build one.
What’s Happening & Why It Matters
What Recursive Self-Improvement Actually Means
Anthropic’s warning about recursive AI self-improvement centres on a specific technical threshold. Human researchers and engineering teams train today’s AI models. They improve because humans design better training regimes, larger datasets, and more sophisticated architectures. By contrast, recursive self-improvement describes a regime in which AI agents — the autonomous task-executing systems that frontier models increasingly deploy — become capable of designing and training their successors. In this scenario, Claude could continuously improve. The human engineering team becomes a supervisor of a system that is improving at rates and in directions that humans can no longer fully predict or validate.
That threshold is not a distant speculation. Anthropic provided direct evidence of its proximity. Jack Clark told the BBC that its AI already performs 80% of Anthropic’s own coding work. He said that the proportion could reach 100% within a couple of years. At that point, the company’s AI would be autonomously producing the code that trains and improves its own successors. “It’s a choice whether AI companies let it get that far without stopping it,” Clark said. “We think this is a topic that the world should be talking more about.”
The Gas Pedal Without a Brake Pedal
Clark used a specific and memorable metaphor in his media appearances. “The AI industry right now has a gas pedal, but it doesn’t have a brake pedal in the car, and we want to do some of the work to build that pedal.” That framing captures the structural problem precisely. The competitive dynamics of the AI industry — enormous capital investment, national security pressures, first-mover commercial advantages — all push toward acceleration. There is no structural mechanism that allows any single company, government, or international body to pause or slow AI development when a threshold of concern is reached.
By contrast, Clark noted that such mechanisms have been built in other domains before. “In the height of the Cold War, under highly tense situations between rivalrous countries, they found ways to stabilise aspects of the nuclear arms race,” he told CNN. “All of this has been done before in other domains, and it may need to be something we do in the domain of AI.” The nuclear comparison is neither new nor alarmist. It is the most historically accurate analogy available. Two highly competitive superpowers — each with the capacity to destroy the other — agreed to arms control regimes because the alternative was worse for both. Anthropic is making the same argument at an earlier stage of a comparable trajectory.
The Evidence Within Anthropic’s Own Systems
The blog post, written by Clark and Favaro, provided internal data that gives the warning specific credibility. Anthropic staff are producing eight times as much code as they were between 2021 and 2025 — because AI is handling the bulk of the actual coding work. That eightfold increase in productivity is simultaneously a commercial advantage and a structural signal. The more AI takes over the engineering process, the more the human team transitions from builders to supervisors. And the further supervision recedes from the actual technical work, the harder it becomes to validate what the AI is producing. Favaro and Clark wrote directly. “If systems are capable of fully building their own successors, the ways we secure them, monitor them and shape their behaviour all grow much more important.”
The Risks: Cyberattacks, Economic Disruption, and Control Loss
Anthropic’s warning on recursive AI self-improvement identifies three specific risk categories. The first is catastrophic digital risk — specifically, the weaponisation of AI vulnerability discovery. Anthropic has already withheld Mythos from public release, specifically due to cyberattack risks. As TF covered in its Mythos Project Glasswing expansion article, the company is instead deploying it under strict controlled access. A recursively self-improving AI that can also discover and exploit security vulnerabilities — operating without human oversight of its improvement process — is the scenario that makes the brake pedal argument urgent.
The second risk is macroeconomic disruption. Anthropic warned explicitly that autonomous AI agents are beginning to overtake routine employment — and that recent mass tech layoffs are already connected to AI handling software engineering tasks. By contrast, the company acknowledged the same technology could accelerate breakthroughs in science and healthcare. Both things are simultaneously true. The argument about the brake pedal is not anti-AI. It is pro-managed AI. The third risk is loss of control — the most fundamental concern. “How do you maintain control over fleets of scientists that are much, much larger and much faster than ones you’ve had before?” Clark asked CNN. He acknowledged the science fiction dimension directly. “Yeah, we read science fiction and watch science fiction here as well, so it’s not lost on us.”
The Tension: A Company Selling the Gas Pedal Wants a Brake
The most intellectually uncomfortable element of Anthropic‘s warning is structural. The company is simultaneously filing for an IPO targeting a $965 billion valuation, raising $65 billion in growth capital, and publishing a warning that the AI industry needs to slow down. Dario Amodei has previously estimated a 25% chance that AI’s trajectory goes “really, really badly.” Those two positions — accelerating and warning simultaneously — are not necessarily hypocritical. They reflect a genuine dilemma. A company that unilaterally slows down while competitors do not simply loses. A company that calls for collective action while continuing to invest is attempting to change the system from within while it still has leverage. Both things are true. Anthropic offered to suspend its own work on advanced frontier systems as part of a coordinated agreement. That offer is only credible if competitors participate. Getting competitors to participate requires the kind of public pressure this blog post is designed to generate.
What a Brake Pedal Would Actually Look Like
Anthropic did not specify technical or policy details of what a “brake pedal” mechanism would involve. By contrast, several frameworks already exist that the company may be drawing on. The International Atomic Energy Agency (IAEA) model — international inspection and monitoring of high-risk facilities — provides one template. The Asilomar Conference on Recombinant DNA in 1975 — where scientists voluntarily paused research until safety guidelines were established — provides another. The NIST AI Risk Management Framework provides a domestic US starting point. As TF covered in its Trump AI executive order article, the US government signed a voluntary 30-day review mechanism this week that represents the most minimal version of what Anthropic is calling for. Clark’s Cold War analogy suggests the company envisions something considerably more robust.
TF Summary: What’s Next
Anthropic‘s blog post has prompted immediate commentary across the AI industry. Neither OpenAI, Google DeepMind, Meta AI, nor Elon Musk‘s xAI has responded publicly as of Friday evening. Jack Clark is expected to continue media appearances advocating for the brake pedal framework. The Anthropic Institute will hold its first major convening later in 2026 — an event likely to further develop the policy framework behind the blog post.
MY FORECAST: Anthropic’s recursive AI self-improvement warning will prove to be the most important public statement any AI company has made in 2026 — not because it immediately changes behaviour, but because it establishes the terms of the debate that is coming. Clark is correct that recursive self-improvement is approaching. He is also correct that the industry lacks a mechanism to pause or slow when it arrives. The brake pedal will not be built voluntarily by the AI industry alone — competitive pressure makes unilateral slowdown commercially suicidal. It will be built through a combination of regulation, international agreement, and selective demonstrated restraint. Anthropic has positioned itself as the company that identified the problem and proposed the solution — before the problem became a crisis. Whether that positioning produces a brake pedal or simply a compelling IPO narrative about responsible AI development will be determined over the next 12 to 24 months after this article is published.