Understanding the Need for AI Safety and Governance
In May 1972, President Richard Nixon and Soviet Premier Leonid Brezhnev signed the SALT I agreements, culminating years of negotiations focused on the mutual concern of nuclear weapon destruction. Even with the high stakes of nuclear annihilation motivating cooperation, neither side fully trusted the other’s assurances. Consequently, the agreements depended on “national technical means of verification,” which allowed both nations to monitor each other’s activities using their advanced technologies, including reconnaissance satellites.
This historical context of verification and cooperation is echoed in Dario Amodei’s recent essay, “We Must Pace the Frontier.” Amodei, co-founder of Anthropic, warns that the rapid advancement of artificial intelligence could outpace our ability to comprehend and control the technology, especially as AI begins to create its own successors. His proposed solutions include granting independent evaluators continuous access to cutting-edge labs to ensure safety protocols, establishing unified safety standards among leading AI organizations in democratic nations, and pursuing international agreements, particularly with China, regarding high-risk AI capabilities.
The Challenge of Collective Restraint
One significant challenge in implementing Amodei’s proposals is the inherent conflict between the competitive nature of AI development and the call for collective self-restraint among companies striving for significant economic gains. Achieving independent verification requires organizations to disclose more about their operations, whereas pacing development often contradicts their inherent incentive to innovate rapidly. This discrepancy highlights a deeper issue: the intrinsic drive of scientists and engineers to build and innovate. While regulations can impose boundaries on what is developed and its applications, asking the tech community to intentionally limit its progress is an uphill battle against fundamental economic forces and the historical drive for scientific advancement.
Efforts to coordinate on safety and regulation face additional hurdles at the governmental level. Stakeholders will inevitably debate which entities—whether the United States, China, Europe, or international bodies—should determine the appropriate pace of AI development. Moreover, the absence of affected parties from the decision-making process deepens the complexity. Once agreements are reached, questions remain: Who decides what constitutes a boundary violation, and what consequences follow if a nation decides that the benefits of rapid AI advancement outweigh the risks of breaching those agreements?
Industry Division Over AI Safety Proposals
Amodei’s essay swiftly ignited a division within the AI industry, crystallizing into three distinct camps. The first camp quickly embraced the proposal; for instance, Sam Altman of OpenAI committed to aligning with Anthropic’s pledge for independent evaluators shortly after its publication. Other notable endorsements came from Elon Musk and Demis Hassabis of Google DeepMind, both viewing the essay as a significant step toward a shared path forward.
Conversely, the second camp acknowledged the concerns raised but rejected the idea of a coordinated slowdown. Meta CEO Mark Zuckerberg argued that competitive pressures and legal liabilities already incentivize safe development practices without necessitating external constraints. Investor David Sacks provocatively urged any company advocating for a slower pace to voluntarily decelerate its innovations.
The third camp raises further skepticism towards the motives behind the push for caution, with investor Michael Burry categorizing the calls for a slowdown as self-serving, primarily beneficial to incumbent players in the market who face rising competition.
The Complexity of AI Governance
Amidst this division, it’s crucial to recognize that even if all parties sought to cooperate sincerely, the rapidity of the dissent emphasizes a key lesson: while coordination may be beneficial, it cannot serve as the cornerstone of a robust safety strategy. The aviation industry often serves as a model for effectively managing complex technologies. However, even as regulatory frameworks become more sophisticated, perils still arise, as illustrated by the Boeing 737 MAX incidents in which established procedures failed to prevent tragic outcomes due to deeply rooted issues in oversight and compliance.
AI introduces unique challenges that complicated traditional governance structures cannot address. These systems can respond dynamically to varied contexts, continually evolving in capabilities while expanding their tools and interactions. Rapidly evolving agentic AI systems outpace human evaluators, who struggle to make sense of their implications after actions have occurred.
The Necessity for Machine-Speed Governance Solutions
The implications shed light on the urgent need for governance frameworks that can operate as swiftly as AI technologies themselves. Notably, Deloitte’s 2026 survey revealed that only 21% of organizations report having mature governance structures for agentic AI, with IBM highlighting a significant gap between the speed of technology deployment and organizational readiness.
Human oversight alone cannot sustain governance at these speeds. Successful governance will require a multifaceted approach, where human-defined parameters coexist with AI systems capable of monitoring, assessing, and enforcing compliance in real time. Independent institutions must also play a vital role in verifying adherence to regulations without being entangled in the interests of those producing the technologies.
Incentivizing Accountability in AI Development
The ongoing global race for increasingly capable AI systems reflects significant investment and talent focused on enhancing model efficiency and utility. Consequently, the governance of such systems must match this ambition with equal rigor—developing robust evaluation methodologies, independent verification processes, continuous oversight, and enforceable policy frameworks capable of adapting to advancements in capabilities.
To recalibrate the skewed incentives surrounding AI development, collaborative governance efforts cannot be relegated to post-deployment compliance obligations. Instead, they must become integral to the architecture of technology development from the outset, treating governance as an engineering discipline rather than an afterthought.
The gap between AI capabilities and our capacity for oversight will not close by mere goodwill. Current conditions require governance structures that can effectively operate alongside innovative technologies, ensuring that all stakeholders remain accountable for their creations while minimizing the risks associated with their deployment.
As we explore pathways to mitigate AI risks, it’s crucial to recognize that while pacing may provide necessary breathing space, it must not become a substitute for a comprehensive governance strategy capable of keeping pace with the accelerating domain of AI technology.
Emre Kazim is the co-founder and co-CEO of Holistic AI. He contributed this article to SiliconANGLE.