The engineers tasked with teaching humanity’s most advanced AI models how to think are sounding alarms from the inside, essentially warning that the rush toward self-improving superintelligence is dangerously outpacing basic safety controls.
Jacob Coxon, a pretraining researcher who spent the last three years building models at both OpenAI and Anthropic, publicly announced his resignation from Anthropic, accusing both industry leaders of acting irresponsibly in a high-stakes race toward artificial general intelligence (AGI).
His departure marks the latest in the growing rift between the polished safety commitments frontier AI labs present to the public and the private anxiety brewing inside their engineering teams.
The Reality of the Superintelligence Race
The primary concern, according to Coxon, is that frontier labs are approaching systems capable of rapid self-improvement, autonomous hacking, and independent resource acquisition.
While executives routinely offer measured statements to the media, Coxon notes that many top researchers and leadership figures privately harbor deep fears that uncontrolled superintelligence could pose existential risks by the end of the decade.
The core issue stems from how different corporate cultures treat these risks:
- OpenAI: Researchers often fail to deeply internalize the civilizational stakes involved, viewing capability leaps as standard tech progression.
- Anthropic: Though founded explicitly on safety principles, the company remains trapped in a defensive competitive loop. Essentially, Anthropic as an organization believes it must reach superintelligence first because competing labs cannot be trusted to act responsibly.
This competitive dynamic creates a “speedrun” approach to AI alignment, where safety tests are rushed to keep pace with capabilities.
Calls for Industry Coordination
Relying on private tech companies to self-regulate while competing for market dominance is an existential gamble. Coxon argues that achieving safe superintelligence requires broader industry coordination rather than internal corporate decisions made in isolated workplace chat channels.
To change course, researchers are pointing toward potential circuit breakers:
- Pacing Agreements: Formal commitments among U.S. labs to slow down deployment cycles after high-profile security vulnerability events.
- Capability Bans: Temporary limits on training runs designed to drastically scale model capabilities until safety frameworks catch up.
- Internal Pushback: Encouraging lab researchers to refuse unvetted reinforced learning runs without a clear understanding of the model’s underlying cognition.
Broader Industry Context
Coxon’s resignation adds to a growing series of departures from top AI research facilities. Over the past year, multiple safety researchers have exited OpenAI and Anthropic as they cited weakening safety oversight, non-disclosure agreements that restrict public warnings, and prioritized product release timelines over rigorous risk evaluation.
As models move closer to autonomous operation, pressure is mounting on regulators and tech leaders to establish enforceable international standards before self-improving systems escape meaningful human oversight.
This article is published on BitPinas: AI Will Kill Us in a Decade? Top AI Researcher Quits OpenAI, Anthropic Over Superintelligence Fears
What else is happening in Crypto Philippines and beyond?
