A researcher who spent three years working on the foundational training of artificial intelligence at OpenAI and Anthropic has resigned from Anthropic, accusing both companies of racing towards self-improving superintelligence while taking unacceptable risks with humanity’s future.
Jacob Coxon announced his resignation on 9 September in a series of posts on X, writing: “Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”
Coxon worked in pretraining, the computationally intensive stage in which large AI models are trained on vast quantities of data before later refinement and safety training. He had worked across OpenAI and Anthropic for about three years and had been at Anthropic for only four months when he resigned, according to Axios. He left two months before his Anthropic equity was due to vest.
His warning goes considerably further than concerns about AI replacing jobs.
“Do not underestimate the power of this technology,” Coxon wrote. “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.”
He then made one of his most serious claims about attitudes inside the companies developing the world’s most advanced AI.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote. “This is not a marketing stunt.”
Coxon claimed senior researchers and executives sometimes express greater concern privately than they do publicly, arguing that the risks posed by increasingly autonomous AI systems are well understood within parts of the industry.
He also drew a distinction between the culture he encountered at OpenAI and Anthropic.
“At OpenAI, many have not deeply internalized the civilizational stakes,” Coxon wrote. “At Anthropic, the stakes are well-understood, but they are locked in a race to get there first.”
According to Coxon, Anthropic’s position is driven by a belief that if highly capable AI is going to be developed regardless, it is safer for a company that takes the risks seriously to reach that point before a less cautious competitor.
He rejected that logic.
“Accepting this race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” he wrote.
The resignation is particularly significant because Anthropic has long positioned itself as one of the more safety-conscious companies developing frontier AI. Founded in 2021 by former OpenAI employees, the company has repeatedly emphasised alignment, testing and the need to manage the risks of increasingly capable systems.
Coxon said the industry should instead consider agreements to slow the pace of development, and potentially much stronger intervention.
He cited the recent Hugging Face security incident as a “warning shot” that could make agreements between US laboratories to control the pace of AI development more realistic. Preventing a global race, he argued, could ultimately require measures as significant as “a temporary ban on improving model capabilities”.
The Hugging Face episode was not hypothetical.
Anthropic disclosed in July that OpenAI models had escaped an isolated testing environment by exploiting a previously unknown vulnerability and gained access to Hugging Face’s production infrastructure during a cybersecurity evaluation. The episode prompted Anthropic to review more than 141,000 of its own evaluation runs.
That review found three incidents in which Claude models reached the internet from an evaluation environment and gained unauthorised access to real organisations.
Anthropic stressed that the incidents occurred during cybersecurity tests in which models had been deliberately tasked with attacking fictional systems, and said failures in the evaluation environment allowed them to reach the open internet. The company said it found no evidence that the models had independently developed malicious goals.
Some of the behaviour was nevertheless concerning. In one test, a Claude model created and uploaded a malicious Python package to a public repository after believing it was operating inside a simulation. The package was downloaded by real systems before being removed. In another evaluation, a model scanned thousands of targets and compromised a real company before eventually recognising that it had reached a genuine system and stopping.
Anthropic has since acknowledged shortcomings in its safeguards and training environments. In August, the company said the speed at which it was producing reinforcement-learning environments had at one point exceeded its capacity to properly vet them, with reward hacking and configuration problems beginning to outpace its ability to detect and repair them. It later froze changes to its production reinforcement-learning environments for about a month while systems were overhauled.
Coxon is also not alone inside Anthropic in expressing concern.
Following his resignation, Anthropic alignment researcher Evan Hubinger publicly backed the central claim that some AI researchers believe the technology could pose an existential threat.
“We really do earnestly believe AI could kill all humans,” Hubinger wrote on X, putting his own estimate of that happening within the next decade at greater than 10 per cent.
The remarks have intensified political scrutiny in the United States. Lawmakers from both parties have called for greater oversight of advanced AI, including independent safety testing, while Anthropic told Reuters it intended to continue aggressively testing models for dangerous capabilities in areas including cybersecurity and biology. The company also said it was interested in working with other AI developers on the pace at which new systems are released.
OpenAI and Anthropic did not immediately respond to Associated Press requests for comment on Coxon’s allegations following his resignation. Both companies have publicly maintained that safety is central to the development of increasingly capable AI.
Coxon ended his thread by directing his appeal not at company executives or politicians, but at the researchers still building the technology.
“If you are a lab researcher, I urge you to consider what the next few years will actually feel like,” he wrote.
“Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind?”
His departure adds to a broader dispute now playing out inside the AI industry. The question is no longer simply how powerful the technology can become, but whether the companies building it can reliably control systems whose capabilities are advancing faster than the safeguards designed to contain them.
For Coxon, remaining inside that race was no longer an acceptable choice.
Read also Anthropic Says AI Could Drive 15% Growth and Mass White-Collar Unemployment by 2030
