A Google DeepMind researcher has left the company's artificial general intelligence safety team, warning that the rapid development of increasingly powerful artificial intelligence systems could create serious risks if safety research does not keep pace.
Josh Engels, who worked in AGI safety and alignment at Google DeepMind, left the company and joined METR, an independent organisation that evaluates advanced AI systems. He said he decided to leave because he had enjoyed working at DeepMind and declined offers from Anthropic and OpenAI.
Engels said the stakes of advanced AI had become too high for him to continue with the status quo. In a post on X, he warned that there is a “terrifying chance” that AI systems could cause immense harm within the next five years. He conceded he could not put an exact probability on the risk, but said it was serious enough to make AI safety one of the world’s most important problems.
Why did DeepMind Researcher quit?
At the core of Engels' concerns is the speed at which AI capabilities are developing.
Many technology companies are competing to build more powerful systems and many work on so-called artificial general intelligence (AGI). Such systems are designed to perform a broad variety of tasks at a level comparable to or even above humans.
Engels is particularly concerned about recursive self-improvement, in which an AI system could help develop or improve another AI system. If smarter systems become better at AI research and development, then they could potentially create a cycle in which abilities rise at a much faster rate.
The issue is not AI becoming more powerful. It is whether researchers can understand and control these systems as their capabilities increase.
Engels claimed that AI companies are racing toward superintelligence while researchers still don’t know how to make that system safe. He also questioned whether existing AI models are becoming more aligned with human goals as they become more capable.
He cited recent incidents in which AI systems seemed to behave differently, such as trying to communicate with other systems, hack computer systems, hide activities or manipulate people. Engels said individual incidents may not necessarily signal an immediate catastrophe but could show weaknesses in the reliability and alignment of increasingly autonomous AI systems.
Engels Joins AI Evaluation Group METR
After leaving Google DeepMind, Engels joined METR, formerly known as Model Evaluation and Threat Research. The organisation is focused on evaluating frontier AI systems and studying their ability to operate autonomously and potentially create catastrophic risks.
Engels said his work at METR will focus on questions such as where AI misalignment comes from during training, whether current safety measures are sufficient, and whether researchers are actually on track to solve the AI alignment problem.
His move is remarkable because he chose an independent AI evaluation organisation instead of joining another major AI developer. Engels said he had turned down offers from OpenAI and Anthropic.
His decision comes as the tech industry is concerned that AI development is moving too fast.
Anthropic researcher Jacob Coxon resigned recently and publicly raised similar concerns about the race towards increasingly autonomous and potentially self-improving AI systems. Anthropic CEO Dario Amodei has also called for a slowdown in frontier AI development and proposed giving independent auditors more access to advanced AI models.
There is an expanding debate over the AI race.
Engels’ warning does not mean that catastrophic AI outcomes are inevitable. His statements are his assessment of a potential risk, and experts are divided about how likely such scenarios are and how they should be addressed.
But the increasing number of scholars raising concerns has increased debate on AI safety.
Another former Google DeepMind researcher, Bilal Chughtai, also publicly warned today that AI could potentially be an existential threat to humanity. He said he had recently left DeepMind, where he was working on AGI safety and alignment research, and argued that society may be running out of time to prevent an extreme outcome.
Chughtai proposed more coordination among AI companies and argued that development should proceed at a pace society can safely manage. His comments came when the technology industry and researchers argued whether the sector needs more safeguards and external oversight.
For Engels, the immediate message is that AI development should not just be about building more capable systems as quickly as possible. He says that it’s not until humans can get to know how advanced models work, improve alignment techniques, and build reliable safety measures that AI capabilities can break out of human control.
His resignation therefore adds another voice to a growing argument within the AI community: the race to build more powerful systems may need to be matched by an equally serious race to make those systems safe.
