Geoffrey Hinton’s ‘Kindergarten’ Warning: Could Superintelligent AI Outsmart and Manipulate Humans?

Geoffrey Hinton, one of the pioneers of artificial intelligence, describes in an easy but striking analogy one of the big questions about the future of AI: what would happen if machines become much more intelligent than humans?

Geoffrey Hinton | Photo Credit: https://x.com/alvinfoo
Geoffrey Hinton | Photo Credit: https://x.com/alvinfoo

Hinton compares the potential relationship between humans and a superintelligent AI to the relationship between adults and kindergarten children.

The comparison shows how a major difference in intelligence could make it difficult for the less intelligent side to understand what the more intelligent side is actually doing.

Children may know that adults are influencing their decisions, but they often do not yet have the knowledge and experience to fully understand the reasons behind those decisions.

And Hinton says that a similar problem could arise if AI systems are able to develop better than humans eventually.

The concern is not that an advanced AI would attack or threaten people, after all. The danger could be of little influence that humans may not be able to see in advance.

A good enough system could understand human behavior, incentives and weaknesses to the point that its strategies are difficult to detect.

The key question is how to control. If humans cannot fully understand how a very intelligent AI makes decisions, it will become increasingly difficult for the system to follow human intentions.

The analogy also emphasizes the limits of human oversight. Most AI systems today are designed, trained and evaluated by people. Researchers can test their behavior, analyze their outputs and put some safeguards.

But a future system with much more reasoning power can behave in ways that are not anticipated by current methods.

Hinton’s warning comes at a time when we are seeing more and more discussions about developing AI models that are more and more capable.

Systems and organizations are investing in AI systems that are able to reason, write software to analyze data and do very complex things. And when these abilities become better, our questions about AI safety and alignment also come into play.

The kindergarten analogy is particularly powerful because it does not require imagining a science-fiction scenario. It focuses on a familiar situation: when one party has substantially more knowledge and understanding than another.

For Hinton, the most significant worry is that humans could find themselves in the weaker position without immediately realizing it.

If a superintelligent system could influence people through strategies that humans are unable to recognize, traditional methods of supervision might be insufficient.

The discussion is not that superintelligent AI will manipulate humanity as much as human beings do. It's just a warning to us of a worse kind of future that could happen and we need to have some safety measures in place before AI can achieve that level of capabilities.

As artificial intelligence develops, Hinton's analogy also highlights the point about this more general problem: humanity may need to solve the problem of understanding and controlling systems that may eventually be much more capable than their creators.

The debate over superintelligence is no longer limited to what AI can do today.

Increasingly, researchers are also wondering how humans can remain in control if machines one day become vastly more intelligent.