How to give AI the ability to 'think' about its 'thinking'

"This process, becoming aware of something not working and then changing what you're doing, is the essence of metacognition, or thinking about thinking. It's your brain monitoring its own thinking, recognizing a problem, and controlling or adjusting your approach. In fact, metacognition is fundamental to human intelligence and, until recently, has been understudied in artificial intelligence systems. My colleagues Charles Courchaine, Hefei Qiu, Joshua Iacoboni, and I are working to change that."

"We've developed a mathematical framework designed to allow generative AI systems, specifically large language models like ChatGPT or Claude, to monitor and regulate their own internal "cognitive" processes. In some sense, you can think of it as giving generative AI an inner monologue, a way to assess its own confidence, detect confusion, and decide when to think harder about a problem."

"Today's generative AI systems are remarkably capable but fundamentally unaware. They generate responses without genuinely knowing how confident or confused their response might be, whether it contains conflicting information, or whether a problem deserves extra attention. This limitation becomes critical when generative AI's inability to recognize its own uncertainty can have serious consequences, particularly in high-stakes applications such as medical diagnosis, financial advice, and autonomous vehicle decision-making."

Metacognition is the process of becoming aware that a current strategy is failing and then changing that strategy. Human brains monitor their own thinking, recognize problems, and control or adjust approaches. A mathematical framework can enable large language models to monitor and regulate internal cognitive processes, providing an inner monologue that assesses confidence, detects confusion, and determines when to apply deeper reasoning. Generative AI currently produces confident outputs without reliable self-awareness, creating risks in medical diagnosis, financial advice, and autonomous vehicles. Self-monitoring capabilities allow systems to flag uncertainty, detect contradictions, and decide to pause, reflect, or seek additional information.

Read at Fast Company