Fixing Hallucinations Would Destroy ChatGPT, Expert Finds

"They found that the way we evaluate the output of large language models, like the ones driving ChatGPT, means they're "optimized to be good test-takers" and that "guessing when uncertain improves test performance." In simple terms, the creators of AI incentivize them to guess rather than admit they don't know the answer - which might be a good strategy on an exam, but is outright dangerous when giving high-stakes advice about topics like medicine or law."

"While OpenAI claimed in an accompanying blog post that "there is a straightforward fix" - tweaking evaluations to "penalize confident errors more than you penalize uncertainty and give partial credit for appropriate expressions of uncertainty" - one expert is warning that the strategy could pose devastating business realities. In an essay for The Conversation, University of Sheffield lecturer and AI optimization expert Wei Xing argued that the AI industry wouldn't be economically incentivized to make these changes, as doing so could dramatically increase costs."

"Worse yet, having an AI repeatedly admit it can't answer a prompt with a sufficient degree of confidence could deter users, who love a confidently positioned answer, even if it's ultimately incorrect. Even if ChatGPT admitted that it doesn't know the answer just 30 percent of the time, users could quickly become frustrated and move on, Xing argued. "Users accustomed to receiving confident answers to virtually any question would likely abandon such systems rapidly," the researcher wrote."

Evaluation methods for large language models incentivize being good test-takers; guessing when uncertain improves measured test performance. Incentivizing confident guesses leads models to assert answers rather than express uncertainty, increasing hallucinations and risks in high-stakes domains like medicine and law. One proposed mitigation is to penalize confident errors more than uncertainty and to give partial credit for appropriate expressions of uncertainty. Economic and user-behavior barriers threaten adoption: such changes can substantially increase computational costs, and frequent admissions of uncertainty can frustrate users accustomed to confident answers, reducing product engagement. Established uncertainty-quantification methods exist but may require much more computation.

Read at Futurism