"If you ask AI to figure out what ails you based on inputting a series of symptoms, the AI will use mathematical probability to calculate the appropriate sequence of words that would generate the most valuable output given the specific prompt. The AI has no intrinsic or learned understanding of what "body," "illness," "pain," or "disease" mean. Such practically meaningful concepts to humans are, to the bot, just letters encountered in the training set frequently paired with other letters."
"Recently, a team of researchers set out to investigate whether AIs that achieved near-perfect accuracy on medical benchmarks like MedQA actually reasoned through medical problems or simply exploited statistical patterns in their training data. If doctors and patients more widely rely on AI tools for diagnosis, it becomes critical to understand the capability of AI when faced with novel clinical scenarios."
"The researchers took 100 questions from MedQA, a standard dataset of multiple-choice medical questions collected from professional medical board exams, and replaced the original correct answer choice with "None of the other answers." If the AI was simply pattern-matching to its training data, the change should prove devastating to its accuracy. On the other hand, if there was reasoning behind its answers the negative effect should be minimal."
Many people now turn to AI for medical diagnosis, but AI generates responses by calculating probabilistic sequences of words rather than by understanding medical concepts. AI lacks intrinsic or learned comprehension of terms like "body," "illness," "pain," or "disease," treating them as letter patterns from training data. Controlled tests that alter familiar answer choices in medical benchmarks such as MedQA severely reduce AI accuracy, demonstrating dependence on statistical patterns instead of reasoning. Reliance on AI tools for diagnosis therefore requires caution, especially when encountering novel or atypical clinical cases outside the training distribution.
Read at Psychology Today
Unable to calculate read time
Collection
[
|
...
]