Doctors' AI Systems Are Hallucinating Nonexistent Medical Issues During Appointments With Patients

"Earlier this week, Ontario's auditor general - an accountability officer acting under the Legislative Assembly of Ontario - released a special report warning that AI medical scribes were "not evaluated adequately," and may present "fabricated information" to medical professionals. First reported by Global News, the audit took a look at 20 AI scribe platforms and found that "all AI scribe systems from the 20 [government] approved vendors showed one or more inaccuracies at the procurement testing phase," such as "hallucinations (fabrication), incorrect information, or missing or incomplete information.""

""Inaccuracies in medical notes generated by AI Scribe systems could potentially result in inadequate or harmful treatment plans that may potentially impact patient health outcomes," the report declared. Muddying the waters, Ontario's Minister of Public and Business Service Delivery and Procurement, Stephen Crawford, noted that the hallucinations were observed during testing by state regulators, and had not been recording during actual medical visits."

""Let's be very clear about that, that's not actually in operational use with doctors, that's in the optional stage where we're reviewing the various scribes," Crawford told Global News. Still, the auditor general, Shelley Spence, noted that the various scribes are nonetheless in use by around 5,000 doctors across Ontario. Talking to reporters, Spence said she went so far as to ask her physician to "please look at the transcript when you're done with my own visit.""

AI scribe software listens to medical conversations, transcribes them, and formats them into medical notes. Ontario’s auditor general released a special report warning that AI medical scribes were not evaluated adequately and may produce fabricated information. The audit reviewed 20 AI scribe platforms from government-approved vendors and found inaccuracies during procurement testing, including hallucinations, incorrect information, and missing or incomplete details. Inaccuracies in generated medical notes could lead to inadequate or harmful treatment plans and affect patient health outcomes. Ontario’s procurement minister said hallucinations were observed during regulator testing and were not recorded during actual visits, describing the issue as part of an optional review stage. The auditor general said scribes are used by about 5,000 doctors and urged physicians to review transcripts after visits.

Read at Futurism