Your AI-dar probably doesn't work

Harvard students use large language models widely, and some learn ways to bypass technical measures such as hidden text intended to flag AI-generated work. Instructors often stop reporting suspected AI use to the honor council because there is no reliable method to prove when AI was actually used. Some professors plan to respond by analyzing the tone and “vibes” of submissions and by requiring students to redo assignments when work appears AI-generated. One syllabus states that if a submission seems like AI work, the student must redo the entire assignment, with emphasis on producing better work in a unique voice. Students in that class must also submit Google Doc version history for analysis. Claims that AI writing has detectable tells, such as em dashes or balanced phrasing, may not hold up against manipulation of LLM outputs.

"“If your submission reads like it might be AI work, I'll have you redo the assignment in its entirety. I am uninterested in proving whether you did or did not use AI,” one professor's syllabus states, according to the Crimson. “I'll just ask for better work, in your unique voice, reflective of your unique interests; that's all.” (Just in case, students taking this class must submit their Google Doc version history, which may be analyzed, too)."

"Harvard students are already using LLMs widely, and some have learned to evade professors' more technical countermeasures, including hidden text meant to flag AI-generated work. Teachers have given up on reported suspected AI use to the school's honor council, because there's no real way to prove when AI was actually used. Some instructors even suggest they may push back on the technology's proliferation by simply analyzing the-well-vibes of student submission."

"These alleged AI proclivities lead plenty of people on the internet to say they simply know when something is written by AI. But perhaps these people don't know! The challenge for amateur AI investigators is in the long run, their detection methods might be doomed to fail. This is unfortunately obvious if you use these systems: it's relatively easy to coax an LLM to perform to meet various standards set out by a professor, whethe"

Read at Fast Company