Learning With AI Falls Short Compared to Old-Fashioned Web Search

"Most of the studies used the same basic paradigm: Participants were asked to learn about a topic such as how to grow a vegetable garden and were randomly assigned to do so by using either an LLM like ChatGPT or the old-fashioned way, by navigating links using a standard Google search. No restrictions were put on how they used the tools; they could search on Google as long as they wanted and could continue to prompt ChatGPT if they felt they wanted more information."

"The data revealed a consistent pattern: People who learned about a topic through an LLM versus web search felt that they learned less, invested less effort in subsequently writing their advice, and ultimately wrote advice that was shorter, less factual and more generic. In turn, when this advice was presented to an independent sample of readers, who were unaware of which tool had been used to learn about the topic, they found the advice to be less informative, less helpful."

Seven experimental studies with more than 10,000 participants compared learning via large language model summaries to learning via standard web search. Participants were assigned to learn about everyday topics by either interacting with an LLM or by navigating search links, with no restrictions on tool use. Participants who used LLMs reported learning less, invested less effort when writing advice, and produced advice that was shorter, more generic, and less factual. Independent readers rated advice from LLM-learners as less informative and less helpful, and were less likely to adopt it. These differences were robust across contexts.

Read at gizmodo.com