The gap between AI hype and newsroom reality

"Well, that assertion might be a bit more questionable now that the Columbia Journalism Review has gone and published its own study about how AI tools performed in that role for some specific journalistic use cases. The result, according to CJR: AI can be a surprisingly uninformed researcher, and it might not even be a great summarizer, at least in some cases."

"For summarization in particular, the tools-including ChatGPT, Claude, Perplexity, and Gemini-were asked to summarize transcripts and minutes from local government meetings, not articles or PowerPoints. So some of the results may go against intuition, but I also think it makes them much more useful: For artificial intelligence to be the force for workplace transformation as it's often hyped to be, it needs to give helpful output in workplace-specific use cases."

AI models were evaluated on newsroom-relevant tasks such as summarizing transcripts and minutes from local government meetings. Major tools including ChatGPT, Claude, Perplexity, and Gemini were tested for summarization and research assistance. The evaluation found that AI can act as a surprisingly uninformed researcher and sometimes fails to produce reliable summaries. Most editorial policies still forbid using AI to write stories, and many journalists resist AI-generated writing. Domain-specific, systematic testing is necessary because AI must provide helpful output in workplace-specific use cases. Non-engineers can meaningfully judge AI usefulness in their jobs.

Read at Fast Company