LLMs' impact on science: Booming publications, stagnating quality

"This effect was likely to be most pronounced in people that weren't native speakers of English. If the researchers limited the analysis to people with Asian names working at institutions in Asia, their rate of submissions to bioRxiv and SSRN nearly doubled once they started using AI and rose by over 40 percent at the arXiv. This suggests that people who may not have the strongest English skills are using LLMs to overcome a major bottleneck: producing compelling text."

""Papers with clear but complex language are perceived to be stronger and are cited more frequently," the researchers note, suggesting that we may use the quality of writing as a proxy for the quality of the research it's describing. And they found some indication of that here, as non-LLM-assisted papers were more likely to be published in the peer reviewed literature if they used complex language (the abstracts were scored for language complexity using a couple of standard measures)."

LLM usage by people with Asian names at Asian institutions nearly doubled submissions to bioRxiv and SSRN and increased submissions to arXiv by over 40 percent. Those who are not native English speakers appear to use LLMs to overcome writing bottlenecks and produce more compelling text. Non-LLM-assisted papers that used complex language were more likely to be published, indicating a positive correlation between linguistic complexity and publication. LLM-assisted manuscripts showed higher linguistic complexity but were less likely to be published, reversing that positive correlation. LLM-assisted papers cited a broader range of sources, including books and recent papers, which could diversify cited literature.

Read at Ars Technica