"An analysis of publicly available peer reviews for thousands of papers finds that papers receiving tough reviews go on to have a higher impact in science than those that sail through the review process."
"The journal has been making these files public for papers that the journal accepts since 2016, as long as the papers' authors give their consent. It does this for transparency and to inform discussion of published papers in the research community, the journal says."
"The authors asked the LLM - a version of Claude, which was created by artificial-intelligence company Anthropic, in San Francisco, California - to score reviewers' comments for how constructive they were and for how strong their opinions were. For example, a comment that the main conclusion of the paper "is not supported by the evidence" would score highly in opinion strength, Zeng says. But it would not score highly as a constructive comment without further suggestions for how the authors could address the problem."
"The model also assessed the 'quality' of comments, which was a measure of how specific and well-reasoned they are. To draw insights from the review correspondence, the team prompted a large language model (LLM) to evaluate these files for 8,000 published papers - 1,000 randomly selected per year from the period 2017 to 2024."
Public peer-review correspondence for thousands of papers was analyzed to compare later scientific impact. The analysis used publicly available reviewer files for Nature Communications papers accepted since 2016, with author consent. A large language model evaluated review correspondence for 8,000 published papers sampled from 2017 to 2024. The model scored reviewer comments for constructiveness, opinion strength, and comment quality based on specificity and reasoning. Comments that challenged conclusions without offering guidance were treated as strong in opinion strength but less constructive. The results linked tougher reviews with higher subsequent impact in science, suggesting that extensive revision may benefit research outcomes.
Read at Nature
Unable to calculate read time
Collection
[
|
...
]