Author knows best? Top AI conference asks for self-ranked papers amid paper deluge

"The number of papers submitted to the top artificial-intelligence conferences has shot up, with some events seeing numbers rise more than tenfold over the past decade. This is not just a side effect of the rapid increase in global AI research output, says Buxin Su, a mathematician at the University of Pennsylvania in Philadelphia. AI conferences tend to attract multiple submissions from the same author, which poses a serious problem: who's going to sort through them all to find the most exciting and high-quality work?"

"In a study posted on the preprint server arXiv in October, Su and his colleagues describe a system that requires authors making more than one submission to directly compare their papers, ranking them by quality and potential impact. The rankings are calibrated against the assessments of peer reviewers (who do not see the self-ranking information) to highlight the top picks."

"Su says that the self-rankings and calibrated scores provide important insights that could be used to predict a paper's future performance. For example, the team found that papers with the highest self-rankings went on to receive twice as many citations as those with the lowest ones. "The authors' rankings are a very good predictor of the long-term impact," says Su. "The calibrated scores better reflect the true quality.""

Submissions to top AI conferences have increased dramatically, with some events experiencing more than tenfold growth over a decade. To address multiple submissions from the same author, a system requires authors with more than one paper to directly compare and rank their own submissions by quality and potential impact. Those self-rankings are calibrated against peer-review assessments (which remain blind to self-ranks) to surface top papers. The method was tested on 2,592 papers from 1,342 researchers submitted to ICML 2023 and validated by comparing calibrated scores to citation counts sixteen months later. Papers ranked highest by authors received about twice as many citations as lowest-ranked papers, and ICML 2026 will formally adopt the self-ranking approach.

Read at Nature