📅 Save the date for the 2023 IDEA Conference & Council Meeting - November 30, 2023 from 1-5pm ET. 📅

Founded in 1947, in 2022 AIBS celebrates its 75th anniversary

"What news from the sea?"

The fish replied: "I have a lot to say, but my mouth is full of water." - Armenian proverb

The San Diego, California shoreline. Credit: Frank McKenna

A small semi-transparent triangle for visual interest
Science Marches On

News & Events

Explore the most recent news about AIBS's initiatives, programs, resources, and events.

Bullet peer-review · Aug 01, 2023

Score, then Rank: Researchers Propose an Integrated Approach to Grant Review Assessments

The public funding of science is responsible for many of the biomedical and other scientific breakthroughs on which our lives depend. However, the process through which funding decisions are made, the peer review of grant proposals, has been historically understudied, and current approaches can lead to undesirable outcomes. Writing in Research Integrity and Peer Review, Stephen A. Gallo, then affiliated with the American Institute of Biological Sciences, and Michael Pearce, Carole J. Lee, and Elena A. Erosheva from the University of Washington highlight problems related to proposal rating in grant review and propose a protocol to ameliorate them.

In many review programs, say the authors, proposals are funded primarily on the basis of reviewer ratings and summary statistics derived from them that represent reviewers’ assessment of proposal quality. Relying solely on ratings, however, can pose problems for establishing funding priorities, “especially when distinguishing between similarly rated proposals, especially for projects of high quality, where only a few minor weaknesses can make the difference between a funded project and a rejection.” Furthermore, say Gallo and colleagues, the use of ratings rubrics “often forces reviewers to penalize a project based on a series of identified but minor weaknesses, despite the fact that they may find the application overall to be potentially significant.”

To address these concerns, the authors suggest adding rankings to the traditional rating approach and describe an innovative methodology called the Mallows-Binomial, which can then produce integrated proposal scores, as well as an induced preference ordering. In so doing, the approach allows funders to consider reviewers’ scores alongside their opinions about how well the proposals stack up against one another. Pearce states that the inclusion of ranking will “more accurately distill reviewer opinion into a useful output to make an informed funding decision.”

For research funders, the authors suggest clear benefits of the Mallows-Binomial protocol in comparison with ratings-only approaches. These include “providing a ranked priority list with confidence metrics, a higher degree of discrimination between similarly rated proposals, and a robustness to outliers and reviewer inconsistencies,” features that will be important in distilling reviewer opinion “into a useful output to make the best, most informed decision” for funding crucial research.