๐Ÿ“… Save the date for the 2023 IDEA Conference & Council Meeting - November 30, 2023 from 1-5pm ET. ๐Ÿ“…

Founded in 1947, in 2022 AIBS celebrates its 75th anniversary

Expert review is critical for strong science

Our work not only benefits and informs research program decisions, it also strengthens the integrity of the science that will benefit society.

Credit: Michal Jarmoluk

A small semi-transparent triangle for visual interest

Open Peer Review Data Sets

To study the process of peer review, data are needed to inform analysis and decisions on the effectiveness and efficiency of the process. As part of the AIBS commitment to studying the Science of Peer Review, we have synthesized a large amount of data on various facets of the peer review process. We believe that open data are a key component strengthening the study of peer review by providing the opportunity for wider evaluation and diverse insights, which will ultimately will enhance the integrity of the peer review process.

Compiled below is list of available 16 open data sets, from AIBS and others, concerning the approaches, outcomes and validations of the scientific peer review process, particularly focusing on peer review of applications for funding. This list is sorted by topic area and intended to serve as a resource for those conducting research on the peer review process.

Aibs Data Sets Up

Barnett AG, Glisson SR, Gallo SA. โ€œDo funding applications where peer reviewers disagree have higher citations? A cross-sectional study.โ€ f1000research.com 2018; 7:1030

> Dataset available at https://doi.org/10.5281/zenodo.1452073.

Carpenter AS, Sullivan JH, Deshmukh A, Glisson SR, & Gallo SA. โ€œA retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels.โ€ BMJ Open. 8 September, 2015.

> Dataset available at http://dx.doi.org/10.6084/m9.figshare.1495503.

Erosheva EA, Martinkovรก P, Lee CJ. When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review. J R Stat Soc Series A. 2021;184:904โ€“919.

> Dataset available at https://doi.org/10.6084/m9.figshare.12728087.v1.

Gallo SA, Schmaling KB (2022) Peer review: Risk and risk tolerance. PLoS ONE 17(8): e0273813. https://doi.org/10.1371/journal.pone.0273813

> Dataset available at https://osf.io/fu83d/.

Gallo SA, Sullivan JH, Glisson SR โ€œThe Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.โ€ PLOS ONE. 21 October, 2016.

> Dataset available at https://doi.org/10.1371/journal.pone.0165147.s002.

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. โ€œRisk evaluation in peer review of grant applications.โ€ Environment Systems and Decisions. 24 February, 2018.
Gallo SA, Thompson LA, Schmaling KB, Glisson SR. โ€œThe Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey.โ€ Sci Eng Ethics. July, 2019. (Pre-print available)
Gallo, S.A., Schmaling, K.B., Thompson, L.A., and Glisson, S.R. Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion. Res Integr Peer Rev. 2020; 5:(7).
Gallo SA, Schmaling KB, Thompson LA, and Glisson SR. โ€œGrant Review Feedback: Appropriateness and Usefulness.โ€ Science and Engineering Ethics. 27 (18) March, 2021.

> Dataset available at https://doi.org/10.6084/m9.figshare.8132453.v1.

External Data Sets Up

Sattler DN, McKnight PE, Naney L, Mathis R. Grant Peer Review: Improving Inter-Rater Reliability with Training. PLoS One. 2015 Jun 15;10(6):e0130450.

> Dataset available at http://figshare.com/authors/Patrick_Mcknight/695619.

Severin A, Martins J, Delavy F, Jorstad A, Egger M, Heyard R. Gender and other potential biases in peer review: Analysis of 38,250 external peer review reports. PeerJ Preprints. 2019; 7:e27587v3

> Dataset available at https://doi.org/10.5281/zenodo.2592509.

Snell RR. Menage a Quoi? Optimal Number of Peer Reviewers. PLoS ONE. 2015; 10(4): e0120838.

> Dataset available at https://doi.org/10.1371/journal.pone.0120838.s001.

Steiner Davis MLE, Conner TR, Miller-Bains K, Shapard L. What makes an effective grants peer reviewer? An exploratory study of the necessary skills. PLoS ONE. 2020; 15(5): e0232327.

> Dataset available at https://doi.org/10.1371/journal.pone.0232327.s004.

von Hippel T, von Hippel C. To Apply or Not to Apply: A Survey Analysis of Grant Writing Costs and Benefits. PLoS ONE. 2015; 10(3): e0118494.

> Dataset available at https://doi.org/10.1371/journal.pone.0118494.s002.