Open Peer Review Data Sets
To study the process of peer review, data are needed to inform analysis and decisions on the effectiveness and efficiency of the process. As part of the AIBS commitment to studying the Science of Peer Review, we have synthesized a large amount of data on various facets of the peer review process. We believe that open data are a key component strengthening the study of peer review by providing the opportunity for wider evaluation and diverse insights, which will ultimately will enhance the integrity of the peer review process.
Compiled below is list of available 16 open data sets, from AIBS and others, concerning the approaches, outcomes and validations of the scientific peer review process, particularly focusing on peer review of applications for funding. This list is sorted by topic area and intended to serve as a resource for those conducting research on the peer review process.
Aibs Data Sets 
Barnett AG, Glisson SR, Gallo SA. “Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study.” f1000research.com 2018; 7:1030
> Dataset available at https://doi.org/10.5281/zenodo.1452073.
Carpenter AS, Sullivan JH, Deshmukh A, Glisson SR, & Gallo SA. “A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels.” BMJ Open. 8 September, 2015.
> Dataset available at http://dx.doi.org/10.6084/m9.figshare.1495503.
Erosheva EA, Martinková P, Lee CJ. When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review. J R Stat Soc Series A. 2021;184:904–919.
> Dataset available at https://doi.org/10.6084/m9.figshare.12728087.v1.
Gallo SA et al. “The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies.” 2014 PLoS ONE 9(9): e106474.
> Dataset available at https://doi.org/10.1371/journal.pone.0106474.s008.
Gallo SA, Schmaling KB (2022) Peer review: Risk and risk tolerance. PLoS ONE 17(8): e0273813. https://doi.org/10.1371/journal.pone.0273813
> Dataset available at https://osf.io/fu83d/.
Gallo SA, Sullivan JH, Glisson SR “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.” PLOS ONE. 21 October, 2016.
> Dataset available at https://doi.org/10.1371/journal.pone.0165147.s002.
Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “Risk evaluation in peer review of grant applications.” Environment Systems and Decisions. 24 February, 2018.
Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey.” Sci Eng Ethics. July, 2019. (Pre-print available)
Gallo, S.A., Schmaling, K.B., Thompson, L.A., and Glisson, S.R. Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion. Res Integr Peer Rev. 2020; 5:(7).
Gallo SA, Schmaling KB, Thompson LA, and Glisson SR. “Grant Review Feedback: Appropriateness and Usefulness.” Science and Engineering Ethics. 27 (18) March, 2021.
> Dataset available at https://doi.org/10.6084/m9.figshare.8132453.v1.
External Data Sets
Eblen MK, Wagner RM, RoyChowdhury D, Patel KC, Pearson K. How Criterion Scores Predict the Overall Impact Score and Funding Outcomes for National Institutes of Health Peer-Reviewed Applications. PLoS ONE. 2016; 11(6): e0155060.
> Dataset available at https://doi.org/10.1371/journal.pone.0155060.s003.
Kang D et al. A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications. ArXiv. 2018; abs/1804.09635.
> Dataset available at https://github.com/allenai/PeerRead.
Morgan B, Yu L-M, Solomon T, Ziebland S. Assessing health research grant applications: A retrospective comparative review of a one-stage versus a two-stage application assessment process. PLoS ONE. 2020; 15(3): e0230118.
> Dataset available at https://doi.org/10.1371/journal.pone.0230118.s001.
Sattler DN, McKnight PE, Naney L, Mathis R. Grant Peer Review: Improving Inter-Rater Reliability with Training. PLoS One. 2015 Jun 15;10(6):e0130450.
> Dataset available at http://figshare.com/authors/Patrick_Mcknight/695619.
Severin A, Martins J, Delavy F, Jorstad A, Egger M, Heyard R. Gender and other potential biases in peer review: Analysis of 38,250 external peer review reports. PeerJ Preprints. 2019; 7:e27587v3
> Dataset available at https://doi.org/10.5281/zenodo.2592509.
Snell RR. Menage a Quoi? Optimal Number of Peer Reviewers. PLoS ONE. 2015; 10(4): e0120838.
> Dataset available at https://doi.org/10.1371/journal.pone.0120838.s001.
Steiner Davis MLE, Conner TR, Miller-Bains K, Shapard L. What makes an effective grants peer reviewer? An exploratory study of the necessary skills. PLoS ONE. 2020; 15(5): e0232327.
> Dataset available at https://doi.org/10.1371/journal.pone.0232327.s004.
Weber-Main AM, McGee R, Eide Boman K, Hemming J, Hall M, Unold T, et al. Grant application outcomes for biomedical researchers who participated in the National Research Mentoring Network’s Grant Writing Coaching Programs. PLoS ONE. 2020; 15(11): e0241851.
> Dataset available at https://doi.org/10.1371/journal.pone.0241851.s002.
von Hippel T, von Hippel C. To Apply or Not to Apply: A Survey Analysis of Grant Writing Costs and Benefits. PLoS ONE. 2015; 10(3): e0118494.
> Dataset available at https://doi.org/10.1371/journal.pone.0118494.s002.