Expert review is critical for strong science

Barnett AG, Glisson SR, Gallo SA. “Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study.” f1000research.com.

Carpenter AS, Sullivan JH, Deshmukh A, Glisson SR, & Gallo SA. “A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels.” BMJ Open. 8 September, 2015.

Chandrasekharan, S., Zaka, M., Gallo, S., Zhao, W., Korobskiy, D., Warnow, T., & Chacko, G. (2020). Finding scientific communities in citation graphs: Articles and authors. Quantitative Science Studies. Advance publication. https://doi.org/10.1162 /qss_a_00095

Gallo SA et al. “The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies.” 2014 PLoS ONE 9(9): e106474. doi:10.1371/journal.pone.0106474

Gallo SA, Carpenter AS, Glisson SR (2013) “Teleconference Versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes.” PLoS ONE 8(8).

Gallo SA, LeMaster M, Glisson SR. “Frequency and Type of Conflicts of Interest in the Peer Review of Basic Biomedical Research Funding Applications: Self-Reporting Versus Manual Detection.” Sci Eng Ethics. 2015 Feb 4.

Gallo SA, Schmaling KB (2022) “Peer review: Risk and risk tolerance.” PLoS ONE 17(8): e0273813. https://doi.org/10.1371/journal.pone.0273813

Gallo SA, Schmaling KB, Thompson LA, and Glisson SR. “Grant Review Feedback: Appropriateness and Usefulness.” Science and Engineering Ethics. 27 (18) March, 2021.

Gallo SA, Sullivan JH, & Croslan DR. “Scientists from Minority-Serving Institutions and Their Participation in Grant Peer Review.” BioScience. 2022; 72 (3) 289–299.

Gallo SA, Sullivan JH, Glisson SR “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.” PLOS ONE. 21 October, 2016.

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey..” Sci Eng Ethics. July, 2019. (Pre-print available)

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “Risk evaluation in peer review of grant applications.” Environment Systems and Decisions. 24 February, 2018.

Gallo, S.A., Schmaling, K.B., Thompson, L.A., and Glisson, S.R. Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion. Res Integr Peer Rev. 2020; 5:(7).

Gallo, SA, Pearce, M, Lee, CJ & Erosheva, EA. “A new approach to grant review assessments: score, then rank.” Res Integr Peer Rev. 2023; 8, 10.

Irwin D, Gallo SA, Glisson SR. “Opinion: Learning from Peer Review.” The Scientist. 2013 May 24.

Schmaling, KB & Gallo, SA. “Gender differences in peer reviewed grant applications, awards, and amounts: a systematic review and meta-analysis.” Res Integr Peer Rev. 2023; 8, 2.

Schmaling, KB, Evenson, GR, Marble, BK, & Gallo, SA. “Perceptions of grant peer reviewers: a mixed methods study.” Research Evaluation, 2024; Volume 33 rvae050

Stephen A. Gallo, Scott R. Glisson. “External Tests of Peer Review Validity Via Impact Measures.” Front. Res. Metr. Anal. 23 August 2018

Cook,WD, Golany B, Kress M, Penn M. Optimal Allocation of Proposals to Reviewers to Facilitate Effective Ranking. Manage Sci. 2005;51(4):655-61.

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey..” Sci Eng Ethics. July, 2019. (Pre-print available)

Li L, Wang Y, Liu G, Wang M, Wu X. Context-Aware Reviewer Assignment for Trust Enhanced Peer Review. PLoS One. 2015 Jun 19;10(6):e0130493.

Snell, R. R. Menage a quoi? Optimal number of peer reviewers. PloS One 2015; 10(4), e0120838

Abdoul H, Perrey C, Tubach F, Amiel P, Durand-Zaleski I, Alberti C. Non-financial Conflicts of Interest in Academic Grant Evaluation: A Qualitative Study of Multiple Stakeholders in France. PLoS One. 2012;7(4):e35247.

Arms W. Ethics: Trust and Reputation on the Web. Nature 2006.

Bekelman JE, Li Y, Gross CP. Scope and Impact of Financial Conflicts of Interest in Biomedical Research: A Systematic Review. JAMA. 2003 Jan 22-29;289(4):454-65.

Benos D. Ethics: Detecting Misconduct. Nature 2006.

Bhattacharjee Y. Science Funding. NSF’s ‘Big Pitch’ Tests Anonymized Grant Reviews. Science. 2012 May 25;336(6084):969-70.

Bornmann L, Daniel HD. Selection of Research Fellowship Recipients by Committee Peer Review. Reliability, Fairness and Predictive Validity of Board of Trustees’ Decisions. Scientometrics. 2005;63(2):297-320.

Bornmann L. Research Misconduct–Definitions, Manifestations and Extent. Publications. 2013;1(3):87-98.

Boudreau KJ, Eva G, Lakhani KR, Riedl C. Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance and Resource Allocation in Science. Available at SSRN. 2014.

Bromham L, Dinnage R, and Hua X. “Interdisciplinary research has consistently lower funding success.” Nature 534, no. 7609 (2016): 684.

Chandrasekharan, S., Zaka, M., Gallo, S., Zhao, W., Korobskiy, D., Warnow, T., & Chacko, G. (2020). Finding scientific communities in citation graphs: Articles and authors. Quantitative Science Studies. Advance publication. https://doi.org/10.1162 /qss_a_00095

Coats AJ, Nijjer SS, Francis DP. Protecting the Pipeline of Science: Openness, Scientific Methods and the Lessons from Ticagrelor and the PLATO Trial. Int J Cardiol. 2014 Oct 20;176(3):600-4.

De Martino B, Kumaran D, Seymour B, Dolan RJ. Frames, Biases, and Rational Decision-Making in the Human Brain. Science. 2006 Aug 4;313(5787):684-7.

Drazen JM, De Leeuw PW, Laine C, Mulrow C, Deangelis CD, Frizelle FA, Godlee F, Haug C, Hébert PC, James A, Kotzin S, Marusic A, Reyes H, Rosenberg J, Sahni P, Van Der Weyden MB, Zhaori G. Toward More Uniform Conflict Disclosures: The Updated ICMJE Conflict of Interest Reporting Form. Natl Med J India. 2010 Jul-Aug;23(4):196-7.

Gallo SA, LeMaster M, Glisson SR. “Frequency and Type of Conflicts of Interest in the Peer Review of Basic Biomedical Research Funding Applications: Self-Reporting Versus Manual Detection.” Sci Eng Ethics. 2015 Feb 4.

Gallo SA, Schmaling KB (2022) “Peer review: Risk and risk tolerance.” PLoS ONE 17(8): e0273813. https://doi.org/10.1371/journal.pone.0273813

Gallo SA, Schmaling KB, Thompson LA, and Glisson SR. “Grant Review Feedback: Appropriateness and Usefulness.” Science and Engineering Ethics. 27 (18) March, 2021.

Gallo SA, Sullivan JH, & Croslan DR. “Scientists from Minority-Serving Institutions and Their Participation in Grant Peer Review.” BioScience. 2022; 72 (3) 289–299.

Gallo SA, Sullivan JH, Glisson SR “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.” PLOS ONE. 21 October, 2016.

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “Risk evaluation in peer review of grant applications.” Environment Systems and Decisions. 24 February, 2018.

Ginther, D. K., Basner, J., Jensen, U., Schnell, J., Kington, R., & Schaffer, W. T. (2018). “Publications as predictors of racial and ethnic differences in NIH research awards.” PloS one, 13(11), e0205929.

Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., & Kington, R. Race, ethnicity, and NIH research awards. Science. 2011;333(6045), 1015-1019.

Hodgson C. Evaluation of Cardiovascular Grant-in-aid Applications by Peer Review: Influence of Internal and External Reviewers and Committees. Can J Cardiol. 1995 Nov;11(10):864-8.

Hoppe TA, Litovitz A, Willis KA, Meseroll RA, Perkins MJ, Hutchins BI, Davis AF, Lauer MS, Valantine HA, Anderson JM, Santangelo GM. “Topic choice contributes to the lower rate of NIH awards to African-American/black scientists.” Science Advances. 2019 Oct 1;5(10):eaaw7238.

Kaatz A, Magua W, Zimmerman DR, Carnes M. A Quantitative Linguistic Analysis of National Institutes of Health R01 Application Critiques from Investigators at One Institution. Acad Med. 2015 Jan;90(1):69-75

Kolehmainen C, & Carnes M (2018). “Who resembles a scientific leader—Jack or Jill? How implicit bias could influence research grant funding.” Circulation, 137(8), 769-770.

Kotchen TA, Lindquist T, Miller Sostek A, Hoffmann R, Malik K, Stanfield B. Outcomes of National Institutes of Health Peer Review of Clinical Grant Applications. J Investig Med. 2006 Jan;54(1):13-9.

Langfeldt L. The Policy Challenges of Peer Review: Managing Bias, Conflict of Interests and Interdisciplinary Assessments. Res Eval. 2006;15(1):31-41.

Langfeldt L. Decision-Making in Expert Panels Evaluating Research: Constraints, Processes and Bias. 2001;dissertation: The University of Oslo.

Lee C. Sugimoto CR, Zhang G, Cronin B. Bias in Peer Review. J Am Soc Inf Sci Technol. 2013;64(1):2-17.

Li D. Expertise vs. Bias in Evaluation: Evidence from the NIH. 2013;Northwestern University.

Li L, Wang Y, Liu G, Wang M, Wu X. Context-Aware Reviewer Assignment for Trust Enhanced Peer Review. PLoS One. 2015 Jun 19;10(6):e0130493.

Lo B, Field MJ, Editors. Conflict of Interest in Medical Research, Education, and Practice. Committee on Conflict of Interest in Medical Research, Education, and Practice. 2009;Institute of Medicine

Magua W, Zhu X, Bhattacharya A, Filut A, Potvien A, Leatherberry R, Lee YG, Jens M, Malikireddy D, Carnes M, Kaatz A. “Are female applicants disadvantaged in National Institutes of Health peer review? Combining algorithmic text mining and qualitative methods to detect evaluative differences in R01 reviewers’ critiques.” Journal of Women’s Health. 2017 May 1;26(5):560-70.

Marsh HW, Jayasinghe UW, Bond NW. Improving the Peer-Review Process for Grant Applications: Reliability, Validity, Bias, and Generalizability. Am Psychol. 2008 Apr;63(3):160-8.

NIH. Revised Policy: Managing Conflict of Interest in the Initial Peer Review of NIH Grant and Cooperative Agreement Applications. NOT-OD-11-120. 2011 Sep 26.

NIH. Managing Conflict of Interest in NIH Peer Review of Grants and Contracts.

Oleinik A. Conflict(s) of Interest in Peer Review: Its Origins and Possible Solutions. Sci Eng Ethics. 2014 Mar;20(1):55-75.

Pham-Kanter G. Revisiting Financial Conflicts of Interest in FDA Advisory Committees. Milbank Q. 2014 Sep;92(3):446-70.

Rockey SJ, Collins FS. Managing Financial Conflict of Interest in Biomedical Research. JAMA. 2010 Jun 16;303(23):2400-2.

Stehbens WE. Basic Philosophy and Concepts Underlying Scientific Peer Review. Med Hypotheses. 1999 Jan;52(1):31-6.

Tamblyn, R., Girard, N., Qian, C. J., & Hanley, J. Assessment of potential bias in research grant peer review in Canada. CMAJ 2018;190(16), E489-E499.

Wager E. Ethics: What is it for? Nature 2006.

Wood SF, Mador JK. Science and Regulation. Uncapping Conflict of Interest? Science. 2013 Jun 7;340(6137):1172-3.

Bantum EO, Owen JE. Evaluating the Validity of Computerized Content Analysis Programs for Identification of Emotional Expression in Cancer Narratives. Psychol Assess. 2009 Mar;21(1):79-88.

Bornmann L, Wolf M, Daniel HD. Closed Versus Open Reviewing of Journal Manuscripts: How Far do Comments Differ in Language Use? Scientometrics. 2012;91(3):843-56.

Fuller EO, Hasselmeyer EG, Hunter JC, Abdellah FG, Hinshaw AS. Summary Statements of the NIH Nursing Research Grant Applications. Nurs Res. 1991 Nov-Dec;40(6):346-51.

Hume KM, Giladi AM, Chung KC. Factors Impacting Successfully Competing for Research Funding: An Analysis of Applications Submitted to the Plastic Surgery Foundation. Plast Reconstr Surg. 2015 Feb;135(2):429e-35e.

Kaatz A, Magua W, Zimmerman DR, Carnes M. A Quantitative Linguistic Analysis of National Institutes of Health R01 Application Critiques from Investigators at One Institution. Acad Med. 2015 Jan;90(1):69-75

Magua W, Zhu X, Bhattacharya A, Filut A, Potvien A, Leatherberry R, Lee YG, Jens M, Malikireddy D, Carnes M, Kaatz A. “Are female applicants disadvantaged in National Institutes of Health peer review? Combining algorithmic text mining and qualitative methods to detect evaluative differences in R01 reviewers’ critiques.” Journal of Women’s Health. 2017 May 1;26(5):560-70.

Pier EL, Raclaw J, Kaatz A, Brauer M, Carnes M, Nathan MJ, & Ford CE (2017). . “‘Your comments are meaner than your score’: score calibration talk influences intra-and inter-panel variability during scientific grant peer review.” Research Evaluation, 26(1), 1-14.

Pier, E.L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J., Nathan, M.J., Ford, C.E. and Carnes, M., 2018. “Low agreement among reviewers evaluating the same NIH grant applications.”Proceedings of the National Academy of Sciences, 115(12), pp.2952-2957.

Tausczik YR, Pennebaker JW. The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods. J Lang Soc Psychol. 2010;29(1):24-54.

Vener KJ, Calkins BM. Analysis of SBIR Phase I and Phase II Review Results at the National Institutes of Health. FASEB J. 1991 Sep;5(12):2640-4.

De Martino B, Kumaran D, Seymour B, Dolan RJ. Frames, Biases, and Rational Decision-Making in the Human Brain. Science. 2006 Aug 4;313(5787):684-7.

Denes-Raj V, Epstein S. Conflict Between Intuitive and Rational Processing: When People Behave Against Their Better Judgment. J Pers Soc Psychol. 1994 May;66(5):819-29.

Finucane ML, Alhakami A, Slovic P, Johnson SM. The Affect Heuristic in Judgments of Risks and Benefits. J Behav Decis Mak. 2000;13(1):1-17.

Gallo SA, Schmaling KB (2022) “Peer review: Risk and risk tolerance.” PLoS ONE 17(8): e0273813. https://doi.org/10.1371/journal.pone.0273813

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “Risk evaluation in peer review of grant applications.” Environment Systems and Decisions. 24 February, 2018.

Gregory R, Mendelsohn R. Perceived Risk, Dread, and Benefits. Risk Anal. 1993;13(3):259-64.

Hagman W. Affective Biases and Heuristics in Decision Making: Emotion Regulation as a Factor for Decision Making Competence. 2013. Linköping University. Master Thesis.

Langfeldt L. The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome. Social Studies of Science. 2001 Dec;31(6):820-41.

Lipshitz R, Strauss O. Coping with Uncertainty: A Naturalistic Decision-Making Analysis. Organ Behav Hum Decis Process. 1997;69(2):149-63.

Pachur T, Hertwig R, Steinmann F. How do People Judge Risks: Availability Heuristic, Affect Heuristic, or Both? J Exp Psychol Appl. 2012 Sep;18(3):314-30.

Rand DG, Epstein ZG. Risking Your Life Without a Second Thought: Intuitive Decision-Making and Extreme Altruism. PLoS One. 2014 Oct 15;9(10):e109687.

Schmaling, KB, Evenson, GR, Marble, BK, & Gallo, SA. “Perceptions of grant peer reviewers: a mixed methods study.” Research Evaluation, 2024; Volume 33 rvae050

Sheeran P, Harris PR, Epton T. Does Heightening Risk Appraisals Change People’s Intentions and Behavior? A Meta-Analysis of Experimental Studies. Psychol Bull. 2014 Mar;140(2):511-43.

Sjöberg L. Factors in Risk Perception. Risk Anal. 2000;20(1):1-12.

Slovic P, Finucane ML, Peters E, MacGregor DG. Risk as Analysis and Risk as Feelings: Some Thoughts About Affect, Reason, Risk, and Rationality. Risk Anal. 2004 Apr;24(2):311-22.

Slovic P, Peters E. Risk Perception and Affect. Curr Dir Psychol Sci. 2006;15(6):322-5.

Taleb NN. Silent Risk: Technical Incerto, Lectures on Risk and Probability. Available at SSRN. 2014;1.

Tversky, Amos, and Daniel Kahneman. “Judgment under uncertainty: Heuristics and biases.” science 185, no. 4157 (1974): 1124-1131.

Azoulay P, Graff Zivin JS, Manso G. Incentives and Creativity: Evidence from the Academic Life Sciences. NBER Working Paper No. 15466. 2009.

Bollen, J., Crandall, D., Junk, D., Ding, Y., & Börner, K. (2014). “From funding agencies to scientific agency.” EMBO reports, 15(2), 131-133.

Bolli R. Actions Speak Much Louder Than Words: For Midcareer and Senior Investigators, the Track Record of Productivity Should be Paramount in Selecting Grant Recipients. Circ Res. 2014 Dec 5;115(12):962-6.

Boyington, J. E., Antman, M. D., Patel, K. C., & Lauer, M. S. (2016). “Towards Independence: Resubmission Rate of Unfunded National Heart, Lung, and Blood Institute R01 Research Grant Applications among Early Stage Investigators.” Academic medicine: journal of the Association of American Medical Colleges, 91(4), 556.

Fang, F.C. and Casadevall, A., 2016. “Research Funding: the Case for a Modified Lottery.” mBio, 7(2).

Gallo SA, Sullivan JH, Glisson SR “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.” PLOS ONE. 21 October, 2016.

Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., & Kington, R. Race, ethnicity, and NIH research awards. Science. 2011;333(6045), 1015-1019.

Lauer MS. Personal Reflections on Big Science, Small Science, or The Right Mix. Circ Res. 2014 Mar 28;114(7):1080-2.

Vermeulen N, Parker JN, Penders B. Big, Small or Mezzo? Lessons from Science Studies for the Ongoing Debate About ‘Big’ Versus ‘Little’ Research Projects. EMBO Rep. 2010 Jun;11(6):420-3.

Bornmann L, Mutz R, Daniel HD. Gender Differences in Grant Peer Review: A Meta-Analysis. J Informetr. 2007 Jul;1(3):226-38.

Gallo SA, Schmaling KB, Thompson LA, and Glisson SR. “Grant Review Feedback: Appropriateness and Usefulness.” Science and Engineering Ethics. 27 (18) March, 2021.

Homma MK, Motohashi R, Ohtsubo H. Japan’s Lagging Gender Equality. Science. 2013 Apr 26;340(6131):428-30.

Kaatz A, Magua W, Zimmerman DR, Carnes M. A Quantitative Linguistic Analysis of National Institutes of Health R01 Application Critiques from Investigators at One Institution. Acad Med. 2015 Jan;90(1):69-75

Kaufman RR, Chevan J. The Gender Gap in Peer-Reviewed Publications by Physical Therapy Faculty Members: A Productivity Puzzle. Phys Ther. 2011 Jan;91(1):122-31.

Kolehmainen C, & Carnes M (2018). “Who resembles a scientific leader—Jack or Jill? How implicit bias could influence research grant funding.” Circulation, 137(8), 769-770.

Magua W, Zhu X, Bhattacharya A, Filut A, Potvien A, Leatherberry R, Lee YG, Jens M, Malikireddy D, Carnes M, Kaatz A. “Are female applicants disadvantaged in National Institutes of Health peer review? Combining algorithmic text mining and qualitative methods to detect evaluative differences in R01 reviewers’ critiques.” Journal of Women’s Health. 2017 May 1;26(5):560-70.

Mutz R, Bornmann L, Daniel HD. Does Gender Matter in Grant Peer Review?: An Empirical Investigation Using the Example of the Austrian Science Fund. Z Psychol. 2012;220(2):121-129.

Schmaling, KB & Gallo, SA. “Gender differences in peer reviewed grant applications, awards, and amounts: a systematic review and meta-analysis.” Res Integr Peer Rev. 2023; 8, 2.

Tamblyn, R., Girard, N., Qian, C. J., & Hanley, J. Assessment of potential bias in research grant peer review in Canada. CMAJ 2018;190(16), E489-E499.

van Arensbergen P, van der Weijden I, van den Besselaar P. Academic Talent Selection in Grant Review Panels. In book: (Re)Searching Academic Careers, Chapter: 2. 2014. Publisher: Russian Academy of Sciences.

AICPA. Summary of the Nature, Objectives, Scope, Limitations of, and Procedures Performed in System and Engagement Reviews and Quality Control Reviews (as referred to in a peer review report). 2012.

Abdoul H, Perrey C, Amiel P, Tubach F, Gottot S, Durand-Zaleski I, Alberti C. Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices. PLoS One. 2012;7(9):e46054.

Alberts B, Kirschner MW, Tilghman S, Varmus H. Rescuing US Biomedical Research from its Systemic Flaws. Proc Natl Acad Sci U S A. 2014 Apr 22;111(16):5773-7.

Bandeh-Ahmadi A. Editorial Decisions and Measuring Information Contents of Reviews. Available at SSRN. 2013.

Benos DJ, Bashari E, Chaves JM, Gaggar A, Kapoor N, LaFrance M, Mans R, Mayhew D, McGowan S, Polter A, Qadri Y, Sarfare S, Schultz K, Splittgerber R, Stephenson J, Tower C, Walton RG, Zotov A. The Ups and Downs of Peer Review. Adv Physiol Educ. 2007 Jun;31(2):145-52.

Berg JM. Science Policy: Well-Funded Investigators Should Receive Extra Scrutiny. Nature. 2012 Sep 13;489(7415):203.

Bhattacharya A. Science Funding: Duel to the Death. Nature. 2012 Aug 2;488(7409):20-2.

Bielski A, Harris R, Gillis N. Summary Report of Comments Received on NIH System to Support Biomedical and Behavioral Research and Peer Review. 2007 November.

Bollen, J., Crandall, D., Junk, D., Ding, Y., & Börner, K. (2014). “From funding agencies to scientific agency.” EMBO reports, 15(2), 131-133.

Bornmann L. Scientific Peer Review: An Analysis of the Peer Review Process from the Perspective of Sociology of Science Theories. Human Architecture: Journal of the Sociology of Self-Knowledge. 2008;VI(2):23-38.

Brown T. Perspective: ‘I Don’t Know What to Believe’. Nature 2006.

Cantor M, Gero S. The Missing Metric: Quantifying Contributions of Reviewers. R Soc Open Sci. 2015 Feb 11;2(2):140540.

Carter G. Peer Review, Citations, and Biomedical Research Policy: NIH Grants to Medical School Faculty. RAND Corporation, 1974.

Chandrasekharan, S., Zaka, M., Gallo, S., Zhao, W., Korobskiy, D., Warnow, T., & Chacko, G. (2020). Finding scientific communities in citation graphs: Articles and authors. Quantitative Science Studies. Advance publication. https://doi.org/10.1162 /qss_a_00095

Chang WR, McLean IP. CUSUM: A Tool for Early Feedback About Performance? BMC Med Res Methodol. 2006 Mar 2;6:8.

Chetty R. How Small Changes Can Influence Reviewer Behavior. Elsevier. 2014.

Cole S, Rubin L, Cole J. Peer Review in the National Science Foundation: Phase One of a Study. Washington, DC: National Academy of Sciences, 1978.

Council of Canadian Academies. Informing Research Choices: Indicators and Judgment. 2012.

Das NK, Froehlich LA. Quantitative Evaluation of Peer Review of Program Project and Center Applications in Allergy and Immunology. J Clin Immunol. 1985 Jul;5(4):220-7.

Demicheli V, Di Pietrantonj C. Peer Review for Improving the Quality of Grant Applications. Cochrane Database Syst Rev. 2007 Apr 18;(2):MR000003.

Derraik JG. The Principles of Fair Allocation of Peer-Review: How Much Should a Researcher be Expected to Contribute? Sci Eng Ethics. 2014 Sep 7.

DiSilvestro RL. The First Half Century: A History of AIBS. BioScience. 1997 Nov;47(10):643-9.

Dickersin K, Ssemanda E, Mansell C, Rennie D. What do the JAMA Editors Say When They Discuss Manuscripts That They Are Considering for Publication? Developing a Schema for Classifying the Content of Editorial Discussion. BMC Med Res Methodol. 2007 Sep 25;7:44.

Dumanis SB, Ullrich L, Washington PM, Forcelli PA. It’s Money! Real-World Grant Experience Through a Student-Run, Peer-Reviewed Program. CBE Life Sci Educ. 2013 Fall;12(3):419-28.

Fang, F.C. and Casadevall, A., 2016. “Research Funding: the Case for a Modified Lottery.” mBio, 7(2).

Fleurence RL, Forsythe LP, Lauer M, Rotter J, Ioannidis JP, Beal A, Frank L, Selby JV. Engaging Patients and Stakeholders in Research Proposal Review: The Patient-Centered Outcomes Research Institute. Ann Intern Med. 2014 Jul 15;161(2):122-30.

Fliesler SJ. Rethinking Grant Peer Review. Science. 1997 Mar 7;275(5305):1399.

Gallo SA et al. “The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies.” 2014 PLoS ONE 9(9): e106474. doi:10.1371/journal.pone.0106474

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey..” Sci Eng Ethics. July, 2019. (Pre-print available)

Gallo, SA, Pearce, M, Lee, CJ & Erosheva, EA. “A new approach to grant review assessments: score, then rank.” Res Integr Peer Rev. 2023; 8, 10.

Garrison HH, Deschamps AM. NIH Research Funding and Early Career Physician Scientists: Continuing Challenges in the 21st Century. FASEB J. 2014 Mar;28(3):1049-58.

Gillespie GW, Jr., Chubin DE, Kurzon GM. Experience with NIH Peer Review: Researchers’ Cynicism and Desire for Change. Sci Technol Human Values. 1985;10(3):45-54.

Gillett R. Research Performance Indicators Based on Peer Review: A Critical Analysis. High Educ Q. 2007;43(1):20-38.

Glass B, Krauss RW. AIBS Presidents Revisit the Past. BioScience. 1997 Nov;47(10):650-6.

Glenn JF. Uses and Needs for Peer Review in Army Medical Research. Technol Innov. 2010;12:241-7.

Graves N, Barnett AG, Clarke P. Funding Grant Proposals for Scientific Research: Retrospective Analysis of Scores by Members of Grant Review Panel. BMJ. 2011 Sep 27;343:d4797.

Greaves S, Scott J, Clarke M, Miller M, Hannay T, Thomas A, Campbell P. Overview: Nature’s Peer Review Trial. Nature. 2006.

Groves T. Quality and Value: How Can We Get the Best Out of Peer Review? Nature. 2006.

Guthrie S, Guerin B, Wu H, Ismail S, Wooding S. Alternatives to Peer Review in Research Project Funding. Santa Monica, CA: RAND Corporation, 2013.

Hand E. Extra Scrutiny for ‘Grandee Grantees’. Nature. 2012 Feb 20;482(7386):450-1.

Hartmann I, Neidhardt F. Peer Review at the Deutsche Deutsche Forschungsgemeinschaft. Scientometrics. 1990 Nov 01;19(5-6):419-25.

Hartonen T, Alava MJ. How Important Tasks Are Performed: Peer Review. Sci Rep. 2013;3:1679.

Hayes M Grant Review in focus.” Publons 2019

Herbert DL, Barnett AG, Clarke P, Graves N. On the Time Spent Preparing Grant Proposals: An Observational Study of Australian Researchers. BMJ Open. 2013 May 28;3(5). pii: e002800.

Herbert DL, Barnett AG, Graves N. Funding: Australia’s Grant System Wastes time. Nature. 2013 Mar 21;495(7441):314.

Herbert DL, Coveney J, Clarke P, Graves N, Barnett AG. The Impact of Funding Deadlines on Personal Workloads, Stress and Family Relationships: A Qualitative Study of Australian Researchers. BMJ Open. 2014 Mar 28;4(3):e004462.

Herrmann-Lingen C, Brunner E, Hildenbrand S, Loew TH, Raupach T, Spies C, Treede RD, Vahl CF, Wenz HJ. Evaluation of Medical Research Performance–Position Paper of the Association of the Scientific Medical Societies in Germany (AWMF). Ger Med Sci. 2014 Jun 26;12:Doc11.

Hickam D, Totten A, Berg A, Rader K, Goodman S, Newhouse R, Editors. The PCORI Methodology Report. 2013.

Ho RC, Mak KK, Tao R, Lu Y, Day JR, Pan F. Views on the Peer Review System of Biomedical Journals: An Online Survey of Academics from High-Ranking Universities. BMC Med Res Methodol. 2013 Jun 7;13:74.

Houser SR. How to Obtain a National Heart, Lung, and Blood Institute-Sponsored K08 and K99/R00 Grant in the Current Funding Climate. Circ Res. 2012 Mar 30;110(7):907-9.

IOM. Strategies for managing the Breast Cancer Research Program: A Report to the U.S. Army Medical Research and Development Command. Washington, DC: National Academy Press, 1993.

Irwin D, Gallo SA, Glisson SR. “Opinion: Learning from Peer Review.” The Scientist. 2013 May 24.

Ismail S, Farrands A, Wooding S. Evaluating Grant Peer Review in the Health Sciences: A review of the Literature. Santa Monica, CA: RAND Corporation, 2009.

Jefferson T, Wager E, Davidoff F. Measuring the Quality of Editorial Peer Review. JAMA. 2002 Jun 5;287(21):2786-90.

Jefferson T. Quality and Value: Models of Quality Control for Scientific Research. Nature. 2006.

Jennings C. Quality and Value: The True Purpose of Peer Review. Nature. 2006.

Johnson VE. Statistical Analysis of the National Institutes of Health Peer Review System. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

Kemp E. Open Letter to Senior Editors of Peer-Review Journals Publishing in the Field of Stem Cell Biology. EuroStemCell. 2009 Jul.

King JC, Lawrence TS, Murphy SB, Davidson NE, Mayer RJ. The American Society of Clinical Oncology Cancer Foundation Grants Program: A 25-Year Report and a Look Toward the Future. J Clin Oncol. 2010 Mar 20;28(9):1616-21.

Koonin E, Landweber L, Lipman D, Dignon R. Systems: Reviving a Culture of Scientific Debate. Nature. 2006.

Kostoff RN. Research Program Peer Review: Purposes, Principles, Practices, Protocols. 2004. Office of Naval Research.

Kupfer DJ, Murphree AN, Pilkonis PA, Cameron JL, Giang RT, Dodds NE, Godard KA, Lewis DA. Using Peer Review to Improve Research and Promote Collaboration. Acad Psychiatry. 2014 Feb;38(1):5-10.

Lahiri D. Perspective: The Case for Group Review. Nature 2006.

Lee C. Perspective: Peer Review of Interdisciplinary Scientific Papers. Nature 2006.

Lee K, Bero L. Ethics: Increasing Accountability. Nature 2006.

Lee K, Brownstein JS, Mills RG, Kohane IS. Does Collocation Inform the Impact of Collaboration? PLoS One. 2010 Dec 15;5(12):e14279.

Liaw L, Freedman JE, Becker LB, Mehta NN, & Liscum L. (2017). “Peer review practices for evaluating biomedical research grants: A scientific statement from the American Heart Association.” Circulation research, 121(4), e9-e19.

Lyman RL. A Three-Decade History of the Duration of Peer Review. Journal of Scholarly Publishing. 2013 Apr;44(3):211-20.

McGarity TO. Peer Review in Awarding Federal Grants in the Arts and Sciences. Berkeley Tech LJ. 1994;9(1).

McGeary M, Hanna KE. Strategies to Leverage Research Funding: Guiding DOD’s Peer Reviewed Medical Research Programs. Washington, DC: National Academies Press, 2004.

Mervis J. Peer Review. Beyond the Data. Science. 2011 Oct 14;334(6053):169-71.

Mietchen D. The Transformative Nature of Transparency in Research Funding. PLoS Biol. 2014 Dec 30;12(12):e1002027.

Moed HF, Glanzel W, Schmoch U. Handbook of Quantitative Science and Technology Research: The Use of Publications and Patent Statistics in Studies of S&T Systems. New York, NY: Kluwer Academic Publishers, 2004.

Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Int J Surg. 2010;8(5):336-41.

Moore J. Perspective: Does Peer Review Mean the Same to the Public as it Does to Scientists? Nature 2006.

Myers ER, Alciati MH, Ahlport KN, Sung NS. Similarities and Differences in Philanthropic and Federal Support for Medical Research in the United States: An Analysis of Funding by Nonprofits in 2006-2008. Acad Med. 2012 Nov;87(11):1574-81.

Nicholson J, Ioannidis J. Research Grants: Conform and be Funded. Nature. 2012 Dec 6;492:34-36.

Ozonoff D. Quality and Value: Statistics in Peer Review. Nature. 2006.

Parliamentary Office of Science and Technology. Peer Review. 2002;182.

Pawelczyk JA, Strawbridge LM, Schultz AM, Liverman CT. A Review of NASA Human Research Program’s Scientific Merit Processes: Letter Report. Washington, DC: National Academies Press, 2012.

Pederson T. The “Study” Role of Past National Institutes of Health Study Sections. Mol Biol Cell. 2012 Sep;23(17):3281-4.

Ranalli B. A Prehistory of Peer Review: Religious Blueprints from the Hartlib Circle. Spontaneous Generations. 2011;5(1):12-18.

Riley B. Systems: Trusting Data’s Quality. Nature. 2006.

Roebber PJ, Schultz DM. Peer Review, Program Officers and Science Funding. PLoS One. 2011 Apr 12;6(4):e18680.

Russell AS, Thorn BD, Grace M. Peer Review: A Simplified Approach. J Rheumatol. 1983 Jun;10(3):479-81.

Sandewall E. Systems: Opening up the Process. Nature. 2006.

Sandstrom U, Hallsten M. Persistent Nepotism in Peer-Review. Scientometrics. 2008 Feb;74(2):175-89.

Schmaling, KB, Evenson, GR, Marble, BK, & Gallo, SA. “Perceptions of grant peer reviewers: a mixed methods study.” Research Evaluation, 2024; Volume 33 rvae050

Schroter S, Groves T, Højgaard L. Surveys of Current Status in Biomedical Science Grant Review: Funding Organisations’ and Grant Reviewers’ Perspectives. BMC Med. 2010 Oct 20;8:62.

Servick K. Bioinformatics. Top Contenders Blast Pentagon’s New Bioterror Detection Prize. Science. 2013 Aug 2;341(6145):449.

Shashok K. Content and Communication: How Can Peer Review Provide Helpful Feedback About the Writing? BMC Med Res Methodol. 2008 Jan 31;8:3.

Shideed O, Al-Gasseer N. Appraisal of the Research Grant Schemes of the World Health Organization Regional Office for the Eastern Mediterranean: The Way Forward. East Mediterr Health J. 2012 May;18(5):515-21.

Sieber J. Quality and Value: How Can We Research Peer Review? Nature 2006.

Smith MA, Kaufman NJ, Dearlove AJ. External Community Review Committee: A New Strategy for Engaging Community Stakeholders in Research Funding Decisions. Prog Community Health Partnersh. 2013 Fall;7(3):301-12.

Smith R. Classical Peer Review: An Empty Gun. Breast Cancer Res. 2010 Dec 20;12 Suppl 4:S13.

Smith R. Peer Review: A Flawed Process at the Heart of Science and Journals. J R Soc Med. 2006 Apr;99(4):178-82.

Spier R. The History of the Peer-Review Process. Trends Biotechnol. 2002 Aug;20(8):357-8.

Squazzoni F, Gandelli, C. Peer Review Under the Microscope. An Agent-Based Model of Scientific Collaboration. Simulation Conference (WSC), Proceedings of the 2012 Winter. Dec 2012:1-12.

Squazzoni F, Takacs, K. Social Simulation that “Peers into Peer Review.” J Artif Soc Soc Simulat. 2011 Oct 31;14(4):3.

Stahel PF, Moore EE. Peer Review for Biomedical Publications: We Can Improve the System. BMC Med. 2014 Sep 26;12(1):179.

Stehbens WE. Basic Philosophy and Concepts Underlying Scientific Peer Review. Med Hypotheses. 1999 Jan;52(1):31-6.

Turner RS. Best Practices in Peer Review Assure Quality, Value, Objectivity. Journal of the National Grants Management Association. 2009;17(1):43-8.

United States General Accounting Office. Federal Research: Peer Review Practices at Federal Science Agencies Vary. Report to Congressional Requesters. 2009 March.

Van de Sompel H. Technical Solutions: Certification in a Digital Era. Nature 2006.

Walker R, Rocha da Silva P. Emerging Trends in Peer Review-a Survey. Front Neurosci. 2015 May 27;9:169.

Wessely S. Peer Review of Grant Applications: What do we Know? Lancet. 1998 Jul 25;352(9124):301-5.

Zare RN, Winnacker EL. China’s Science Funding. Science. 2011 Oct 28;334(6055):433.

van Arensbergen P, van der Weijden I, van den Besselaar P. Academic Talent Selection in Grant Review Panels. In book: (Re)Searching Academic Careers, Chapter: 2. 2014. Publisher: Russian Academy of Sciences.

Enhancing the Postdoctoral Experience for Scientists and Engineers: A Guide for Postdoctoral Scholars, Advisers, Institutions, Funding Organizations, and Disciplinary Societies. National Academy of Sciences, National Academy of Engineering, Institute of Medicine. 2000.

Readers’ Poll Results: Funding Environment. Science. 2013 Apr 26;340(6131):429.

American Society for Biochemistry and Molecular Biology. Towards a Sustainable Biomedical Research Enterprise. 2014.

Bond Q, Gallin EK. Commentary: Recognizing and Tracking Philanthropy’s Critical Support of Health Research. Acad Med. 2012 Nov;87(11):1466-7.

Boudreau KJ, Eva G, Lakhani KR, Riedl C. Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance and Resource Allocation in Science. Available at SSRN. 2014.

Boyack KW, Klavans R, Borner K. Mapping the Backbone of Science. Scientometrics. 2005;64(3):351-74.

Buttliere BT. Using Science and Psychology to Improve the Dissemination and Evaluation of Scientific Work. Front Comput Neurosci. 2014 Aug 19;8:82.

Cohen LR, Noll RG. The Future of the National Laboratories. PNAS. 1996;93(23):12678-85.

Couzin-Frankel J. Chasing the Money. Science. 2014 Apr 4;344(6179):24-5.

Davis, G. Doctors Without Orders. American Scientist. 2005;93(3, supplement).

Furmana JL, Porter ME, Stern S. The Determinants of National Innovative Capacity. Res Policy. 2002;31(6):899-933.

Garrison HH, Deschamps AM. Physician Scientists: Assessing the Workforce. FASEB. 2013.

Garrison HH. Ngo K. Education and Employment of Biological and Medical Scientists 2011: Data from National Surveys. FASEB. 2012 Feb.

Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., & Kington, R. Race, ethnicity, and NIH research awards. Science. 2011;333(6045), 1015-1019.

Halffman W, Leydesdorff L. Is Inequality Among Universities Increasing? Gini Coefficients and the Elusive Rise of Elite Universities. Minerva. 2010 Mar;48(1):55-72.

Ioannidis JP. How to Make More Published Research True. PLoS Med. 2014 Oct 21;11(10):e1001747.

Lopes GR, da Silva R, Moro MM, Palazzo Moreira de Oliveira J. Scientific Collaboration in Research Networks: A Quantification Method by Using Gini Coefficient. IJCSA. 2012;9(2):15-31.

Muscio A, Quaglione D, Vallanti G. Does Government Research Funding to Universities Substitute, Complement or Leverage Industry Funding? Working Papers CELEG. 2010.

National Institute of Allergy and Infectious Diseases. FY 2014 Budget.

National Institute of Allergy and Infectious Diseases. The Edge of Discovery: A Portrait of the National Institute of Allergy and Infectious Diseases. June 2009;NIH Publication No. 09-6389.

Press WH. Presidential Address. What’s so Special About Science (and How Much Should We Spend on it?). Science. 2013 Nov 15;342(6160):817-22.

Rice JA, Wu CO. Nonparametric Mixed Effects Models for Unequally Sampled Noisy Curves. Biometrics. 2001 Mar;57(1):253-9.

Sullivan Faith K. Patterns of Creation and Discovery: An Analysis of Defense Laboratory Patenting and Innovation. 2013;dissertation:Pardee Rand Graduate School.

Taleb NN, Douady R. Mathematical Definition, Mapping, and Detection of (Anti)Fragility. Quant Finance. 2013;13(11):1677-89.

Van den Besselaar P, Leydesdorf L. Past Performance as Predictor of Successful Grant Applications: A Case Study. Den Haag, Rathenau Instituut SciSA rapport 0704.

Auranen O, Nieminen M. University Research Funding and Publication Performance–An International Comparison. Res Policy. 2010;39(6):822-34.

Barnett AG, Glisson SR, Gallo SA. “Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study.” f1000research.com.

Belter CW. Can Bibliometric Indicators Predict Institutional Citation Patterns? Presented at ACRL 2013, 10-13 April, Indianapolis IN.

Bornmann L, Daniel HD. The State of H Index Research. Is the H Index the Ideal Way to Measure Research Performance? EMBO Rep. 2009;10(1):2-6.

Bornmann L, Leydesdorff L. On the Meaningful and Non-Meaningful Use of Reference Sets in Bibliometrics. J Informetr. 2014;8(1):273-5.

Bornmann L, Marx W. How good is research really? Measuring the Citation Impact of Publications with Percentiles Increases Correct Assessments and Fair Comparisons. EMBO Rep. 2013 Mar 1;14(3):226-30.

Bornmann L, Mutz R, Neuhaus C, Daniel HD. Citation Counts for Research Evaluation: Standards of Good Practice for Analyzing Bibliometric Data and Presenting and Interpreting Results. ESEP. 2008; 8(1):93-102.

Bornmann L. Marx W. Methods for the Generation of Normalized Citation Impact Scores in Bibliometrics: Which Method Best Reflects the Judgements of Experts? 2014.

Bornmann, L, de Moya Anegon F, Leydesdorff L. The New Excellence Indicator in the World Report of the SCImago Institutions Rankings 2011. 2011.

Boyack KW, Chen MC, Chacko G. Characterization of the Peer Review Network at the Center for Scientific Review, National Institutes of Health. PLoS One. 2014 Aug 13;9(8):e104244.

Boyack KW, Jordan P. Metrics Associated with NIH Funding: A High-Level View. J Am Med Inform Assoc. 2011 Jul-Aug;18(4):423-31.

Cantor M, Gero S. The Missing Metric: Quantifying Contributions of Reviewers. R Soc Open Sci. 2015 Feb 11;2(2):140540.

Carter G. Peer Review, Citations, and Biomedical Research Policy: NIH Grants to Medical School Faculty. RAND Corporation, 1974.

Danthi NS, Wu CO, DiMichele DM, Hoots WK, Lauer MS. Citation Impact of NHLBI R01 Grants Funded Through the American Recovery and Reinvestment Act as Compared to R01 Grants Funded Through a Standard Payline. Circ Res. 2015 Feb 27;116(5):784-8.

Didegah F. Determinants of Research Citation Impact: A Combined Statistical Modelling.

Doyle JM, Quinn K, Bodenstein YA, Wu CO, Danthi N, Lauer MS. Association of Percentile Ranking with Citation Impact and Productivity in a Large Cohort of de novo NIMH-funded R01 Grants. Mol Psychiatry. 2015 Jun 2.

Farhadi H, Salehi H, Md Yunus M, Chadegani AA, Farhadi M, Fooladi M, Ebrahim NA. Does it Matter Which Citation Tool is Used to Compare the H-Index of a Group of Highly Cited Researchers? Aust J Basic Appl Sci. 2013;7(4):198-202.

Garfield E. Current Comments: Lifetime Citations Rates. Essays of an Information Scientist. 1980 Jan;4(2):355-8.

Gauffriau M, Larsen PO, Maye I, Roulin-Perriard A, von Ins M. Publication, Cooperation and Productivity Measures in Scientific Research. Scientometrics. 2007;73(2).175-214.

Ginther, D. K., Basner, J., Jensen, U., Schnell, J., Kington, R., & Schaffer, W. T. (2018). “Publications as predictors of racial and ethnic differences in NIH research awards.” PloS one, 13(11), e0205929.

Harzing AW. A Preliminary Test of Google Scholar as a Source for Citation Data: A Longitudinal Study of Nobel Prize Winners. Scientometrics. 2013;4(3):1057-75.

Hirsch JE. An Index to Quantify an Individual’s Scientific Research Output. Proc Natl Acad Sci U S A. 2005 Nov 15;102(46):16569-72.

Hirsch JE. Does the H Index Have Predictive Power? Proc Natl Acad Sci U S A. 2007 Dec 4;104(49):19193-8.

Ioannidis JP. Concentration of the Most-Cited Papers in the Scientific Literature: Analysis of Journal Ecosystems. PLoS One. 2006 Dec 20;1:e5.

Kaltman JR, Evans FJ, Danthi NS, Wu CO, DiMichele DM, Lauer MS. Prior Publication Productivity, Grant Percentile Ranking, and Topic-Normalized Citation Impact of NHLBI Cardiovascular R01 Grants. Circ Res. 2014 Sep 12;115(7):617-24.

Khan NR, Thompson CJ, Taylor DR, Gabrick KS, Choudhri AF, Boop FR, Klimo P Jr. Part II: Should the H-Index be Modified? An Analysis of the M-Quotient, Contemporary H-Index, Authorship Value, and Impact Factor. World Neurosurg. 2013 Dec;80(6):766-74.

Kinney AL. National Scientific Facilities and Their Science Impact on Nonbiomedical Research. Proc Natl Acad Sci U S A. 2007 Nov 13;104(46):17943-7.

Knoth P, Herrmannova D. Towards Semantometrics: A New Semantic Similarity Based Measure for Assessing a Research Publication’s Contribution. D-Lib Magazine. 2014;20(11/12).

Larivière V, Macaluso B, Archambault E, Gingras Y. Which Scientific Elite? On the Concentration of Funding, Productivity and Citations. In Proceedings of the 12th International Conference of the International Society for Scientometrics and Informetrics. 2009.

Lazaridis T. Ranking University Departments Using the Mean H-Index. Scientometrics. 2010 Feb;82(2):211-16.

Marx W, Schier H, Andersen OK. Using Time Dependent Citation Rates (Sales Curves) for Comparing Scientific Impacts. 2006. arXiv:physics/0611284 [physics.soc-ph].

Meho LI. Rogers I. Citation Counting, Citation Ranking, and H-Index of Human-Computer Interaction Researchers: A Comparison Between Scopus and Web of Science. J Am Soc Inf Sci Technol. 2008 Sep;59(11):1711-26.

Molinari JF, Molinari A. A New Methodology for Ranking Scientific Institutions. Scientometrics. 2008 Apr;75(1):163-74.

Nieminen P, Carpenter J, Rucker G, Schumacher M. The Relationship Between Quality of Research and Citation Frequency. BMC Med Res Methodol. 2006 Sep 1;6:42.

Opthof T, Leydesdorff L. Caveats for the Journal and Field Normalizations in the CWTS (“Leiden”) Evaluations of Research Performance. J Informetr. 2010;4(3):423-30.

Sidiropoulos A, Katsaros D, Manolopoulos Y. Generalized Hirsch H-Index for Disclosing Latent Facts in Citation Networks. Scientometrics. 2007 Aug 1;72(2):253-280.

Sypsa V, Hatzakis A. Assessing the Impact of Biomedical Research in Academic Institutions of Disparate Sizes. BMC Med Res Methodol. 2009 May 29;9:33.

Thomas Reuters. Using Bibliometrics: A Guide to Evaluating Research Performance with Citation Data. 2008.

Waltman L, van Eck NJ, van Leeuwen TN, Visser MS, van Raan AF. On the Correlation Between Bibliometric Indicators and Peer Review: Reply to Opthof and Leydesdorff. Scientometrics. 2011 Sep;88(3):1017-1022.

van Raan AFJ. Comparison of the Hirsch-Index with Standard Bibliometric Indicators and With Peer Iudgment for 147 Chemistry Research Groups. Scientometrics. 2006;67(3):491-502.

Boudreau KJ, Eva G, Lakhani KR, Riedl C. Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance and Resource Allocation in Science. Available at SSRN. 2014.

Bromham L, Dinnage R, and Hua X. “Interdisciplinary research has consistently lower funding success.” Nature 534, no. 7609 (2016): 684.

Chandrasekharan, S., Zaka, M., Gallo, S., Zhao, W., Korobskiy, D., Warnow, T., & Chacko, G. (2020). Finding scientific communities in citation graphs: Articles and authors. Quantitative Science Studies. Advance publication. https://doi.org/10.1162 /qss_a_00095

Gallo SA, Schmaling KB (2022) “Peer review: Risk and risk tolerance.” PLoS ONE 17(8): e0273813. https://doi.org/10.1371/journal.pone.0273813

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “Risk evaluation in peer review of grant applications.” Environment Systems and Decisions. 24 February, 2018.

Horrobin DF. The Philosophical Basis of Peer Review and the Suppression of Innovation. JAMA. 1990 Mar 9;263(10):1438-41.

Kaime EM, Moore KH, Goldberg SF. CDMRP: Fostering Innovation Through Peer Review. Technol Innov. 2010;12:233-40.

Kaplan D. Point: Statistical Analysis in NIH Peer Review–Identifying Innovation. FASEB J. 2007 Feb;21(2):305-8.

Luukkonen T. Conservatism and Risk-Taking in Peer Review: Emerging ERC Practices. Research Evaluation. 2012;21:48-60.

Nicholson J, Ioannidis J. Research Grants: Conform and be Funded. Nature. 2012 Dec 6;492:34-36.

Bornmann L, Mutz R, Daniel HD. A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and its Determinants. PLoS One. 2010 Dec 14;5(12):e14331.

Carpenter AS, Sullivan JH, Deshmukh A, Glisson SR, & Gallo SA. “A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels.” BMJ Open. 8 September, 2015.

Cicchetti DV. The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation. Behav Brain Sci. 1991;14:119-86.

Colliver J. Commentary on Cichetti’s “Reliability of Peer Review.” Teach Learn Med. 2002;14(3):142-3.

Fleiss JL, Shrout PE. The Effects of Measurement Errors on Some Multivariate Procedures. Am J Public Health. 1977 December; 67(12): 1188-91.

Fogelholm M, Leppinen S, Auvinen A, Raitanen J, Nuutinen A, Väänänen K. Panel Discussion Does Not Improve Reliability of Peer Review for Medical Research Grant Proposals. J Clin Epidemiol. 2012 Jan;65(1):47-52.

Gallo SA, Schmaling KB (2022) “Peer review: Risk and risk tolerance.” PLoS ONE 17(8): e0273813. https://doi.org/10.1371/journal.pone.0273813

Gallo SA, Schmaling KB, Thompson LA, and Glisson SR. “Grant Review Feedback: Appropriateness and Usefulness.” Science and Engineering Ethics. 27 (18) March, 2021.

Gallo SA, Sullivan JH, Glisson SR “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.” PLOS ONE. 21 October, 2016.

Giraudeau B, Leyrat C, Le Gouge A, Léger J, Caille A. Peer Review of Grant Applications: A Simple Method to Identify Proposals with Discordant Reviews. PLoS One. 2011;6(11):e27557.

Hallgren KA. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol. 2012;8(1):23-34.

Hargens LL, Herting JR. Neglected Considerations in the Analysis of Agreement Among Journal Referees. Scientometrics. 1990 Jul 01;19(1-2):91-106.

Hodgson C. How Reliable is Peer Review? An Examination of Operating Grant Proposals Simultaneously Submitted to Two Similar Peer Review Systems. J Clin Epidemiol. 1997 Nov;50(11):1189-95.

Jayasinghe UW, Marsh HW, Bond, N. A Multilevel Cross-Classified Modelling Approach to Peer Review of Grant Proposals: The Effects of Assessor and Researcher Attributes on Assessor Ratings. J R Statist Soc A. 2003;166(3):279-300.

Johnson VE. Statistical Analysis of the National Institutes of Health Peer Review System. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

Kravitz RL, Franks P, Feldman MD, Gerrity M, Byrne C, Tierney WM. Editorial Peer Reviewers’ Recommendations at a General Medical Journal: Are They Reliable and do Editors Care? PLoS One. 2010 Apr 8;5(4):e10072.

Marsh HW, Jayasinghe UW, Bond NW. Improving the Peer-Review Process for Grant Applications: Reliability, Validity, Bias, and Generalizability. Am Psychol. 2008 Apr;63(3):160-8.

Mutz R, Bornmann L, Daniel HD. Heterogeneity of Inter-Rater Reliabilities of Grant Peer Reviews and its Determinants: A General Estimating Equations Approach. PLoS One. 2012;7(10):e48509.

Pier EL, Raclaw J, Kaatz A, Brauer M, Carnes M, Nathan MJ, & Ford CE (2017). . “‘Your comments are meaner than your score’: score calibration talk influences intra-and inter-panel variability during scientific grant peer review.” Research Evaluation, 26(1), 1-14.

Pier, E.L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J., Nathan, M.J., Ford, C.E. and Carnes, M., 2018. “Low agreement among reviewers evaluating the same NIH grant applications.”Proceedings of the National Academy of Sciences, 115(12), pp.2952-2957.

Rothwell PM, Martyn CN. Reproducibility of Peer Review in Clinical Neuroscience. Is Agreement Between Reviewers Any Greater Than Would be Expected by Chance Alone? Brain. 2000 Sep;123 (Pt 9):1964-9.

Sattler DN, McKnight PE, Naney L, Mathis R. Grant Peer Review: Improving Inter-Rater Reliability with Training. PLoS One. 2015 Jun 15;10(6):e0130450.

Berg J. Scoring Analysis with Funding and Investigator Status. NIGMS Feedback Loop Blog. 2010 Sep 14.

Berg J. NIH-Wide Correlations Between Overall Impact Scores and Criterion Scores. NIGMS Feedback Loop Blog. 2010 Sep 30.

Bielski A, Harris R, Gillis N. Summary Report of Comments Received on NIH System to Support Biomedical and Behavioral Research and Peer Review. 2007 November.

Collins FS, Tabak LA. Policy: NIH Plans to Enhance Reproducibility. Nature. 2014 Jan 30;505(7485):612-3.

Gallo SA, Schmaling KB, Thompson LA, and Glisson SR. “Grant Review Feedback: Appropriateness and Usefulness.” Science and Engineering Ethics. 27 (18) March, 2021.

Gillespie GW, Jr., Chubin DE, Kurzon GM. Experience with NIH Peer Review: Researchers’ Cynicism and Desire for Change. Sci Technol Human Values. 1985;10(3):45-54.

Ginther, D. K., Basner, J., Jensen, U., Schnell, J., Kington, R., & Schaffer, W. T. (2018). “Publications as predictors of racial and ethnic differences in NIH research awards.” PloS one, 13(11), e0205929.

Hoppe TA, Litovitz A, Willis KA, Meseroll RA, Perkins MJ, Hutchins BI, Davis AF, Lauer MS, Valantine HA, Anderson JM, Santangelo GM. “Topic choice contributes to the lower rate of NIH awards to African-American/black scientists.” Science Advances. 2019 Oct 1;5(10):eaaw7238.

Kaiser J. Biomedical Research. A Call for NIH Youth Movement. Science. 2014 Oct 10;346(6206):150-1.

Magua W, Zhu X, Bhattacharya A, Filut A, Potvien A, Leatherberry R, Lee YG, Jens M, Malikireddy D, Carnes M, Kaatz A. “Are female applicants disadvantaged in National Institutes of Health peer review? Combining algorithmic text mining and qualitative methods to detect evaluative differences in R01 reviewers’ critiques.” Journal of Women’s Health. 2017 May 1;26(5):560-70.

Moore RF, Catevenis K, Pearson K, Wagner RM. AIRI Statistics: Trends in NIH Extramural Funding. Presented at the Association of Independent Research Institutes (AIRI) 51st Annual Meeting. 2012, 3 Oct, Philadelphia, PA.

NCI. The NCI Consumers’ Guide to Peer Review. 2009 Nov.

NIH. Center for Scientific Review Advisory Council Meeting. October 25, 2011.

NIH. Enhancing Peer Review: Implementation of Recommended Actions. Transcript accompanying video of Dr. Alan Willard. 2009 Feb.

NIH. A Self Study by the NIH in Partnership with the Scientific Community to Strengthen Peer Review in Changing Times. 2008.

NIH. 2007-2008 Peer Review Self-Study. 2008.

NIH. Scoring and Review Changes. 2009 Oct 22.

NIH. NIH Peer Review: Grants and Cooperative Agreements. 2013.

NIH. Encouraging Early Transition to Research Independence: Modifying the NIH New Investigator Policy to Identify Early Stage Investigators. NOT-OD-08-121. 2008 Sep 26.

NIH. Enhancing Peer Review: The NIH Announces Enhanced Review Criteria for Evaluation of Research Applications Received for Potential FY2010 Funding. NOT-OD-09-025. 2008 Dec 2.

NIH. Revised Policy: Managing Conflict of Interest in the Initial Peer Review of NIH Grant and Cooperative Agreement Applications. NOT-OD-11-120. 2011 Sep 26.

NIH. Enhancing Peer Review: The NIH Announces New Scoring Procedures for Evaluation of Research Applications Received for Potential FY2010 Funding. NOT-OD-09-024. 2008 Dec 2.

NIH. Research Project Grant (RPG) Critique Template. R01/R03/R15/R21/R34 Review. 2012 Mar 5.

NIH. Chair Orientation. 2012 Mar 5.

NIH. Side-by-side comparison of enhanced and former review criteria. 2009 Dec 1.

NIH. Overall Impact Versus Significance. 2012 Mar 5.

NIH. NIH Reviewer Orientation. 2012 Dec 14.

NIH. Managing Conflict of Interest in NIH Peer Review of Grants and Contracts.

NIH. Department of Health and Humans National Institutes of Health FY 2013 Budget. 2012.

NIH. Evaluation Summary of New 1-9 Scoring System. 2009 Oct 13.

NIH. Moving Forward After an Unsuccessful A1 Application. 2012 May.

NIH. NIH Success Rate Definition. 2009 Feb.

NIH. R01 Guide for Reviewers. Investigator Initiated Research Project Grant Applications. 2010 Feb 2.

NIH. NIH Success Rate Definition. 2012 Feb.

NIH. Instructions for Use of the Review Critique Template. 2009 Apr 21.

NIH. Scoring System and Procedure. 2009 Mar 20.

NSF. National Science Foundation’s Merit Review Criteria: Review and Revisions. 2011 Dec 14.

Tabak LW. Report on Enhancing Peer Review at NIH Implementation Plan. NIH. 2008 Jun 6.

Bornmann L, Wolf M, Daniel HD. Closed Versus Open Reviewing of Journal Manuscripts: How Far do Comments Differ in Language Use? Scientometrics. 2012;91(3):843-56.

DeCoursey T. Perspective: The Pros and Cons of Open Peer Review. Nature 2006.

Hettyey A, Griggio M, Mann M, Raveh S, Schaedelin FC, Thonhauser KE, Thoss M, van Dongen WF, White J, Zala SM, Penn DJ. Peerage of Science: Will it Work? Trends Ecol Evol. 2012 Apr;27(4):189-90.

Koop T, Pöschl U. Systems: An Open, Two-Stage Peer-Review Journal. Nature. 2006.

Mietchen D. The Transformative Nature of Transparency in Research Funding. PLoS Biol. 2014 Dec 30;12(12):e1002027.

Moylan EC, Harold S, O’Neill C, Kowalczuk MK. Open, Single-Blind, Double-Blind: Which Peer Review Process do You Prefer? BMC Pharmacol Toxicol. 2014 Sep 30;15:55.

Stehbens WE. Basic Philosophy and Concepts Underlying Scientific Peer Review. Med Hypotheses. 1999 Jan;52(1):31-6.

Walker R, Rocha da Silva P. Emerging Trends in Peer Review-a Survey. Front Neurosci. 2015 May 27;9:169.

Carpenter AS, Sullivan JH, Deshmukh A, Glisson SR, & Gallo SA. “A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels.” BMJ Open. 8 September, 2015.

Cole S, Cole JR, Simon GA. Chance and Consensus in Peer Review. Science. 1981 Nov 20;214(4523):881-6.

Fogelholm M, Leppinen S, Auvinen A, Raitanen J, Nuutinen A, Väänänen K. Panel Discussion Does Not Improve Reliability of Peer Review for Medical Research Grant Proposals. J Clin Epidemiol. 2012 Jan;65(1):47-52.

Gallo, S.A., Schmaling, K.B., Thompson, L.A., and Glisson, S.R. Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion. Res Integr Peer Rev. 2020; 5:(7).

Johnson VE. Statistical Analysis of the National Institutes of Health Peer Review System. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

Martin MR, Kopstein A, Janice JM. An Analysis of Preliminary and Post-Discussion Priority Scores for Grant Applications Peer Reviewed by the Center for Scientific Review at the NIH. PLoS One. 2010 Nov 17;5(11):e13526.

Obrecht M, Tibelius K, D’Aloisio G. Examining the Value Added by Committee Discussion in the Review of Applications for Research Awards. Research Evaluation. 2007 Jun;16(2):79-91.

Pier EL, Raclaw J, Kaatz A, Brauer M, Carnes M, Nathan MJ, & Ford CE (2017). . “‘Your comments are meaner than your score’: score calibration talk influences intra-and inter-panel variability during scientific grant peer review.” Research Evaluation, 26(1), 1-14.

Akerman R. Technical Solutions: Evolving Peer Review for the Internet. Nature 2006.

Anderson C. Technical Solutions: Wisdom of the Crowds. Nature 2006.

Bloom T. Systems: Online Frontiers of the Peer-Reviewed Literature. Nature. 2006.

Bohannon J. National Science Foundation. Meeting for Peer Review at a Resort That’s Virtually Free. Science. 2011 Jan 7;331(6013):27.

Carpenter AS, Sullivan JH, Deshmukh A, Glisson SR, & Gallo SA. “A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels.” BMJ Open. 8 September, 2015.

Doran MR, Lott WB, Doran SE. Communication: Use Multimedia in Grant Applications. Nature. 2014 Jan 16;505(7483):291.

Driskell JE, Radtke PH, Salas E. Virtual Teams: Effects of Technological Mediation on Team Performance. Group Dyn. 2003 Dec;7(4):297-323.

Gallo SA, Carpenter AS, Glisson SR (2013) “Teleconference Versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes.” PLoS ONE 8(8).

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey..” Sci Eng Ethics. July, 2019. (Pre-print available)

Jenkins SL, Iyengar R, Diverse-Pierluissi MA, Chan AM, Devi LA, Sobie EA, Ting AT, Weinstein DC. Teaching Resources. Using Web-Based Discussion Forums as a Model of the Peer-Review Process and a Tool for Assessment. Sci Signal. 2008;1(9):tr2.

Vo, NM, Quiggle, GM, Wadhwani, K. Comparative outcomes of face-to-face and virtual review meetings. International Journal of Surgery Open. 2016. Volume 4 , 38 - 41

Vo, NM, Trocki, R. Virtual and Peer Reviews of Grant Applications at the Agency for Healthcare Research and Quality. Southern Medical Journal. 2015. 108. 622-626.

Bornmann L, Mutz R, Daniel H-R. Latent Markov Modeling Applied to Grant Peer Review. J Informetr. 2008;2(4):217-28.

Boyington, J. E., Antman, M. D., Patel, K. C., & Lauer, M. S. (2016). “Towards Independence: Resubmission Rate of Unfunded National Heart, Lung, and Blood Institute R01 Research Grant Applications among Early Stage Investigators.” Academic medicine: journal of the Association of American Medical Colleges, 91(4), 556.

Kolehmainen C, & Carnes M (2018). “Who resembles a scientific leader—Jack or Jill? How implicit bias could influence research grant funding.” Circulation, 137(8), 769-770.

Gallo SA, Schmaling KB, Thompson LA, and Glisson SR. “Grant Review Feedback: Appropriateness and Usefulness.” Science and Engineering Ethics. 27 (18) March, 2021.

Gallo SA, Sullivan JH, Glisson SR “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.” PLOS ONE. 21 October, 2016.

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey..” Sci Eng Ethics. July, 2019. (Pre-print available)

Ginther, D. K., Basner, J., Jensen, U., Schnell, J., Kington, R., & Schaffer, W. T. (2018). “Publications as predictors of racial and ethnic differences in NIH research awards.” PloS one, 13(11), e0205929.

Hayes M Grant Review in focus.” Publons 2019

Jones BF, Weinberg BA. Age Dynamics in Scientific Creativity. Proc Natl Acad Sci U S A. 2011 Nov 22;108(47):18910-4.

Kaplan D, Lacetera N, Kaplan C. Sample Size and Precision in NIH Peer Review. PLoS One. 2008 Jul 23;3(7):e2761.

Zerhouni E. Some Observations on Demographics of NIH-Funded Scientists: Policy Implications for New Investigators. 2007 Dec 7.

Cicchetti DV, Shoinralter D, Tyrer PJ. The Effect of Number of Rating Scale Categories on Levels of Interrater Reliability: A Monte Carlo Investigation. Appl Psychol Meas. 1985 Mar;9(1):31-6.

Gallo, SA, Pearce, M, Lee, CJ & Erosheva, EA. “A new approach to grant review assessments: score, then rank.” Res Integr Peer Rev. 2023; 8, 10.

Green JG, Calhoun F, Nierzwicki L, Brackett J, Meier P. Rating Intervals: An Experiment in Peer Review. FASEB J. 1989 Jun;3(8):1987-92.

NIH. Evaluation Summary of New 1-9 Scoring System. 2009 Oct 13.

Barabási AL. Sociology. Network Theory–the Emergence of the Creative Enterprise. Science. 2005 Apr 29;308(5722):639-41.

Borgatti SP, Mehra A, Brass DJ, Labianca G. Network Analysis in the Social Sciences. Science. 2009 Feb 13;323(5916):892-95.

Carpenter AS, Sullivan JH, Deshmukh A, Glisson SR, & Gallo SA. “A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels.” BMJ Open. 8 September, 2015.

Cooke, Nancy J., and Margaret L. Hilton, eds. “Enhancing the effectiveness of team science.” Washington, DC: National Academies Press, 2015.

Cur?eu PL, Jansen RJ, Chappin MM. Decision Rules and Group Rationality: Cognitive Gain or Standstill? PLoS One. 2013;8(2):e56454.

Driskell JE, Radtke PH, Salas E. Virtual Teams: Effects of Technological Mediation on Team Performance. Group Dyn. 2003 Dec;7(4):297-323.

Duch J, Waitzman JS, Amaral LA. Quantifying the Performance of Individual Players in a Team Activity. PLoS One. 2010 Jun 16;5(6):e10937.

Evans JA, Foster JG. Metaknowledge. Science. 2011 Feb 11;331(6018):721-5.

Guimerà R, Uzzi B, Spiro J, Amaral LA. Team Assembly Mechanisms Determine Collaboration Network Structure and Team Performance. Science. 2005 Apr 29;308(5722):697-702.

Jeffcott SA, Mackenzie CF. Measuring Team Performance in Healthcare: Review of Research and Implications for Patient Safety. J Crit Care. 2008 Jun;23(2):188-96.

Langfeldt L. Decision-Making in Expert Panels Evaluating Research: Constraints, Processes and Bias. 2001;dissertation: The University of Oslo.

Manthous CA, Hollingshead AB. Team Science and Critical Care. Am J Respir Crit Care Med. 2011 Jul 1;184(1):17-25.

Olbrecht M, Bornmann L. Panel Peer Review of Grant Applications: What Do We Know From Research in Social Psychology On Judgment and Decision-Making in Groups? Res Eval. 2010;19 (4):293-304.

Roebber PJ, Schultz DM. Peer Review, Program Officers and Science Funding. PLoS One. 2011 Apr 12;6(4):e18680.

Rogelberg SG, O’Connor MS, Sederburg M. Using the Stepladder Technique to Facilitate the Performance of Audioconferencing Groups. J Appl Psychol. 2002 Oct;87(5):994-1000.

Rosen MA, Weaver SJ, Lazzara EH, Salas E, Wu T, Silvestri S, Schiebel N, Almeida S, King HB. Tools for Evaluating Team Performance in Simulation-Based Training. J Emerg Trauma Shock. 2010 Oct;3(4):353-9.

Stokols D, Misra S, Moser RP, Hall KL, Taylor BK. The Ecology of Team Science: Understanding Contextual Influences on Transdisciplinary Collaboration. Am J Prev Med. 2008 Aug;35(2 Suppl):S96-115.

Wickens CD, Holland JG, Parasuraman R, Banbury S. Engineering Psychology and Human Performance (4th ed). Upper Saddle River, NJ: Prentice Hall, 2012.

Garfield JM, Kaye AD, Kolinsky DC, Urman RD. A Systematic Guide for Peer Reviewers for a Medical Journal. J Med Pract Manage. 2015 Mar-Apr;30(6 Spec No):13-7.

Rosen MA, Weaver SJ, Lazzara EH, Salas E, Wu T, Silvestri S, Schiebel N, Almeida S, King HB. Tools for Evaluating Team Performance in Simulation-Based Training. J Emerg Trauma Shock. 2010 Oct;3(4):353-9.

Sattler DN, McKnight PE, Naney L, Mathis R. Grant Peer Review: Improving Inter-Rater Reliability with Training. PLoS One. 2015 Jun 15;10(6):e0130450.

Bornmann L, Mutz R, Daniel H-R. Latent Markov Modeling Applied to Grant Peer Review. J Informetr. 2008;2(4):217-28.

Vener KJ, Feuer EJ, Gorelic L. A Statistical Model Validating Triage for the Peer Review Process: Keeping the Competitive Applications in the Review Pipeline. FASEB J. 1993 Nov;7(14):1312-9.

Armstrong PW, Caverson MM, Adams L, Taylor M, Olley PM. Evaluation of the Heart and Stroke Foundation of Canada Research Scholarship Program: Research Productivity and Impact. Can J Cardiol. 1997 May;13(5):507-16.

Beacham L, Li E, Wasserman A. Measuring the Performance of Extramural Funding at the National Library of Medicine. The College of William and Mary. 2008.

Berg J. Productivity Metrics and Peer Review Scores. NIGMS Feedback Loop Blog. 2011 Jun 2.

Bollen, J., Crandall, D., Junk, D., Ding, Y., & Börner, K. (2014). “From funding agencies to scientific agency.” EMBO reports, 15(2), 131-133.

Bornmann L, Daniel HD. Selection of Research Fellowship Recipients by Committee Peer Review. Reliability, Fairness and Predictive Validity of Board of Trustees’ Decisions. Scientometrics. 2005;63(2):297-320.

Bornmann L, Daniel HD. The Usefulness of Peer Review for Selecting Manuscripts for Publication: A Utility Analysis Taking as an Example a High-Impact Journal. PLoS One. 2010 Jun 28;5(6):e11344.

Bornmann L, Wallon G, Ledin A. Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes. PLoS One. 2008;3(10):e3480.

Bornmann, L, Daniel H-D. Reliability, Fairness and Predictive Validity of Committee Peer Review: Evaluation of the Selection of Post-Graduate Fellowship Holders by the Boehringer Ingelheim Fonds. B.I.F. FUTURA. 2004;19.

Burroughs Wellcome Fund. Biomedical Sciences: Program Outcomes. 2010 Report.

Cabezas-Clavijo A, Robinson-García N, Escabias M, Jiménez-Contreras E. Reviewers’ Ratings and Bibliometric Indicators: Hand in Hand When Assessing Over Research Proposals? PLoS One. 2013 Jun 28;8(6):e68258.

Claveria LE, Guallar E, Cami J, Conde J, Pastor R, Ricoy JR, Rodriguez-Farre E, Ruiz-Palomo F, Munoz E. Does Peer Review Predict the Performance of Research Projects in Health Sciences? Scientometrics. 2000 Jan 1;47(1):11-23.

Cole S, Cole JR, Simon GA. Chance and Consensus in Peer Review. Science. 1981 Nov 20;214(4523):881-6.

Crespia GA, Geunab A. An Empirical Study of Scientific Production: A Cross Country Analysis, 1981-2002. Res Policy. 2008;37(4):565-79.

Danthi N, Wu CO, Shi P, Lauer M. Percentile Ranking and Citation Impact of a Large Cohort of National Heart, Lung, and Blood Institute-Funded Cardiovascular R01 Grants. Circ Res. 2014 Feb 14;114(4):600-6.

Danthi NS, Wu CO, DiMichele DM, Hoots WK, Lauer MS. Citation Impact of NHLBI R01 Grants Funded Through the American Recovery and Reinvestment Act as Compared to R01 Grants Funded Through a Standard Payline. Circ Res. 2015 Feb 27;116(5):784-8.

Doyle JM, Quinn K, Bodenstein YA, Wu CO, Danthi N, Lauer MS. Association of Percentile Ranking with Citation Impact and Productivity in a Large Cohort of de novo NIMH-funded R01 Grants. Mol Psychiatry. 2015 Jun 2.

Eloy JA, Svider PF, Kanumuri VV, Folbe AJ, Setzen M, Baredes S. Do AAO-HNSF CORE Grants Predict Future NIH Funding Success? Otolaryngol Head Neck Surg. 2014 May 20. [Epub ahead of print]

Escobar-Alvarez SN, Myers ER. The Doris Duke Clinical Scientist Development Award: Implications for Early-Career Physician Scientists. Acad Med. 2013 Nov;88(11):1740-6.

Fang D, Meyer RE. Effect of Two Howard Hughes Medical Institute Research Training Programs for Medical Students on the Likelihood of Pursuing Research Careers. Acad Med. 2003 Dec;78(12):1271-80.

Fang, F.C. and Casadevall, A., 2016. “Research Funding: the Case for a Modified Lottery.” mBio, 7(2).

Gallo SA et al. “The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies.” 2014 PLoS ONE 9(9): e106474. doi:10.1371/journal.pone.0106474

Goldbeck-Wood S. Evidence on Peer Review–Scientific Quality Control or Smokescreen? BMJ. 1999 January 2;318(7175):44-5.

Hodgson C. Evaluation of Cardiovascular Grant-in-aid Applications by Peer Review: Influence of Internal and External Reviewers and Committees. Can J Cardiol. 1995 Nov;11(10):864-8.

Holbrook JB, Hrotic S. Blue Skies, Impacts, and Peer Review. RT. 2013 Jul;1(1):2013.

Hoppe TA, Litovitz A, Willis KA, Meseroll RA, Perkins MJ, Hutchins BI, Davis AF, Lauer MS, Valantine HA, Anderson JM, Santangelo GM. “Topic choice contributes to the lower rate of NIH awards to African-American/black scientists.” Science Advances. 2019 Oct 1;5(10):eaaw7238.

Ip EH, Wasserman R, Barkin S. Comparison of Intraclass Correlation Coefficient Estimates and Standard Errors Between Using Cross-Sectional and Repeated Measurement Data: The Safety Check Custer Randomized Trial. Contemp Clin Trials. 2011 Mar;32(2):225-32.

Jackson JL, Srinivasan M, Rea J, Fletcher KE, Kravitz RL. The Validity of Peer Review in a General Medicine Journal. PLoS One. 2011;6(7):e22475.

Jacob B, Lefgren L. The Impact of Research Grant Funding on Scientific Productivity. NBER Working Paper No. 13519. 2007.

Kaltman JR, Evans FJ, Danthi NS, Wu CO, DiMichele DM, Lauer MS. Prior Publication Productivity, Grant Percentile Ranking, and Topic-Normalized Citation Impact of NHLBI Cardiovascular R01 Grants. Circ Res. 2014 Sep 12;115(7):617-24.

Kotchen TA, Lindquist T, Miller Sostek A, Hoffmann R, Malik K, Stanfield B. Outcomes of National Institutes of Health Peer Review of Clinical Grant Applications. J Investig Med. 2006 Jan;54(1):13-9.

Lauer MS. Thought Exercises on Accountability and Performance Measures at the National Heart, Lung, and Blood Institute (NHLBI): An Invited Commentary for Circulation Research. Circ Res. 2011 Feb 18;108(4):405-9.

Laurence WF, Useche C, Laurance SG, Bradshaw CJA. Predicting Publication Success for Biologists. BioScience. 2013;63(10):817-23.

Li D, Agha L. Research Funding. Big Names or Big Ideas: Do Peer-Review Panels Select the Best Science Proposals? Science. 2015 Apr 24;348(6233):434-8.

Lindner MD, Nakamura RK. Examining the Predictive Validity of NIH Peer Review Scores. PLoS One. 2015 Jun 3;10(6):e0126938.

Marsh HW, Jayasinghe UW, Bond NW. Improving the Peer-Review Process for Grant Applications: Reliability, Validity, Bias, and Generalizability. Am Psychol. 2008 Apr;63(3):160-8.

Martin MR, Lindquist T, Kotchen TA. Why are Peer Review Outcomes Less Favorable for Clinical Science than for Basic Science Grant Applications? Am J Med. 2008 Jul;121(7):637-41.

Mason JL, Lei M, Faupel-Badger JM, Ginsburg EP, Seger YR, Dijoseph L, Schnell JD, Wiest JS. Outcome Evaluation of the National Cancer Institute Career Development Awards Program. J Cancer Educ. 2013 Mar;28(1):9-17.

Mervis J. Peering Into Peer Review. Science. 2014 Feb 7;343(6171):596-8.

Moed HF, Glanzel W, Schmoch U. Handbook of Quantitative Science and Technology Research: The Use of Publications and Patent Statistics in Studies of S&T Systems. New York, NY: Kluwer Academic Publishers, 2004.

Pion G, Ionescu-Pioggia M. Bridging Postdoctoral Training and a Faculty Position: Initial Outcomes of the Burroughs Wellcome Fund Career Awards in the Biomedical Siences. Acad Med. 2003 Feb;78(2):177-86.

Rinia EJ, Leeuwen Th N, Vuren HG, Raan AFJ. Influence of Interdisciplinarity on Peer-Review and Bibliometric Evaluations in Physics Research. Research Policy. 2001 Mar 1;30(3):357-61.

Rinia EJ, Leeuwen Th N, Vuren HG, Raan AFJ. Comparative Analysis of a Set of Bibliometric Indicators and Central Peer Review Criteria: Evaluation of Condensed Matter Physics in the Netherlands. Research Policy. 1998 May;27(1):95-107.

Schmaling, KB, Evenson, GR, Marble, BK, & Gallo, SA. “Perceptions of grant peer reviewers: a mixed methods study.” Research Evaluation, 2024; Volume 33 rvae050

Schroter S, Tite L, Hutchings A, Black N. Differences in Review Quality and Recommendations for Publication Between Peer Reviewers Suggested by Authors or by Editors. JAMA. 2006 Jan 18;295(3):314-7.

Shashok K. Standardization vs Diversity: How Can We Push Peer Review Research Forward? MedGenMed. 2005 Feb 17;7(1):11.

Star Metrics. Science and Technology for America’s Reinvestment: Measuring the Effect of Research on Innovation, Competitiveness and Science.

Stephen A. Gallo, Scott R. Glisson. “External Tests of Peer Review Validity Via Impact Measures.” Front. Res. Metr. Anal. 23 August 2018

Turner J. Best Practices in Merit Review: A Report to the US Department of Energy. 2010 Dec.

Vale RD. Evaluating How We Evaluate. Mol Biol Cell. 2012 Sep;23(17):3285-9.

Waltman L, van Eck NJ, van Leeuwen TN, Visser MS, van Raan AF. On the Correlation Between Bibliometric Indicators and Peer Review: Reply to Opthof and Leydesdorff. Scientometrics. 2011 Sep;88(3):1017-1022.

Yglesias E. Improving Peer Review in the Federal Government. Technol Innov. 2010 Mar;12(3):225-32.