Expert review is critical for strong science

Our work not only benefits and informs research program decisions, it also strengthens the integrity of the science that will benefit society.

Credit: Michal Jarmoluk

A small semi-transparent triangle for visual interest

Our Bibliography of Peer Review Literature

The peer review process is used by organizations around the world to identify good, promising science. However, peer review can be implemented in many ways, and surprisingly, dialogue among organizations managing peer review is limited. Although literature about peer review (and particularly peer review of grant applications) is scant, there have been some studies addressing review practices, challenges and guidelines.

Compiled below is a reference list of 363 scientific articles, sorted into 23 categories, concerning the processes, outcomes and validations of the scientific peer review process, particularly focusing on peer review of applications for funding. This list is sorted by topic area and represents a centralized body of knowledge intended as a reference for those conducting/participating in peer review and those who are interested in the outcomes of the process.

Aibs Papers Up

Carpenter AS, Sullivan JH, Deshmukh A, Glisson SR, & Gallo SA. “A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels.” BMJ Open. 8 September, 2015.

Chandrasekharan, S., Zaka, M., Gallo, S., Zhao, W., Korobskiy, D., Warnow, T., & Chacko, G. (2020). Finding scientific communities in citation graphs: Articles and authors. Quantitative Science Studies. Advance publication. https://doi.org/10.1162 /qss_a_00095

Gallo SA et al. “The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies.” 2014 PLoS ONE 9(9): e106474. doi:10.1371/journal.pone.0106474

Gallo SA, Schmaling KB, Thompson LA, and Glisson SR. “Grant Review Feedback: Appropriateness and Usefulness.Science and Engineering Ethics. 27 (18) March, 2021.

Gallo SA, Sullivan JH, Glisson SR “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.” PLOS ONE. 21 October, 2016.

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey..” Sci Eng Ethics. July, 2019. (Pre-print available)

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “Risk evaluation in peer review of grant applications.” Environment Systems and Decisions. 24 February, 2018.

Gallo, S.A., Schmaling, K.B., Thompson, L.A., and Glisson, S.R. Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion. Res Integr Peer Rev. 2020; 5:(7).

Irwin D, Gallo SA, Glisson SR. “Opinion: Learning from Peer Review.” The Scientist. 2013 May 24.

Stephen A. Gallo, Scott R. Glisson. “External Tests of Peer Review Validity Via Impact Measures.” Front. Res. Metr. Anal. 23 August 2018

Assignment Load Up

Cook,WD, Golany B, Kress M, Penn M. Optimal Allocation of Proposals to Reviewers to Facilitate Effective Ranking. Manage Sci. 2005;51(4):655-61.

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey..” Sci Eng Ethics. July, 2019. (Pre-print available)

Li L, Wang Y, Liu G, Wang M, Wu X. Context-Aware Reviewer Assignment for Trust Enhanced Peer Review. PLoS One. 2015 Jun 19;10(6):e0130493.

Snell, R. R. Menage a quoi? Optimal number of peer reviewers. PloS One 2015; 10(4), e0120838

Conflict of Interest and Bias Up

Abdoul H, Perrey C, Tubach F, Amiel P, Durand-Zaleski I, Alberti C. Non-financial Conflicts of Interest in Academic Grant Evaluation: A Qualitative Study of Multiple Stakeholders in France. PLoS One. 2012;7(4):e35247.

Bekelman JE, Li Y, Gross CP. Scope and Impact of Financial Conflicts of Interest in Biomedical Research: A Systematic Review. JAMA. 2003 Jan 22-29;289(4):454-65.

Benos D. Ethics: Detecting Misconduct. Nature 2006.

Bhattacharjee Y. Science Funding. NSF’s ‘Big Pitch’ Tests Anonymized Grant Reviews. Science. 2012 May 25;336(6084):969-70.

Bornmann L. Research Misconduct–Definitions, Manifestations and Extent. Publications. 2013;1(3):87-98.

Bromham L, Dinnage R, and Hua X. “Interdisciplinary research has consistently lower funding success.” Nature 534, no. 7609 (2016): 684.

Chandrasekharan, S., Zaka, M., Gallo, S., Zhao, W., Korobskiy, D., Warnow, T., & Chacko, G. (2020). Finding scientific communities in citation graphs: Articles and authors. Quantitative Science Studies. Advance publication. https://doi.org/10.1162 /qss_a_00095

De Martino B, Kumaran D, Seymour B, Dolan RJ. Frames, Biases, and Rational Decision-Making in the Human Brain. Science. 2006 Aug 4;313(5787):684-7.

Drazen JM, De Leeuw PW, Laine C, Mulrow C, Deangelis CD, Frizelle FA, Godlee F, Haug C, Hébert PC, James A, Kotzin S, Marusic A, Reyes H, Rosenberg J, Sahni P, Van Der Weyden MB, Zhaori G. Toward More Uniform Conflict Disclosures: The Updated ICMJE Conflict of Interest Reporting Form. Natl Med J India. 2010 Jul-Aug;23(4):196-7.

Gallo SA, Schmaling KB, Thompson LA, and Glisson SR. “Grant Review Feedback: Appropriateness and Usefulness.Science and Engineering Ethics. 27 (18) March, 2021.

Gallo SA, Sullivan JH, Glisson SR “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.” PLOS ONE. 21 October, 2016.

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “Risk evaluation in peer review of grant applications.” Environment Systems and Decisions. 24 February, 2018.

Ginther, D. K., Basner, J., Jensen, U., Schnell, J., Kington, R., & Schaffer, W. T. (2018). “Publications as predictors of racial and ethnic differences in NIH research awards.” PloS one, 13(11), e0205929.

Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., & Kington, R. Race, ethnicity, and NIH research awards. Science. 2011;333(6045), 1015-1019.

Hoppe TA, Litovitz A, Willis KA, Meseroll RA, Perkins MJ, Hutchins BI, Davis AF, Lauer MS, Valantine HA, Anderson JM, Santangelo GM. “Topic choice contributes to the lower rate of NIH awards to African-American/black scientists.” Science Advances. 2019 Oct 1;5(10):eaaw7238.

Kolehmainen C, & Carnes M (2018). “Who resembles a scientific leader—Jack or Jill? How implicit bias could influence research grant funding.” Circulation, 137(8), 769-770.

Kotchen TA, Lindquist T, Miller Sostek A, Hoffmann R, Malik K, Stanfield B. Outcomes of National Institutes of Health Peer Review of Clinical Grant Applications. J Investig Med. 2006 Jan;54(1):13-9.

Langfeldt L. Decision-Making in Expert Panels Evaluating Research: Constraints, Processes and Bias. 2001;dissertation: The University of Oslo.

Lee C. Sugimoto CR, Zhang G, Cronin B. Bias in Peer Review. J Am Soc Inf Sci Technol. 2013;64(1):2-17.

Li D. Expertise vs. Bias in Evaluation: Evidence from the NIH. 2013;Northwestern University.

Li L, Wang Y, Liu G, Wang M, Wu X. Context-Aware Reviewer Assignment for Trust Enhanced Peer Review. PLoS One. 2015 Jun 19;10(6):e0130493.

Lo B, Field MJ, Editors. Conflict of Interest in Medical Research, Education, and Practice. Committee on Conflict of Interest in Medical Research, Education, and Practice. 2009;Institute of Medicine

Magua W, Zhu X, Bhattacharya A, Filut A, Potvien A, Leatherberry R, Lee YG, Jens M, Malikireddy D, Carnes M, Kaatz A. “Are female applicants disadvantaged in National Institutes of Health peer review? Combining algorithmic text mining and qualitative methods to detect evaluative differences in R01 reviewers’ critiques.” Journal of Women’s Health. 2017 May 1;26(5):560-70.

Oleinik A. Conflict(s) of Interest in Peer Review: Its Origins and Possible Solutions. Sci Eng Ethics. 2014 Mar;20(1):55-75.

Pham-Kanter G. Revisiting Financial Conflicts of Interest in FDA Advisory Committees. Milbank Q. 2014 Sep;92(3):446-70.

Rockey SJ, Collins FS. Managing Financial Conflict of Interest in Biomedical Research. JAMA. 2010 Jun 16;303(23):2400-2.

Tamblyn, R., Girard, N., Qian, C. J., & Hanley, J. Assessment of potential bias in research grant peer review in Canada. CMAJ 2018;190(16), E489-E499.

Wager E. Ethics: What is it for? Nature 2006.

Wood SF, Mador JK. Science and Regulation. Uncapping Conflict of Interest? Science. 2013 Jun 7;340(6137):1172-3.

Critique and Text Analysis Up

Bornmann L, Wolf M, Daniel HD. Closed Versus Open Reviewing of Journal Manuscripts: How Far do Comments Differ in Language Use? Scientometrics. 2012;91(3):843-56.

Fuller EO, Hasselmeyer EG, Hunter JC, Abdellah FG, Hinshaw AS. Summary Statements of the NIH Nursing Research Grant Applications. Nurs Res. 1991 Nov-Dec;40(6):346-51.

Magua W, Zhu X, Bhattacharya A, Filut A, Potvien A, Leatherberry R, Lee YG, Jens M, Malikireddy D, Carnes M, Kaatz A. “Are female applicants disadvantaged in National Institutes of Health peer review? Combining algorithmic text mining and qualitative methods to detect evaluative differences in R01 reviewers’ critiques.” Journal of Women’s Health. 2017 May 1;26(5):560-70.

Pier EL, Raclaw J, Kaatz A, Brauer M, Carnes M, Nathan MJ, & Ford CE (2017). . “‘Your comments are meaner than your score’: score calibration talk influences intra-and inter-panel variability during scientific grant peer review.” Research Evaluation, 26(1), 1-14.

Pier, E.L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J., Nathan, M.J., Ford, C.E. and Carnes, M., 2018. “Low agreement among reviewers evaluating the same NIH grant applications.”Proceedings of the National Academy of Sciences, 115(12), pp.2952-2957.

Tausczik YR, Pennebaker JW. The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods. J Lang Soc Psychol. 2010;29(1):24-54.

Decision Making Emotion and Risk Up

De Martino B, Kumaran D, Seymour B, Dolan RJ. Frames, Biases, and Rational Decision-Making in the Human Brain. Science. 2006 Aug 4;313(5787):684-7.

Denes-Raj V, Epstein S. Conflict Between Intuitive and Rational Processing: When People Behave Against Their Better Judgment. J Pers Soc Psychol. 1994 May;66(5):819-29.

Finucane ML, Alhakami A, Slovic P, Johnson SM. The Affect Heuristic in Judgments of Risks and Benefits. J Behav Decis Mak. 2000;13(1):1-17.

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “Risk evaluation in peer review of grant applications.” Environment Systems and Decisions. 24 February, 2018.

Gregory R, Mendelsohn R. Perceived Risk, Dread, and Benefits. Risk Anal. 1993;13(3):259-64.

Lipshitz R, Strauss O. Coping with Uncertainty: A Naturalistic Decision-Making Analysis. Organ Behav Hum Decis Process. 1997;69(2):149-63.

Pachur T, Hertwig R, Steinmann F. How do People Judge Risks: Availability Heuristic, Affect Heuristic, or Both? J Exp Psychol Appl. 2012 Sep;18(3):314-30.

Rand DG, Epstein ZG. Risking Your Life Without a Second Thought: Intuitive Decision-Making and Extreme Altruism. PLoS One. 2014 Oct 15;9(10):e109687.

Sjöberg L. Factors in Risk Perception. Risk Anal. 2000;20(1):1-12.

Slovic P, Finucane ML, Peters E, MacGregor DG. Risk as Analysis and Risk as Feelings: Some Thoughts About Affect, Reason, Risk, and Rationality. Risk Anal. 2004 Apr;24(2):311-22.

Slovic P, Peters E. Risk Perception and Affect. Curr Dir Psychol Sci. 2006;15(6):322-5.

Tversky, Amos, and Daniel Kahneman. “Judgment under uncertainty: Heuristics and biases.” science 185, no. 4157 (1974): 1124-1131.

Funding Strategy Up

Azoulay P, Graff Zivin JS, Manso G. Incentives and Creativity: Evidence from the Academic Life Sciences. NBER Working Paper No. 15466. 2009.

Bollen, J., Crandall, D., Junk, D., Ding, Y., & Börner, K. (2014). “From funding agencies to scientific agency.” EMBO reports, 15(2), 131-133.

Boyington, J. E., Antman, M. D., Patel, K. C., & Lauer, M. S. (2016). “Towards Independence: Resubmission Rate of Unfunded National Heart, Lung, and Blood Institute R01 Research Grant Applications among Early Stage Investigators.” Academic medicine: journal of the Association of American Medical Colleges, 91(4), 556.

Fang, F.C. and Casadevall, A., 2016. “Research Funding: the Case for a Modified Lottery.” mBio, 7(2).

Gallo SA, Sullivan JH, Glisson SR “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.” PLOS ONE. 21 October, 2016.

Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., & Kington, R. Race, ethnicity, and NIH research awards. Science. 2011;333(6045), 1015-1019.

Lauer MS. Personal Reflections on Big Science, Small Science, or The Right Mix. Circ Res. 2014 Mar 28;114(7):1080-2.

Gender Up

Bornmann L, Mutz R, Daniel HD. Gender Differences in Grant Peer Review: A Meta-Analysis. J Informetr. 2007 Jul;1(3):226-38.

Gallo SA, Schmaling KB, Thompson LA, and Glisson SR. “Grant Review Feedback: Appropriateness and Usefulness.Science and Engineering Ethics. 27 (18) March, 2021.

Homma MK, Motohashi R, Ohtsubo H. Japan’s Lagging Gender Equality. Science. 2013 Apr 26;340(6131):428-30.

Kolehmainen C, & Carnes M (2018). “Who resembles a scientific leader—Jack or Jill? How implicit bias could influence research grant funding.” Circulation, 137(8), 769-770.

Magua W, Zhu X, Bhattacharya A, Filut A, Potvien A, Leatherberry R, Lee YG, Jens M, Malikireddy D, Carnes M, Kaatz A. “Are female applicants disadvantaged in National Institutes of Health peer review? Combining algorithmic text mining and qualitative methods to detect evaluative differences in R01 reviewers’ critiques.” Journal of Women’s Health. 2017 May 1;26(5):560-70.

Tamblyn, R., Girard, N., Qian, C. J., & Hanley, J. Assessment of potential bias in research grant peer review in Canada. CMAJ 2018;190(16), E489-E499.

van Arensbergen P, van der Weijden I, van den Besselaar P. Academic Talent Selection in Grant Review Panels. In book: (Re)Searching Academic Careers, Chapter: 2. 2014. Publisher: Russian Academy of Sciences.

General Peer Review Up

Abdoul H, Perrey C, Amiel P, Tubach F, Gottot S, Durand-Zaleski I, Alberti C. Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices. PLoS One. 2012;7(9):e46054.

Alberts B, Kirschner MW, Tilghman S, Varmus H. Rescuing US Biomedical Research from its Systemic Flaws. Proc Natl Acad Sci U S A. 2014 Apr 22;111(16):5773-7.

Bandeh-Ahmadi A. Editorial Decisions and Measuring Information Contents of Reviews. Available at SSRN. 2013.

Benos DJ, Bashari E, Chaves JM, Gaggar A, Kapoor N, LaFrance M, Mans R, Mayhew D, McGowan S, Polter A, Qadri Y, Sarfare S, Schultz K, Splittgerber R, Stephenson J, Tower C, Walton RG, Zotov A. The Ups and Downs of Peer Review. Adv Physiol Educ. 2007 Jun;31(2):145-52.

Bhattacharya A. Science Funding: Duel to the Death. Nature. 2012 Aug 2;488(7409):20-2.

Bollen, J., Crandall, D., Junk, D., Ding, Y., & Börner, K. (2014). “From funding agencies to scientific agency.” EMBO reports, 15(2), 131-133.

Bornmann L. Scientific Peer Review: An Analysis of the Peer Review Process from the Perspective of Sociology of Science Theories. Human Architecture: Journal of the Sociology of Self-Knowledge. 2008;VI(2):23-38.

Cantor M, Gero S. The Missing Metric: Quantifying Contributions of Reviewers. R Soc Open Sci. 2015 Feb 11;2(2):140540.

Chandrasekharan, S., Zaka, M., Gallo, S., Zhao, W., Korobskiy, D., Warnow, T., & Chacko, G. (2020). Finding scientific communities in citation graphs: Articles and authors. Quantitative Science Studies. Advance publication. https://doi.org/10.1162 /qss_a_00095

Chang WR, McLean IP. CUSUM: A Tool for Early Feedback About Performance? BMC Med Res Methodol. 2006 Mar 2;6:8.

Cole S, Rubin L, Cole J. Peer Review in the National Science Foundation: Phase One of a Study. Washington, DC: National Academy of Sciences, 1978.

Council of Canadian Academies. Informing Research Choices: Indicators and Judgment. 2012.

Demicheli V, Di Pietrantonj C. Peer Review for Improving the Quality of Grant Applications. Cochrane Database Syst Rev. 2007 Apr 18;(2):MR000003.

DiSilvestro RL. The First Half Century: A History of AIBS. BioScience. 1997 Nov;47(10):643-9.

Dumanis SB, Ullrich L, Washington PM, Forcelli PA. It’s Money! Real-World Grant Experience Through a Student-Run, Peer-Reviewed Program. CBE Life Sci Educ. 2013 Fall;12(3):419-28.

Fang, F.C. and Casadevall, A., 2016. “Research Funding: the Case for a Modified Lottery.” mBio, 7(2).

Fleurence RL, Forsythe LP, Lauer M, Rotter J, Ioannidis JP, Beal A, Frank L, Selby JV. Engaging Patients and Stakeholders in Research Proposal Review: The Patient-Centered Outcomes Research Institute. Ann Intern Med. 2014 Jul 15;161(2):122-30.

Fliesler SJ. Rethinking Grant Peer Review. Science. 1997 Mar 7;275(5305):1399.

Gallo SA et al. “The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies.” 2014 PLoS ONE 9(9): e106474. doi:10.1371/journal.pone.0106474

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey..” Sci Eng Ethics. July, 2019. (Pre-print available)

Gillespie GW, Jr., Chubin DE, Kurzon GM. Experience with NIH Peer Review: Researchers’ Cynicism and Desire for Change. Sci Technol Human Values. 1985;10(3):45-54.

Glass B, Krauss RW. AIBS Presidents Revisit the Past. BioScience. 1997 Nov;47(10):650-6.

Glenn JF. Uses and Needs for Peer Review in Army Medical Research. Technol Innov. 2010;12:241-7.

Greaves S, Scott J, Clarke M, Miller M, Hannay T, Thomas A, Campbell P. Overview: Nature’s Peer Review Trial. Nature. 2006.

Guthrie S, Guerin B, Wu H, Ismail S, Wooding S. Alternatives to Peer Review in Research Project Funding. Santa Monica, CA: RAND Corporation, 2013.

Hand E. Extra Scrutiny for ‘Grandee Grantees’. Nature. 2012 Feb 20;482(7386):450-1.

Hartmann I, Neidhardt F. Peer Review at the Deutsche Deutsche Forschungsgemeinschaft. Scientometrics. 1990 Nov 01;19(5-6):419-25.

Hartonen T, Alava MJ. How Important Tasks Are Performed: Peer Review. Sci Rep. 2013;3:1679.

Hayes M Grant Review in focus.” Publons 2019

Herbert DL, Barnett AG, Clarke P, Graves N. On the Time Spent Preparing Grant Proposals: An Observational Study of Australian Researchers. BMJ Open. 2013 May 28;3(5). pii: e002800.

Herbert DL, Barnett AG, Graves N. Funding: Australia’s Grant System Wastes time. Nature. 2013 Mar 21;495(7441):314.

Herbert DL, Coveney J, Clarke P, Graves N, Barnett AG. The Impact of Funding Deadlines on Personal Workloads, Stress and Family Relationships: A Qualitative Study of Australian Researchers. BMJ Open. 2014 Mar 28;4(3):e004462.

Herrmann-Lingen C, Brunner E, Hildenbrand S, Loew TH, Raupach T, Spies C, Treede RD, Vahl CF, Wenz HJ. Evaluation of Medical Research Performance–Position Paper of the Association of the Scientific Medical Societies in Germany (AWMF). Ger Med Sci. 2014 Jun 26;12:Doc11.

Hickam D, Totten A, Berg A, Rader K, Goodman S, Newhouse R, Editors. The PCORI Methodology Report. 2013.

Ho RC, Mak KK, Tao R, Lu Y, Day JR, Pan F. Views on the Peer Review System of Biomedical Journals: An Online Survey of Academics from High-Ranking Universities. BMC Med Res Methodol. 2013 Jun 7;13:74.

IOM. Strategies for managing the Breast Cancer Research Program: A Report to the U.S. Army Medical Research and Development Command. Washington, DC: National Academy Press, 1993.

Irwin D, Gallo SA, Glisson SR. “Opinion: Learning from Peer Review.” The Scientist. 2013 May 24.

Ismail S, Farrands A, Wooding S. Evaluating Grant Peer Review in the Health Sciences: A review of the Literature. Santa Monica, CA: RAND Corporation, 2009.

Jefferson T, Wager E, Davidoff F. Measuring the Quality of Editorial Peer Review. JAMA. 2002 Jun 5;287(21):2786-90.

Johnson VE. Statistical Analysis of the National Institutes of Health Peer Review System. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

King JC, Lawrence TS, Murphy SB, Davidson NE, Mayer RJ. The American Society of Clinical Oncology Cancer Foundation Grants Program: A 25-Year Report and a Look Toward the Future. J Clin Oncol. 2010 Mar 20;28(9):1616-21.

Koonin E, Landweber L, Lipman D, Dignon R. Systems: Reviving a Culture of Scientific Debate. Nature. 2006.

Kupfer DJ, Murphree AN, Pilkonis PA, Cameron JL, Giang RT, Dodds NE, Godard KA, Lewis DA. Using Peer Review to Improve Research and Promote Collaboration. Acad Psychiatry. 2014 Feb;38(1):5-10.

Lahiri D. Perspective: The Case for Group Review. Nature 2006.

Lee K, Bero L. Ethics: Increasing Accountability. Nature 2006.

Lee K, Brownstein JS, Mills RG, Kohane IS. Does Collocation Inform the Impact of Collaboration? PLoS One. 2010 Dec 15;5(12):e14279.

Liaw L, Freedman JE, Becker LB, Mehta NN, & Liscum L. (2017). “Peer review practices for evaluating biomedical research grants: A scientific statement from the American Heart Association.” Circulation research, 121(4), e9-e19.

Lyman RL. A Three-Decade History of the Duration of Peer Review. Journal of Scholarly Publishing. 2013 Apr;44(3):211-20.

McGarity TO. Peer Review in Awarding Federal Grants in the Arts and Sciences. Berkeley Tech LJ. 1994;9(1).

McGeary M, Hanna KE. Strategies to Leverage Research Funding: Guiding DOD’s Peer Reviewed Medical Research Programs. Washington, DC: National Academies Press, 2004.

Mervis J. Peer Review. Beyond the Data. Science. 2011 Oct 14;334(6053):169-71.

Mietchen D. The Transformative Nature of Transparency in Research Funding. PLoS Biol. 2014 Dec 30;12(12):e1002027.

Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Int J Surg. 2010;8(5):336-41.

Nicholson J, Ioannidis J. Research Grants: Conform and be Funded. Nature. 2012 Dec 6;492:34-36.

Parliamentary Office of Science and Technology. Peer Review. 2002;182.

Pawelczyk JA, Strawbridge LM, Schultz AM, Liverman CT. A Review of NASA Human Research Program’s Scientific Merit Processes: Letter Report. Washington, DC: National Academies Press, 2012.

Pederson T. The “Study” Role of Past National Institutes of Health Study Sections. Mol Biol Cell. 2012 Sep;23(17):3281-4.

Ranalli B. A Prehistory of Peer Review: Religious Blueprints from the Hartlib Circle. Spontaneous Generations. 2011;5(1):12-18.

Riley B. Systems: Trusting Data’s Quality. Nature. 2006.

Roebber PJ, Schultz DM. Peer Review, Program Officers and Science Funding. PLoS One. 2011 Apr 12;6(4):e18680.

Russell AS, Thorn BD, Grace M. Peer Review: A Simplified Approach. J Rheumatol. 1983 Jun;10(3):479-81.

Sandewall E. Systems: Opening up the Process. Nature. 2006.

Sandstrom U, Hallsten M. Persistent Nepotism in Peer-Review. Scientometrics. 2008 Feb;74(2):175-89.

Smith MA, Kaufman NJ, Dearlove AJ. External Community Review Committee: A New Strategy for Engaging Community Stakeholders in Research Funding Decisions. Prog Community Health Partnersh. 2013 Fall;7(3):301-12.

Smith R. Classical Peer Review: An Empty Gun. Breast Cancer Res. 2010 Dec 20;12 Suppl 4:S13.

Smith R. Peer Review: A Flawed Process at the Heart of Science and Journals. J R Soc Med. 2006 Apr;99(4):178-82.

Spier R. The History of the Peer-Review Process. Trends Biotechnol. 2002 Aug;20(8):357-8.

Squazzoni F, Gandelli, C. Peer Review Under the Microscope. An Agent-Based Model of Scientific Collaboration. Simulation Conference (WSC), Proceedings of the 2012 Winter. Dec 2012:1-12.

Squazzoni F, Takacs, K. Social Simulation that “Peers into Peer Review.” J Artif Soc Soc Simulat. 2011 Oct 31;14(4):3.

Stahel PF, Moore EE. Peer Review for Biomedical Publications: We Can Improve the System. BMC Med. 2014 Sep 26;12(1):179.

Turner RS. Best Practices in Peer Review Assure Quality, Value, Objectivity. Journal of the National Grants Management Association. 2009;17(1):43-8.

United States General Accounting Office. Federal Research: Peer Review Practices at Federal Science Agencies Vary. Report to Congressional Requesters. 2009 March.

Walker R, Rocha da Silva P. Emerging Trends in Peer Review-a Survey. Front Neurosci. 2015 May 27;9:169.

Wessely S. Peer Review of Grant Applications: What do we Know? Lancet. 1998 Jul 25;352(9124):301-5.

Zare RN, Winnacker EL. China’s Science Funding. Science. 2011 Oct 28;334(6055):433.

van Arensbergen P, van der Weijden I, van den Besselaar P. Academic Talent Selection in Grant Review Panels. In book: (Re)Searching Academic Careers, Chapter: 2. 2014. Publisher: Russian Academy of Sciences.

General Science Up

Readers’ Poll Results: Funding Environment. Science. 2013 Apr 26;340(6131):429.

American Society for Biochemistry and Molecular Biology. Towards a Sustainable Biomedical Research Enterprise. 2014.

Boyack KW, Klavans R, Borner K. Mapping the Backbone of Science. Scientometrics. 2005;64(3):351-74.

Cohen LR, Noll RG. The Future of the National Laboratories. PNAS. 1996;93(23):12678-85.

Couzin-Frankel J. Chasing the Money. Science. 2014 Apr 4;344(6179):24-5.

Davis, G. Doctors Without Orders. American Scientist. 2005;93(3, supplement).

Furmana JL, Porter ME, Stern S. The Determinants of National Innovative Capacity. Res Policy. 2002;31(6):899-933.

Garrison HH, Deschamps AM. Physician Scientists: Assessing the Workforce. FASEB. 2013.

Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., & Kington, R. Race, ethnicity, and NIH research awards. Science. 2011;333(6045), 1015-1019.

Ioannidis JP. How to Make More Published Research True. PLoS Med. 2014 Oct 21;11(10):e1001747.

Lopes GR, da Silva R, Moro MM, Palazzo Moreira de Oliveira J. Scientific Collaboration in Research Networks: A Quantification Method by Using Gini Coefficient. IJCSA. 2012;9(2):15-31.

National Institute of Allergy and Infectious Diseases. FY 2014 Budget.

National Institute of Allergy and Infectious Diseases. The Edge of Discovery: A Portrait of the National Institute of Allergy and Infectious Diseases. June 2009;NIH Publication No. 09-6389.

Rice JA, Wu CO. Nonparametric Mixed Effects Models for Unequally Sampled Noisy Curves. Biometrics. 2001 Mar;57(1):253-9.

Sullivan Faith K. Patterns of Creation and Discovery: An Analysis of Defense Laboratory Patenting and Innovation. 2013;dissertation:Pardee Rand Graduate School.

Taleb NN, Douady R. Mathematical Definition, Mapping, and Detection of (Anti)Fragility. Quant Finance. 2013;13(11):1677-89.

Van den Besselaar P, Leydesdorf L. Past Performance as Predictor of Successful Grant Applications: A Case Study. Den Haag, Rathenau Instituut SciSA rapport 0704.

H Index Citations and Bibliometrics Up

Auranen O, Nieminen M. University Research Funding and Publication Performance–An International Comparison. Res Policy. 2010;39(6):822-34.

Belter CW. Can Bibliometric Indicators Predict Institutional Citation Patterns? Presented at ACRL 2013, 10-13 April, Indianapolis IN.

Bornmann L, Leydesdorff L. On the Meaningful and Non-Meaningful Use of Reference Sets in Bibliometrics. J Informetr. 2014;8(1):273-5.

Boyack KW, Jordan P. Metrics Associated with NIH Funding: A High-Level View. J Am Med Inform Assoc. 2011 Jul-Aug;18(4):423-31.

Cantor M, Gero S. The Missing Metric: Quantifying Contributions of Reviewers. R Soc Open Sci. 2015 Feb 11;2(2):140540.

Doyle JM, Quinn K, Bodenstein YA, Wu CO, Danthi N, Lauer MS. Association of Percentile Ranking with Citation Impact and Productivity in a Large Cohort of de novo NIMH-funded R01 Grants. Mol Psychiatry. 2015 Jun 2.

Farhadi H, Salehi H, Md Yunus M, Chadegani AA, Farhadi M, Fooladi M, Ebrahim NA. Does it Matter Which Citation Tool is Used to Compare the H-Index of a Group of Highly Cited Researchers? Aust J Basic Appl Sci. 2013;7(4):198-202.

Garfield E. Current Comments: Lifetime Citations Rates. Essays of an Information Scientist. 1980 Jan;4(2):355-8.

Gauffriau M, Larsen PO, Maye I, Roulin-Perriard A, von Ins M. Publication, Cooperation and Productivity Measures in Scientific Research. Scientometrics. 2007;73(2).175-214.

Ginther, D. K., Basner, J., Jensen, U., Schnell, J., Kington, R., & Schaffer, W. T. (2018). “Publications as predictors of racial and ethnic differences in NIH research awards.” PloS one, 13(11), e0205929.

Hirsch JE. An Index to Quantify an Individual’s Scientific Research Output. Proc Natl Acad Sci U S A. 2005 Nov 15;102(46):16569-72.

Hirsch JE. Does the H Index Have Predictive Power? Proc Natl Acad Sci U S A. 2007 Dec 4;104(49):19193-8.

Kaltman JR, Evans FJ, Danthi NS, Wu CO, DiMichele DM, Lauer MS. Prior Publication Productivity, Grant Percentile Ranking, and Topic-Normalized Citation Impact of NHLBI Cardiovascular R01 Grants. Circ Res. 2014 Sep 12;115(7):617-24.

Khan NR, Thompson CJ, Taylor DR, Gabrick KS, Choudhri AF, Boop FR, Klimo P Jr. Part II: Should the H-Index be Modified? An Analysis of the M-Quotient, Contemporary H-Index, Authorship Value, and Impact Factor. World Neurosurg. 2013 Dec;80(6):766-74.

Kinney AL. National Scientific Facilities and Their Science Impact on Nonbiomedical Research. Proc Natl Acad Sci U S A. 2007 Nov 13;104(46):17943-7.

Larivière V, Macaluso B, Archambault E, Gingras Y. Which Scientific Elite? On the Concentration of Funding, Productivity and Citations. In Proceedings of the 12th International Conference of the International Society for Scientometrics and Informetrics. 2009.

Lazaridis T. Ranking University Departments Using the Mean H-Index. Scientometrics. 2010 Feb;82(2):211-16.

Marx W, Schier H, Andersen OK. Using Time Dependent Citation Rates (Sales Curves) for Comparing Scientific Impacts. 2006. arXiv:physics/0611284 [physics.soc-ph].

Molinari JF, Molinari A. A New Methodology for Ranking Scientific Institutions. Scientometrics. 2008 Apr;75(1):163-74.

Nieminen P, Carpenter J, Rucker G, Schumacher M. The Relationship Between Quality of Research and Citation Frequency. BMC Med Res Methodol. 2006 Sep 1;6:42.

Sidiropoulos A, Katsaros D, Manolopoulos Y. Generalized Hirsch H-Index for Disclosing Latent Facts in Citation Networks. Scientometrics. 2007 Aug 1;72(2):253-280.

Sypsa V, Hatzakis A. Assessing the Impact of Biomedical Research in Academic Institutions of Disparate Sizes. BMC Med Res Methodol. 2009 May 29;9:33.

Waltman L, van Eck NJ, van Leeuwen TN, Visser MS, van Raan AF. On the Correlation Between Bibliometric Indicators and Peer Review: Reply to Opthof and Leydesdorff. Scientometrics. 2011 Sep;88(3):1017-1022.

Innovation Up

Bromham L, Dinnage R, and Hua X. “Interdisciplinary research has consistently lower funding success.” Nature 534, no. 7609 (2016): 684.

Chandrasekharan, S., Zaka, M., Gallo, S., Zhao, W., Korobskiy, D., Warnow, T., & Chacko, G. (2020). Finding scientific communities in citation graphs: Articles and authors. Quantitative Science Studies. Advance publication. https://doi.org/10.1162 /qss_a_00095

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “Risk evaluation in peer review of grant applications.” Environment Systems and Decisions. 24 February, 2018.

Horrobin DF. The Philosophical Basis of Peer Review and the Suppression of Innovation. JAMA. 1990 Mar 9;263(10):1438-41.

Kaime EM, Moore KH, Goldberg SF. CDMRP: Fostering Innovation Through Peer Review. Technol Innov. 2010;12:233-40.

Luukkonen T. Conservatism and Risk-Taking in Peer Review: Emerging ERC Practices. Research Evaluation. 2012;21:48-60.

Nicholson J, Ioannidis J. Research Grants: Conform and be Funded. Nature. 2012 Dec 6;492:34-36.

Inter Rater Reliability Up

Carpenter AS, Sullivan JH, Deshmukh A, Glisson SR, & Gallo SA. “A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels.” BMJ Open. 8 September, 2015.

Colliver J. Commentary on Cichetti’s “Reliability of Peer Review.” Teach Learn Med. 2002;14(3):142-3.

Fleiss JL, Shrout PE. The Effects of Measurement Errors on Some Multivariate Procedures. Am J Public Health. 1977 December; 67(12): 1188-91.

Fogelholm M, Leppinen S, Auvinen A, Raitanen J, Nuutinen A, Väänänen K. Panel Discussion Does Not Improve Reliability of Peer Review for Medical Research Grant Proposals. J Clin Epidemiol. 2012 Jan;65(1):47-52.

Gallo SA, Schmaling KB, Thompson LA, and Glisson SR. “Grant Review Feedback: Appropriateness and Usefulness.Science and Engineering Ethics. 27 (18) March, 2021.

Gallo SA, Sullivan JH, Glisson SR “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.” PLOS ONE. 21 October, 2016.

Giraudeau B, Leyrat C, Le Gouge A, Léger J, Caille A. Peer Review of Grant Applications: A Simple Method to Identify Proposals with Discordant Reviews. PLoS One. 2011;6(11):e27557.

Hallgren KA. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol. 2012;8(1):23-34.

Hargens LL, Herting JR. Neglected Considerations in the Analysis of Agreement Among Journal Referees. Scientometrics. 1990 Jul 01;19(1-2):91-106.

Johnson VE. Statistical Analysis of the National Institutes of Health Peer Review System. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

Kravitz RL, Franks P, Feldman MD, Gerrity M, Byrne C, Tierney WM. Editorial Peer Reviewers’ Recommendations at a General Medical Journal: Are They Reliable and do Editors Care? PLoS One. 2010 Apr 8;5(4):e10072.

Pier EL, Raclaw J, Kaatz A, Brauer M, Carnes M, Nathan MJ, & Ford CE (2017). . “‘Your comments are meaner than your score’: score calibration talk influences intra-and inter-panel variability during scientific grant peer review.” Research Evaluation, 26(1), 1-14.

Pier, E.L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J., Nathan, M.J., Ford, C.E. and Carnes, M., 2018. “Low agreement among reviewers evaluating the same NIH grant applications.”Proceedings of the National Academy of Sciences, 115(12), pp.2952-2957.

Sattler DN, McKnight PE, Naney L, Mathis R. Grant Peer Review: Improving Inter-Rater Reliability with Training. PLoS One. 2015 Jun 15;10(6):e0130450.

Nih and Nsf Up

Berg J. Scoring Analysis with Funding and Investigator Status. NIGMS Feedback Loop Blog. 2010 Sep 14.

Berg J. NIH-Wide Correlations Between Overall Impact Scores and Criterion Scores. NIGMS Feedback Loop Blog. 2010 Sep 30.

Collins FS, Tabak LA. Policy: NIH Plans to Enhance Reproducibility. Nature. 2014 Jan 30;505(7485):612-3.

Gallo SA, Schmaling KB, Thompson LA, and Glisson SR. “Grant Review Feedback: Appropriateness and Usefulness.Science and Engineering Ethics. 27 (18) March, 2021.

Gillespie GW, Jr., Chubin DE, Kurzon GM. Experience with NIH Peer Review: Researchers’ Cynicism and Desire for Change. Sci Technol Human Values. 1985;10(3):45-54.

Ginther, D. K., Basner, J., Jensen, U., Schnell, J., Kington, R., & Schaffer, W. T. (2018). “Publications as predictors of racial and ethnic differences in NIH research awards.” PloS one, 13(11), e0205929.

Hoppe TA, Litovitz A, Willis KA, Meseroll RA, Perkins MJ, Hutchins BI, Davis AF, Lauer MS, Valantine HA, Anderson JM, Santangelo GM. “Topic choice contributes to the lower rate of NIH awards to African-American/black scientists.” Science Advances. 2019 Oct 1;5(10):eaaw7238.

Kaiser J. Biomedical Research. A Call for NIH Youth Movement. Science. 2014 Oct 10;346(6206):150-1.

Magua W, Zhu X, Bhattacharya A, Filut A, Potvien A, Leatherberry R, Lee YG, Jens M, Malikireddy D, Carnes M, Kaatz A. “Are female applicants disadvantaged in National Institutes of Health peer review? Combining algorithmic text mining and qualitative methods to detect evaluative differences in R01 reviewers’ critiques.” Journal of Women’s Health. 2017 May 1;26(5):560-70.

Moore RF, Catevenis K, Pearson K, Wagner RM. AIRI Statistics: Trends in NIH Extramural Funding. Presented at the Association of Independent Research Institutes (AIRI) 51st Annual Meeting. 2012, 3 Oct, Philadelphia, PA.

NIH. Enhancing Peer Review: Implementation of Recommended Actions. Transcript accompanying video of Dr. Alan Willard. 2009 Feb.

NIH. Scoring and Review Changes. 2009 Oct 22.

NIH. Research Project Grant (RPG) Critique Template. R01/R03/R15/R21/R34 Review. 2012 Mar 5.

NIH. Chair Orientation. 2012 Mar 5.

NIH. NIH Reviewer Orientation. 2012 Dec 14.

NIH. NIH Success Rate Definition. 2009 Feb.

NIH. NIH Success Rate Definition. 2012 Feb.

NIH. Scoring System and Procedure. 2009 Mar 20.

Open Peer Review Up

Bornmann L, Wolf M, Daniel HD. Closed Versus Open Reviewing of Journal Manuscripts: How Far do Comments Differ in Language Use? Scientometrics. 2012;91(3):843-56.

Hettyey A, Griggio M, Mann M, Raveh S, Schaedelin FC, Thonhauser KE, Thoss M, van Dongen WF, White J, Zala SM, Penn DJ. Peerage of Science: Will it Work? Trends Ecol Evol. 2012 Apr;27(4):189-90.

Koop T, Pöschl U. Systems: An Open, Two-Stage Peer-Review Journal. Nature. 2006.

Mietchen D. The Transformative Nature of Transparency in Research Funding. PLoS Biol. 2014 Dec 30;12(12):e1002027.

Moylan EC, Harold S, O’Neill C, Kowalczuk MK. Open, Single-Blind, Double-Blind: Which Peer Review Process do You Prefer? BMC Pharmacol Toxicol. 2014 Sep 30;15:55.

Walker R, Rocha da Silva P. Emerging Trends in Peer Review-a Survey. Front Neurosci. 2015 May 27;9:169.

Panel Discussion Up

Carpenter AS, Sullivan JH, Deshmukh A, Glisson SR, & Gallo SA. “A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels.” BMJ Open. 8 September, 2015.

Cole S, Cole JR, Simon GA. Chance and Consensus in Peer Review. Science. 1981 Nov 20;214(4523):881-6.

Fogelholm M, Leppinen S, Auvinen A, Raitanen J, Nuutinen A, Väänänen K. Panel Discussion Does Not Improve Reliability of Peer Review for Medical Research Grant Proposals. J Clin Epidemiol. 2012 Jan;65(1):47-52.

Gallo, S.A., Schmaling, K.B., Thompson, L.A., and Glisson, S.R. Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion. Res Integr Peer Rev. 2020; 5:(7).

Johnson VE. Statistical Analysis of the National Institutes of Health Peer Review System. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

Obrecht M, Tibelius K, D’Aloisio G. Examining the Value Added by Committee Discussion in the Review of Applications for Research Awards. Research Evaluation. 2007 Jun;16(2):79-91.

Pier EL, Raclaw J, Kaatz A, Brauer M, Carnes M, Nathan MJ, & Ford CE (2017). . “‘Your comments are meaner than your score’: score calibration talk influences intra-and inter-panel variability during scientific grant peer review.” Research Evaluation, 26(1), 1-14.

Peer Review Technology Up

Anderson C. Technical Solutions: Wisdom of the Crowds. Nature 2006.

Carpenter AS, Sullivan JH, Deshmukh A, Glisson SR, & Gallo SA. “A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels.” BMJ Open. 8 September, 2015.

Doran MR, Lott WB, Doran SE. Communication: Use Multimedia in Grant Applications. Nature. 2014 Jan 16;505(7483):291.

Driskell JE, Radtke PH, Salas E. Virtual Teams: Effects of Technological Mediation on Team Performance. Group Dyn. 2003 Dec;7(4):297-323.

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey..” Sci Eng Ethics. July, 2019. (Pre-print available)

Jenkins SL, Iyengar R, Diverse-Pierluissi MA, Chan AM, Devi LA, Sobie EA, Ting AT, Weinstein DC. Teaching Resources. Using Web-Based Discussion Forums as a Model of the Peer-Review Process and a Tool for Assessment. Sci Signal. 2008;1(9):tr2.

Vo, NM, Quiggle, GM, Wadhwani, K. Comparative outcomes of face-to-face and virtual review meetings. International Journal of Surgery Open. 2016. Volume 4 , 38 - 41

Vo, NM, Trocki, R. Virtual and Peer Reviews of Grant Applications at the Agency for Healthcare Research and Quality. Southern Medical Journal. 2015. 108. 622-626.

Resubmission Up

Bornmann L, Mutz R, Daniel H-R. Latent Markov Modeling Applied to Grant Peer Review. J Informetr. 2008;2(4):217-28.

Boyington, J. E., Antman, M. D., Patel, K. C., & Lauer, M. S. (2016). “Towards Independence: Resubmission Rate of Unfunded National Heart, Lung, and Blood Institute R01 Research Grant Applications among Early Stage Investigators.” Academic medicine: journal of the Association of American Medical Colleges, 91(4), 556.

Kolehmainen C, & Carnes M (2018). “Who resembles a scientific leader—Jack or Jill? How implicit bias could influence research grant funding.” Circulation, 137(8), 769-770.

Reviewer Pi Demographics Up

Gallo SA, Schmaling KB, Thompson LA, and Glisson SR. “Grant Review Feedback: Appropriateness and Usefulness.Science and Engineering Ethics. 27 (18) March, 2021.

Gallo SA, Sullivan JH, Glisson SR “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.” PLOS ONE. 21 October, 2016.

Gallo SA, Thompson LA, Schmaling KB, Glisson SR. “The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey..” Sci Eng Ethics. July, 2019. (Pre-print available)

Ginther, D. K., Basner, J., Jensen, U., Schnell, J., Kington, R., & Schaffer, W. T. (2018). “Publications as predictors of racial and ethnic differences in NIH research awards.” PloS one, 13(11), e0205929.

Hayes M Grant Review in focus.” Publons 2019

Jones BF, Weinberg BA. Age Dynamics in Scientific Creativity. Proc Natl Acad Sci U S A. 2011 Nov 22;108(47):18910-4.

Kaplan D, Lacetera N, Kaplan C. Sample Size and Precision in NIH Peer Review. PLoS One. 2008 Jul 23;3(7):e2761.

Scoring Scale Up

Cicchetti DV, Shoinralter D, Tyrer PJ. The Effect of Number of Rating Scale Categories on Levels of Interrater Reliability: A Monte Carlo Investigation. Appl Psychol Meas. 1985 Mar;9(1):31-6.

Green JG, Calhoun F, Nierzwicki L, Brackett J, Meier P. Rating Intervals: An Experiment in Peer Review. FASEB J. 1989 Jun;3(8):1987-92.

Team Performance Up

Barabási AL. Sociology. Network Theory–the Emergence of the Creative Enterprise. Science. 2005 Apr 29;308(5722):639-41.

Borgatti SP, Mehra A, Brass DJ, Labianca G. Network Analysis in the Social Sciences. Science. 2009 Feb 13;323(5916):892-95.

Carpenter AS, Sullivan JH, Deshmukh A, Glisson SR, & Gallo SA. “A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels.” BMJ Open. 8 September, 2015.

Cooke, Nancy J., and Margaret L. Hilton, eds. “Enhancing the effectiveness of team science.” Washington, DC: National Academies Press, 2015.

Cur?eu PL, Jansen RJ, Chappin MM. Decision Rules and Group Rationality: Cognitive Gain or Standstill? PLoS One. 2013;8(2):e56454.

Driskell JE, Radtke PH, Salas E. Virtual Teams: Effects of Technological Mediation on Team Performance. Group Dyn. 2003 Dec;7(4):297-323.

Duch J, Waitzman JS, Amaral LA. Quantifying the Performance of Individual Players in a Team Activity. PLoS One. 2010 Jun 16;5(6):e10937.

Evans JA, Foster JG. Metaknowledge. Science. 2011 Feb 11;331(6018):721-5.

Guimerà R, Uzzi B, Spiro J, Amaral LA. Team Assembly Mechanisms Determine Collaboration Network Structure and Team Performance. Science. 2005 Apr 29;308(5722):697-702.

Jeffcott SA, Mackenzie CF. Measuring Team Performance in Healthcare: Review of Research and Implications for Patient Safety. J Crit Care. 2008 Jun;23(2):188-96.

Langfeldt L. Decision-Making in Expert Panels Evaluating Research: Constraints, Processes and Bias. 2001;dissertation: The University of Oslo.

Manthous CA, Hollingshead AB. Team Science and Critical Care. Am J Respir Crit Care Med. 2011 Jul 1;184(1):17-25.

Roebber PJ, Schultz DM. Peer Review, Program Officers and Science Funding. PLoS One. 2011 Apr 12;6(4):e18680.

Rogelberg SG, O’Connor MS, Sederburg M. Using the Stepladder Technique to Facilitate the Performance of Audioconferencing Groups. J Appl Psychol. 2002 Oct;87(5):994-1000.

Rosen MA, Weaver SJ, Lazzara EH, Salas E, Wu T, Silvestri S, Schiebel N, Almeida S, King HB. Tools for Evaluating Team Performance in Simulation-Based Training. J Emerg Trauma Shock. 2010 Oct;3(4):353-9.

Stokols D, Misra S, Moser RP, Hall KL, Taylor BK. The Ecology of Team Science: Understanding Contextual Influences on Transdisciplinary Collaboration. Am J Prev Med. 2008 Aug;35(2 Suppl):S96-115.

Wickens CD, Holland JG, Parasuraman R, Banbury S. Engineering Psychology and Human Performance (4th ed). Upper Saddle River, NJ: Prentice Hall, 2012.

Training Up

Garfield JM, Kaye AD, Kolinsky DC, Urman RD. A Systematic Guide for Peer Reviewers for a Medical Journal. J Med Pract Manage. 2015 Mar-Apr;30(6 Spec No):13-7.

Rosen MA, Weaver SJ, Lazzara EH, Salas E, Wu T, Silvestri S, Schiebel N, Almeida S, King HB. Tools for Evaluating Team Performance in Simulation-Based Training. J Emerg Trauma Shock. 2010 Oct;3(4):353-9.

Sattler DN, McKnight PE, Naney L, Mathis R. Grant Peer Review: Improving Inter-Rater Reliability with Training. PLoS One. 2015 Jun 15;10(6):e0130450.

Triage Up

Bornmann L, Mutz R, Daniel H-R. Latent Markov Modeling Applied to Grant Peer Review. J Informetr. 2008;2(4):217-28.

Validation and Impact Up

Armstrong PW, Caverson MM, Adams L, Taylor M, Olley PM. Evaluation of the Heart and Stroke Foundation of Canada Research Scholarship Program: Research Productivity and Impact. Can J Cardiol. 1997 May;13(5):507-16.

Beacham L, Li E, Wasserman A. Measuring the Performance of Extramural Funding at the National Library of Medicine. The College of William and Mary. 2008.

Berg J. Productivity Metrics and Peer Review Scores. NIGMS Feedback Loop Blog. 2011 Jun 2.

Bollen, J., Crandall, D., Junk, D., Ding, Y., & Börner, K. (2014). “From funding agencies to scientific agency.” EMBO reports, 15(2), 131-133.

Burroughs Wellcome Fund. Biomedical Sciences: Program Outcomes. 2010 Report.

Cabezas-Clavijo A, Robinson-García N, Escabias M, Jiménez-Contreras E. Reviewers’ Ratings and Bibliometric Indicators: Hand in Hand When Assessing Over Research Proposals? PLoS One. 2013 Jun 28;8(6):e68258.

Claveria LE, Guallar E, Cami J, Conde J, Pastor R, Ricoy JR, Rodriguez-Farre E, Ruiz-Palomo F, Munoz E. Does Peer Review Predict the Performance of Research Projects in Health Sciences? Scientometrics. 2000 Jan 1;47(1):11-23.

Cole S, Cole JR, Simon GA. Chance and Consensus in Peer Review. Science. 1981 Nov 20;214(4523):881-6.

Crespia GA, Geunab A. An Empirical Study of Scientific Production: A Cross Country Analysis, 1981-2002. Res Policy. 2008;37(4):565-79.

Doyle JM, Quinn K, Bodenstein YA, Wu CO, Danthi N, Lauer MS. Association of Percentile Ranking with Citation Impact and Productivity in a Large Cohort of de novo NIMH-funded R01 Grants. Mol Psychiatry. 2015 Jun 2.

Eloy JA, Svider PF, Kanumuri VV, Folbe AJ, Setzen M, Baredes S. Do AAO-HNSF CORE Grants Predict Future NIH Funding Success? Otolaryngol Head Neck Surg. 2014 May 20. [Epub ahead of print]

Fang, F.C. and Casadevall, A., 2016. “Research Funding: the Case for a Modified Lottery.” mBio, 7(2).

Gallo SA et al. “The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies.” 2014 PLoS ONE 9(9): e106474. doi:10.1371/journal.pone.0106474

Goldbeck-Wood S. Evidence on Peer Review–Scientific Quality Control or Smokescreen? BMJ. 1999 January 2;318(7175):44-5.

Holbrook JB, Hrotic S. Blue Skies, Impacts, and Peer Review. RT. 2013 Jul;1(1):2013.

Hoppe TA, Litovitz A, Willis KA, Meseroll RA, Perkins MJ, Hutchins BI, Davis AF, Lauer MS, Valantine HA, Anderson JM, Santangelo GM. “Topic choice contributes to the lower rate of NIH awards to African-American/black scientists.” Science Advances. 2019 Oct 1;5(10):eaaw7238.

Jackson JL, Srinivasan M, Rea J, Fletcher KE, Kravitz RL. The Validity of Peer Review in a General Medicine Journal. PLoS One. 2011;6(7):e22475.

Jacob B, Lefgren L. The Impact of Research Grant Funding on Scientific Productivity. NBER Working Paper No. 13519. 2007.

Kaltman JR, Evans FJ, Danthi NS, Wu CO, DiMichele DM, Lauer MS. Prior Publication Productivity, Grant Percentile Ranking, and Topic-Normalized Citation Impact of NHLBI Cardiovascular R01 Grants. Circ Res. 2014 Sep 12;115(7):617-24.

Kotchen TA, Lindquist T, Miller Sostek A, Hoffmann R, Malik K, Stanfield B. Outcomes of National Institutes of Health Peer Review of Clinical Grant Applications. J Investig Med. 2006 Jan;54(1):13-9.

Laurence WF, Useche C, Laurance SG, Bradshaw CJA. Predicting Publication Success for Biologists. BioScience. 2013;63(10):817-23.

Lindner MD, Nakamura RK. Examining the Predictive Validity of NIH Peer Review Scores. PLoS One. 2015 Jun 3;10(6):e0126938.

Martin MR, Lindquist T, Kotchen TA. Why are Peer Review Outcomes Less Favorable for Clinical Science than for Basic Science Grant Applications? Am J Med. 2008 Jul;121(7):637-41.

Mason JL, Lei M, Faupel-Badger JM, Ginsburg EP, Seger YR, Dijoseph L, Schnell JD, Wiest JS. Outcome Evaluation of the National Cancer Institute Career Development Awards Program. J Cancer Educ. 2013 Mar;28(1):9-17.

Mervis J. Peering Into Peer Review. Science. 2014 Feb 7;343(6171):596-8.

Rinia EJ, Leeuwen Th N, Vuren HG, Raan AFJ. Influence of Interdisciplinarity on Peer-Review and Bibliometric Evaluations in Physics Research. Research Policy. 2001 Mar 1;30(3):357-61.

Stephen A. Gallo, Scott R. Glisson. “External Tests of Peer Review Validity Via Impact Measures.” Front. Res. Metr. Anal. 23 August 2018

Vale RD. Evaluating How We Evaluate. Mol Biol Cell. 2012 Sep;23(17):3285-9.

Waltman L, van Eck NJ, van Leeuwen TN, Visser MS, van Raan AF. On the Correlation Between Bibliometric Indicators and Peer Review: Reply to Opthof and Leydesdorff. Scientometrics. 2011 Sep;88(3):1017-1022.

Yglesias E. Improving Peer Review in the Federal Government. Technol Innov. 2010 Mar;12(3):225-32.

Up