Bookmark and Share


Peer Review Literature

The Peer Review process is used by organizations around the world to identify good, promising science. However, peer review can be implemented in many ways and surprisingly, dialogue among organizations managing peer review is limited. Although literature about peer review (and particularly peer review of grant applications) is scant, there have been some studies addressing review practices, challenges and guidelines.

Compiled below is a reference list of scientific articles concerning the processes, outcomes and validations of the scientific Peer Review process, particularly focusing on funding peer review. This list is sorted by topic area and represents a centralized body of knowledge intended as a reference to those conducting/participating in peer review and those who are interested in the outcomes of the process.

Topic Areas:

Peer Review

Assignment Load

Cook,WD, Golany B, Kress M, Penn M. Optimal allocation of proposals to reviewers to facilitate effective ranking. Manage Sci. 2005;51(4):655-61.

Back to top


Conflict of Interest/Fairness

Abdoul H, Perrey C, Tubach F, Amiel P, Durand-Zaleski I, Alberti C. Non-financial conflicts of interest in academic grant evaluation: a qualitative study of multiple stakeholders in France. PLoS One. 2012;7(4):e35247.

Arms W. Ethics: Trust and reputation on the web. Nature 2006.

Benos D. Ethics: Detecting misconduct. Nature 2006.

Bhattacharjee Y. Science funding. NSF's 'Big Pitch' tests anonymized grant reviews. Science. 2012 May 25;336(6084):969-70.

Bornmann L, Daniel HD. Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees' decisions. Scientometrics. 2005;63(2):297-320.

Hodgson C. Evaluation of cardiovascular grant-in-aid applications by peer review: influence of internal and external reviewers and committees. Can J Cardiol. 1995 Nov;11(10):864-8.

Kotchen TA, Lindquist T, Miller Sostek A, Hoffmann R, Malik K, Stanfield B. Outcomes of National Institutes of Health peer review of clinical grant applications. J Investig Med. 2006 Jan;54(1):13-9.

Langfeldt L. Decision-making in expert panels evaluating research: Constraints, processes and bias. 2001;dissertation: The University of Oslo.

Marsh HW, Jayasinghe UW, Bond NW. Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability. Am Psychol. 2008 Apr;63(3):160-8.

Oleinik A. Conflict(s) of Interest in Peer Review: Its Origins and Possible Solutions. Sci Eng Ethics. 2013 Jan 5. [Epub ahead of print]

Wager E. Ethics: What is it for? Nature 2006.

Back to top


Gender

Bornmann L, Mutz R, Daniel HD. Gender differences in grant peer review: A meta-analysis. J Informetr. 2007 Jul;1(3):226-38.

Kaufman RR, Chevan J. The gender gap in peer-reviewed publications by physical therapy faculty members: a productivity puzzle. Phys Ther. 2011 Jan;91(1):122-31.

Mutz R, Bornmann L, Daniel HD. Does Gender Matter in Grant Peer Review?: An Empirical Investigation Using the Example of the Austrian Science Fund. Z Psychol. 2012;220(2):121-129.

Back to top


General Peer Review

Abdoul H, Perrey C, Amiel P, Tubach F, Gottot S, Durand-Zaleski I, Alberti C. Peer review of grant applications: criteria used and qualitative study of reviewer practices. PLoS One. 2012;7(9):e46054.

AICPA. Summary of the nature, objectives, scope, limitations of, and procedures performed in system and engagement reviews and quality control reviews (as referred to in a peer review report). 2012.

Benos DJ, Bashari E, Chaves JM, Gaggar A, Kapoor N, LaFrance M, Mans R, Mayhew D, McGowan S, Polter A, Qadri Y, Sarfare S, Schultz K, Splittgerber R, Stephenson J, Tower C, Walton RG, Zotov A. The ups and downs of peer review. Adv Physiol Educ. 2007 Jun;31(2):145-52.

Berg JM. Science policy: Well-funded investigators should receive extra scrutiny. Nature. 2012 Sep 13;489(7415):203.

Bhattacharya A. Science funding: Duel to the death. Nature. 2012 Aug 2;488(7409):20-2.

Bielski A, Harris R, Gillis N. Summary report of comments received on NIH system to support biomedical and behavioral research and peer review. 2007 November.

Bornmann L. Scientific peer review: an analysis of the peer review process from the perspective of sociology of science theories. Human Architecture: Journal of the Sociology of Self-Knowledge. 2008;VI(2):23-38.

Brown T. Perspective: 'I don't know what to believe'. Nature 2006.

Chang WR, McLean IP. CUSUM: a tool for early feedback about performance? BMC Med Res Methodol. 2006 Mar 2;6:8.

Cole S, Rubin L, Cole J. Peer review in the National Science Foundation: Phase one of a study. Washington, DC: National Academy of Sciences, 1978.

Council of Canadian Academies. Informing research choices: Indicators and judgment. 2012.

Das NK, Froehlich LA. Quantitative evaluation of peer review of program project and center applications in allergy and immunology. J Clin Immunol. 1985 Jul;5(4):220-7.

DeCoursey T. Perspective: The pros and cons of open peer review. Nature 2006.

Demicheli V, Di Pietrantonj C. Peer review for improving the quality of grant applications. Cochrane Database Syst Rev. 2007 Apr 18;(2):MR000003.

Dickersin K, Ssemanda E, Mansell C, Rennie D. What do the JAMA editors say when they discuss manuscripts that they are considering for publication? Developing a schema for classifying the content of editorial discussion. BMC Med Res Methodol. 2007 Sep 25;7:44.

Fliesler SJ. Rethinking grant peer review. Science. 1997 Mar 7;275(5305):1399.

Glenn JF. Uses and needs for peer review in Army medical research. Technol Innov. 2010;12:241-7.

Graves N, Barnett AG, Clarke P. Funding grant proposals for scientific research: retrospective analysis of scores by members of grant review panel. BMJ. 2011 Sep 27;343:d4797.

Greaves S, Scott J, Clarke M, Miller M, Hannay T, Thomas A, Campbell P. Overview: Nature's peer review trial. Nature. 2006.

Groves T. Quality and value: How can we get the best out of peer review? Nature. 2006.

Guthrie S, Guerin B, Wu H, Ismail S, Wooding S. Alternatives to peer review in research project funding. Santa Monica, CA: RAND Corporation, 2013.

Hand E. Extra scrutiny for 'grandee grantees'. Nature. 2012 Feb 20;482(7386):450-1.

Hartmann I, Neidhardt F. Peer review at the Deutsche Deutsche Forschungsgemeinschaft. Scientometrics. 1990 Nov 01;19(5-6):419-25.

Hartonen T, Alava MJ. How important tasks are performed: peer review. Sci Rep. 2013;3:1679.

Herbert DL, Barnett AG, Clarke P, Graves N. On the time spent preparing grant proposals: an observational study of Australian researchers. BMJ Open. 2013 May 28;3(5). pii: e002800.

Herbert DL, Barnett AG, Graves N. Funding: Australia's grant system wastes time. Nature. 2013 Mar 21;495(7441):314.

Ho RC, Mak KK, Tao R, Lu Y, Day JR, Pan F. Views on the peer review system of biomedical journals: an online survey of academics from high-ranking universities. BMC Med Res Methodol. 2013 Jun 7;13:74.

Houser SR. How to obtain a National Heart, Lung, and Blood Institute-sponsored K08 and K99/R00 grant in the current funding climate. Circ Res. 2012 Mar 30;110(7):907-9.

IOM. Strategies for managing the Breast Cancer Research Program: A report to the U.S. Army Medical Research and Development Command. Washington, DC: National Academy Press, 1993.

Irwin D, Gallo SA, Glisson SR. Opinion: Learning from peer review. The Scientist. 2013 May 24.

Ismail S, Farrands A, Wooding S. Evaluating grant peer review in the health sciences: A review of the literature. Santa Monica, CA: RAND Corporation, 2009.

Jefferson T. Quality and value: Models of quality control for scientific research. Nature. 2006.

Jennings C. Quality and value: The true purpose of peer review. Nature. 2006.

Johnson VE. Statistical analysis of the National Institutes of Health peer review system. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

Koonin E, Landweber L, Lipman D, Dignon R. Systems: Reviving a culture of scientific debate. Nature. 2006.

Koop T, Pöschl U. Systems: An open, two-stage peer-review journal. Nature. 2006.

Lahiri D. Perspective: The case for group review. Nature 2006.

Langfeldt L. The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science. 2001 Dec;31(6):820-41.

Langfeldt L. Decision-making in expert panels evaluating research: Constraints, processes and bias. 2001;dissertation: The University of Oslo.

Lee C. Perspective: Peer review of interdisciplinary scientific papers. Nature 2006.

Lee K, Bero L. Ethics: Increasing accountability. Nature 2006.

Lee K, Brownstein JS, Mills RG, Kohane IS. Does collocation inform the impact of collaboration? PLoS One. 2010 Dec 15;5(12):e14279.

McGeary M, Hanna KE. Strategies to leverage research funding: Guiding DOD's Peer Reviewed Medical Research Programs. Washington, DC: National Academies Press, 2004.

Mervis J. Peer review. Beyond the data. Science. 2011 Oct 14;334(6053):169-71.

Moed H, Glanzel W, Schmoch U. Handbook of Quantitative Science and Technology Research: The Use of Publication and Patent Statistics in Studies of S&T Systems. New York, NY: Springer, 2004.

Moore J. Perspective: Does peer review mean the same to the public as it does to scientists? Nature 2006.

Nicholson J, Ioannidis J. Research grants: Conform and be funded. Nature. 2012 Dec 6;492:34-36.

Ozonoff D. Quality and value: Statistics in peer review. Nature. 2006.

Pawelczyk JA, Strawbridge LM, Schultz AM, Liverman CT. A review of NASA Human Research Program's scientific merit processes: Letter report. Washington, DC: National Academies Press, 2012.

Pederson T. The "study" role of past National Institutes of Health study sections. Mol Biol Cell. 2012 Sep;23(17):3281-4.

Peer Review. Parliamentary Office of Science and Technology. 2002;182.

Ranalli B. A prehistory of peer review: Religious blueprints from the Hartlib Circle. Spontaneous Generations. 2011;5(1):12-18.

Riley B. Systems: Trusting data's quality. Nature. 2006.

Roebber PJ, Schultz DM. Peer review, program officers and science funding. PLoS One. 2011 Apr 12;6(4):e18680.

Russell AS, Thorn BD, Grace M. Peer review: a simplified approach. J Rheumatol. 1983 Jun;10(3):479-81.

Sandewall E. Systems: Opening up the process. Nature. 2006.

Sandstrom U, Hallsten M. Persistent nepotism in peer-review. Scientometrics. 2008 Feb;74(2):175-89.

Schroter S, Groves T, Højgaard L. Surveys of current status in biomedical science grant review: funding organisations' and grant reviewers' perspectives. BMC Med. 2010 Oct 20;8:62.

Shashok K. Content and communication: how can peer review provide helpful feedback about the writing? BMC Med Res Methodol. 2008 Jan 31;8:3.

Shideed O, Al-Gasseer N. Appraisal of the research grant schemes of the World Health Organization Regional Office for the Eastern Mediterranean: the way forward. East Mediterr Health J. 2012 May;18(5):515-21.

Sieber J. Quality and value: How can we research peer review? Nature 2006.

Smith R. Classical peer review: an empty gun. Breast Cancer Res. 2010 Dec 20;12 Suppl 4:S13.

Smith R. Peer review: A flawed process at the heart of science and journals. J R Soc Med. 2006 Apr;99(4):178-82.

Squazzoni F, Takacs, K. Social simulation that "peers into peer review." J Artif Soc Soc Simulat. 2011 Oct 31;14(4):3.

Turner RS. Best practices in peer review assure quality, value, objectivity. Journal of the National Grants Management Association. 2009;17(1):43-8.

United States General Accounting Office. Federal research: Peer review practices at federal science agencies vary. Report to Congressional Requesters. 2009 March.

Van de Sompel H. Technical solutions: Certification in a digital era. Nature 2006.

Wessely S. Peer review of grant applications: what do we know? Lancet. 1998 Jul 25;352(9124):301-5.

Zare RN, Winnacker EL. China's science funding. Science. 2011 Oct 28;334(6055):433.

Back to top


H-Index & Citations

Bornmann L, Daniel HD. The state of h index research. Is the h index the ideal way to measure research performance? EMBO Rep. 2009;10(1):2-6.

Farhadi H, Salehi H, Md Yunus M, Chadegani AA, Farhadi M, Fooladi M, Ebrahim NA. Does it Matter Which Citation Tool is Used to Compare the h-index of a Group of Highly Cited Researchers? Aust J Basic Appl Sci. 2013;7(4):198-202.

Hirsch JE. An index to quantify an individual's scientific research output. Proc Natl Acad Sci U S A. 2005 Nov 15;102(46):16569-72.

Kinney AL. National scientific facilities and their science impact on nonbiomedical research. Proc Natl Acad Sci U S A. 2007 Nov 13;104(46):17943-7.

Nieminen P, Carpenter J, Rucker G, Schumacher M. The relationship between quality of research and citation frequency. BMC Med Res Methodol. 2006 Sep 1;6:42.

Thomas Reuters. Using bibliometrics: A guide to evaluating research performance with citation data. 2008.

Sidiropoulos A, Katsaros D, Manolopoulos Y. Generalized Hirsch h-index for disclosing latent facts in citation networks. Scientometrics. 2007 Aug 1;72(2):253-280.

Sypsa V, Hatzakis A. Assessing the impact of biomedical research in academic institutions of disparate sizes. BMC Med Res Methodol. 2009 May 29;9:33.

Back to top


Innovation

Horrobin DF. The philosophical basis of peer review and the suppression of innovation. JAMA. 1990 Mar 9;263(10):1438-41.

Kaime EM, Moore KH, Goldberg SF. CDMRP: Fostering innovation through peer review. Technol Innov. 2010;12:233-40.

Kaplan D. Point: Statistical analysis in NIH peer review--identifying innovation. FASEB J. 2007 Feb;21(2):305-8.

Luukkonen T. Conservatism and risk-taking in peer review: Emerging ERC practices. Research Evaluation. 2012;21:48-60.

Back to top


Inter-rater Reliability

Bornmann L, Mutz R, Daniel HD. A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants. PLoS One. 2010 Dec 14;5(12):e14331.

Cicchetti DV. The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behav Brain Sci. 1991;14:119-86.

Colliver J. Commentary on Cichetti's "reliability of peer review." Teach Learn Med. 2002;14(3):142-3.

Fleiss JL, Shrout PE. The effects of measurement errors on some multivariate procedures. Am J Public Health. 1977 December; 67(12): 1188-91.

Fogelholm M, Leppinen S, Auvinen A, Raitanen J, Nuutinen A, Väänänen K. Panel discussion does not improve reliability of peer review for medical research grant proposals. J Clin Epidemiol. 2012 Jan;65(1):47-52.

Giraudeau B, Leyrat C, Le Gouge A, Léger J, Caille A. Peer review of grant applications: a simple method to identify proposals with discordant reviews. PLoS One. 2011;6(11):e27557.

Hallgren KA. Computing Inter-Rater Reliability for observational data: An Overview and tutorial. Tutor Quant Methods Psychol. 2012;8(1):23-34.

Hargens LL, Herting JR. Neglected considerations in the analysis of agreement among journal referees. Scientometrics. 1990 Jul 01;19(1-2):91-106.

Hodgson C. How reliable is peer review? An examination of operating grant proposals simultaneously submitted to two similar peer review systems. J Clin Epidemiol. 1997 Nov;50(11):1189-95.

Jayasinghe UW, Marsh HW, Bond, N. A multilevel cross-classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings. J R Statist Soc A. 2003;166(3):279-300.

Johnson VE. Statistical analysis of the National Institutes of Health peer review system. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

Kravitz RL, Franks P, Feldman MD, Gerrity M, Byrne C, Tierney WM. Editorial peer reviewers' recommendations at a general medical journal: are they reliable and do editors care? PLoS One. 2010 Apr 8;5(4):e10072.

Marsh HW, Jayasinghe UW, Bond NW. Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability. Am Psychol. 2008 Apr;63(3):160-8.

Mutz R, Bornmann L, Daniel HD. Heterogeneity of inter-rater reliabilities of grant peer reviews and its determinants: A general estimating equations approach. PLoS One. 2012;7(10):e48509.

Rothwell PM, Martyn CN. Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? Brain. 2000 Sep;123 (Pt 9):1964-9.

Back to top


NIH & NSF

Berg J. NIH-wide correlations between overall impact scores and criterion scores. NIGMS Feedback Loop Blog. 2010 Sep 30.

Berg J. Scoring analysis with funding and investigator status. NIGMS Feedback Loop Blog. 2010 Sep 14.

Bielski A, Harris R, Gillis N. Summary report of comments received on NIH system to support biomedical and behavioral research and peer review. 2007 November.

NIH. 2007-2008 Peer review self-study. 2008.

NIH. A self study by the NIH in partnership with the scientific community to strengthen peer review in changing times. 2008.

NIH. Chair orientation. 2012 Mar 5.

NIH. Department of Health and Humans National Institutes of Health FY 2013 Budget. 2012.

NIH. Encouraging early transition to research independence: Modifying the NIH New Investigator policy to identify Early Stage Investigators. NOT-OD-08-121. 2008 Sep 26.

NIH. Enhancing peer review: Implementation of recommended actions. Transcript accompanying video of Dr. Alan Willard. 2009 Feb.

NIH. Enhancing peer review: The NIH announces enhanced review criteria for evaluation of research applications received for potential FY2010 funding. NOT-OD-09-025. 2008 Dec 2.

NIH. Enhancing peer review: The NIH announces new scoring procedures for evaluation of research applications received for potential FY2010 funding. NOT-OD-09-024. 2008 Dec 2.

NIH. Evaluation summary of new 1-9 scoring system. 2009 Oct 13.

NIH. Initial peer review conflict of interest policy.

NIH. Instructions for use of the review critique template. 2009 Apr 21.

NIH. NIH peer review: Grants and cooperative agreements. 2013.

NIH. NIH success rate definition. 2009 Feb.

NIH. NIH success rate definition. 2012 Feb.

NIH. Overall Impact versus Significance. 2012 Mar 5.

NIH. R01 guide for reviewers. Investigator Initiated Research Project grant applications. 2010 Feb 2.

NIH. Research Project Grant (RPG) critique template. R01/R03/R15/R21/R34 Review. 2012 Mar 5.

NIH. Reviewer orientation. 2012 Dec 14.

NIH. Revised policy: Managing conflict of interest in the initial peer review of NIH grant and cooperative agreement applications. NOT-OD-11-120. 2011 Sep 26.

NIH. Scoring and review changes. 2009 Oct 22.

NIH. Scoring system and procedure. 2009 Mar 20.

NIH. Side-by-side comparison of enhanced and former review criteria. 2009 Dec 1.

NSF. National Science Foundation's merit review criteria: Review and revisions. 2011 Dec 14.

Tabak LW. Report on enhancing peer review at NIH Implementation Plan. NIH. 2008 Jun 6.

Zerhouni E. Some observations on demographics of NIH-funded scientists: Policy implications for new investigators. 2007 Dec 7.

Back to top


Panel Discussion

Cole S, Cole JR, Simon GA. Chance and consensus in peer review. Science. 1981 Nov 20;214(4523):881-6.

Fogelholm M, Leppinen S, Auvinen A, Raitanen J, Nuutinen A, Väänänen K. Panel discussion does not improve reliability of peer review for medical research grant proposals. J Clin Epidemiol. 2012 Jan;65(1):47-52.

Johnson VE. Statistical analysis of the National Institutes of Health peer review system. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

Martin MR, Kopstein A, Janice JM. An analysis of preliminary and post-discussion priority scores for grant applications peer reviewed by the Center for Scientific Review at the NIH. PLoS One. 2010 Nov 17;5(11):e13526.

Obrecht M, Tibelius K, D'Aloisio G. Examining the value added by committee discussion in the review of applications for research awards. Research Evaluation. 2007 Jun;16(2):79-91.

Back to top


Peer Review Technology

Akerman R. Technical solutions: Evolving peer review for the internet. Nature 2006.

Anderson C. Technical solutions: Wisdom of the crowds. Nature 2006.

Bloom T. Systems: Online frontiers of the peer-reviewed literature. Nature. 2006.

Bohannon J. National Science Foundation. Meeting for peer review at a resort that's virtually free. Science. 2011 Jan 7;331(6013):27.

Driskell JE, Radtke PH, Salas E. Virtual teams: Effects of technological mediation on team performance. Group Dyn. 2003 Dec;7(4):297-323.

Gallo SA, Carpenter AS, Glisson SR (2013) Teleconference versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes. PLoS ONE 8(8).

Jenkins SL, Iyengar R, Diverse-Pierluissi MA, Chan AM, Devi LA, Sobie EA, Ting AT, Weinstein DC. Teaching resources. Using web-based discussion forums as a model of the peer-review process and a tool for assessment. Sci Signal. 2008;1(9):tr2.

Back to top


Resubmission

Bornmann L, Mutz R, Daniel H-R. Latent Markov modeling applied to grant peer review. J Informetr. 2008;2(4):217-28.

Back to top


Reviewer/PI Demographics

Kaplan D, Lacetera N, Kaplan C. Sample size and precision in NIH peer review. PLoS One. 2008 Jul 23;3(7):e2761.

Zerhouni E. Some observations on demographics of NIH-funded scientists: Policy implications for new investigators. 2007 Dec 7.

Back to top


Scoring Scale

Cicchetti DV, Shoinralter D, Tyrer PJ. The effect of number of rating scale categories on levels of interrater reliability: A Monte Carlo investigation. Appl Psychol Meas. 1985 Mar;9(1):31-6.

Green JG, Calhoun F, Nierzwicki L, Brackett J, Meier P. Rating intervals: an experiment in peer review. FASEB J. 1989 Jun;3(8):1987-92.

NIH. Evaluation summary of new 1-9 scoring system. 2009 Oct 13.

Back to top


Team Performance

Barabási AL. Sociology. Network theory--the emergence of the creative enterprise. Science. 2005 Apr 29;308(5722):639-41.

Borgatti SP, Mehra A, Brass DJ, Labianca G. Network analysis in the social sciences. Science. 2009 Feb 13;323(5916):892-95.

Curşeu PL, Jansen RJ, Chappin MM. Decision rules and group rationality: cognitive gain or standstill? PLoS One. 2013;8(2):e56454.

Duch J, Waitzman JS, Amaral LA. Quantifying the performance of individual players in a team activity. PLoS One. 2010 Jun 16;5(6):e10937.

Evans JA, Foster JG. Metaknowledge. Science. 2011 Feb 11;331(6018):721-5.

Guimerà R, Uzzi B, Spiro J, Amaral LA. Team assembly mechanisms determine collaboration network structure and team performance. Science. 2005 Apr 29;308(5722):697-702.

Jeffcott SA, Mackenzie CF. Measuring team performance in healthcare: review of research and implications for patient safety. J Crit Care. 2008 Jun;23(2):188-96.

Langfeldt L. Decision-making in expert panels evaluating research: Constraints, processes and bias. 2001;dissertation: The University of Oslo.

Manthous CA, Hollingshead AB. Team science and critical care. Am J Respir Crit Care Med. 2011 Jul 1;184(1):17-25.

Roebber PJ, Schultz DM. Peer review, program officers and science funding. PLoS One. 2011 Apr 12;6(4):e18680.

Rogelberg SG, O'Connor MS, Sederburg M. Using the stepladder technique to facilitate the performance of audioconferencing groups. J Appl Psychol. 2002 Oct;87(5):994-1000.

Rosen MA, Weaver SJ, Lazzara EH, Salas E, Wu T, Silvestri S, Schiebel N, Almeida S, King HB. Tools for evaluating team performance in simulation-based training. J Emerg Trauma Shock. 2010 Oct;3(4):353-9.

Stokols D, Misra S, Moser RP, Hall KL, Taylor BK. The ecology of team science: understanding contextual influences on transdisciplinary collaboration. Am J Prev Med. 2008 Aug;35(2 Suppl):S96-115.

Wickens CD, Holland JG, Parasuraman R, Banbury S. Engineering Psychology and Human Performance (4th ed). Upper Saddle River, NJ: Prentice Hall, 2012.

Back to top


Triage

Vener KJ, Feuer EJ, Gorelic L. A statistical model validating triage for the peer review process: keeping the competitive applications in the review pipeline. FASEB J. 1993 Nov;7(14):1312-9.

Back to top


Validation

Beacham L, Li E, Wasserman A. Measuring the performance of extramural funding at the National Library of Medicine. The College of William and Mary. 2008.

Berg J. Productivity metrics and peer review scores. NIGMS Feedback Loop Blog. 2011 Jun 2.

Bornmann L, Daniel HD. Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees' decisions. Scientometrics. 2005;63(2):297-320.

Bornmann L, Wallon G, Ledin A. Does the committee peer review select the best applicants for funding? An investigation of the selection process for two European molecular biology organization programmes. PLoS One. 2008;3(10):e3480.

Bornmann L, Daniel HD. The usefulness of peer review for selecting manuscripts for publication: a utility analysis taking as an example a high-impact journal. PLoS One. 2010 Jun 28;5(6):e11344.

Claveria LE, Guallar E, Cami J, Conde J, Pastor R, Ricoy JR, Rodriguez-Farre E, Ruiz-Palomo F, Munoz E. Does peer review predict the performance of research projects in health sciences? Scientometrics. 2000 Jan 1;47(1):11-23.

Cole S, Cole JR, Simon GA. Chance and consensus in peer review. Science. 1981 Nov 20;214(4523):881-6.

Goldbeck-Wood S. Evidence on peer review--scientific quality control or smokescreen? BMJ. 1999 January 2;318(7175):44-5.

Hodgson C. Evaluation of cardiovascular grant-in-aid applications by peer review: influence of internal and external reviewers and committees. Can J Cardiol. 1995 Nov;11(10):864-8.

Ip EH, Wasserman R, Barkin S. Comparison of intraclass correlation coefficient estimates and standard errors between using cross-sectional and repeated measurement data: the Safety Check cluster randomized trial. Contemp Clin Trials. 2011 Mar;32(2):225-32.

Jackson JL, Srinivasan M, Rea J, Fletcher KE, Kravitz RL. The validity of peer review in a general medicine journal. PLoS One. 2011;6(7):e22475.

Kotchen TA, Lindquist T, Miller Sostek A, Hoffmann R, Malik K, Stanfield B. Outcomes of National Institutes of Health peer review of clinical grant applications. J Investig Med. 2006 Jan;54(1):13-9.

Marsh HW, Jayasinghe UW, Bond NW. Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability. Am Psychol. 2008 Apr;63(3):160-8.

Martin MR, Lindquist T, Kotchen TA. Why are peer review outcomes less favorable for clinical science than for basic science grant applications? Am J Med. 2008 Jul;121(7):637-41.

Moed HF, Glanzel W, Schmoch U. Handbook of Quantitative Science and Technology Research: The Use of Publications and Patent Statistics in Studies of S&T Systems. New York, NY: Kluwer Academic Publishers, 2004.

Rinia EJ, Leeuwen Th N, Vuren HG, Raan AFJ. Comparative analysis of a set of bibliometric indicators and central peer review criteria: Evaluation of condensed matter physics in the Netherlands. Research Policy. 1998 May;27(1):95-107.

Rinia EJ, Leeuwen Th N, Vuren HG, Raan AFJ. Influence of interdisciplinarity on peer-review and bibliometric evaluations in physics research. Research Policy. 2001 Mar 1;30(3):357-61.

Shashok K. Standardization vs diversity: How can we push peer review research forward? MedGenMed. 2005 Feb 17;7(1):11.

Turner J. Best practices in merit review: A report to the US Department of Energy. 2010 Dec.

Vale RD. Evaluating how we evaluate. Mol Biol Cell. 2012 Sep;23(17):3285-9.

van Raan AFJ. Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics. 2006;67(3):491-502.

Waltman L, van Eck NJ, van Leeuwen TN, Visser MS, van Raan AF. On the correlation between bibliometric indicators and peer review: reply to Opthof and Leydesdorff. Scientometrics. 2011 Sep;88(3):1017-1022.

Bookmark and Share


News from SPARS



Scientists Review Peer Review


You and your colleagues at AIBS did an absolutely first rate job in running this review.

— Tinnitus Review, 2011

The review panel was thorough, fair and comprehensive.

— Tinnitus Review, 2011

It was remarkable and very different from traditional NIH study sections.

— CDC Reviewer, 2011

You did a great job putting together a wonderful panel: knowledgeable and passionate with their work, ready to go an extra mile to help new investigators in Africa.

— CDC Reviewer, 2011

We always benefit from critical review of works in our field-it makes us better writers - it points out where confusion and inconsistencies make scientific writing hard to understand or follow. It is always a good thing to critically review others' work!

— Raptor Biologist, 2011

With face-to-face reviews, I meet other scientists with interests and knowledge different than my own, which provides me a resource for my own work.

— Cell Biology NYStem reviewer 2010

I enjoy the discussion and learn from the expertise of other scientists. I also benefit by preparing better grants.

— Hematologist/Oncologist, NYStem reviewer 2010