Abigail Stewart, Ph.D.
Sandra Schwartz Tangri Distinguished University Professor of Psychology and Women's and Gender Studies, University of Michigan
INQUIRY #6: Judging Others: Excellence, Merit, and Peer Review
Universities like those in this consortium rely on decades of reputation for their excellence--and they can. But continued, actual excellence depends on how well our faculty assess it in individuals. It turns out that we're just not as good at that as we think we are.
Relying on abundant research in psychology, medicine and other disciplines, Prof. Abigail Stewart introduces us to the evidence that we are poor judges of "merit" in the myriad opportunities for judgment that faculty have.
"The good news," she concludes, "is that what leaders do matters—or maybe that’s the bad news." Stewart does not leave us hopeless, however: she offers recommendations about what we can do to fix this.
INQUIRY RESOURCES
“Judging Others: Excellence, Merit, and Peer Review” (21m 53s), Prof. Stewart’s video prologue to this inquiry.
Stewart, A. J., & Valian, V. (2018). An Inclusive Academy. MIT Press.
Forecasting
Colombo, D. et al. (2020). Biased affective forecasting: A potential mechanism that enhances resilience and well-being. Frontiers in Psychology, https://doi.org/10.3389/fpsyg.2020.01333
Fisher, K. & Statman, M. (2000). Cognitive biases in market forecasts: The frailty of forecasting. The Journal of Portfolio Management, 27(1), 72-81.
Mellors, B.A, et al., (2015). Identifying and cultivating “superforecasters” as a method of improving probabilistic predictions. Perspectives on Psychological Science. 10 (3): 267–281.
Tetlock, P. E. (2005). Expert political judgment: How good is it? How can we know? Princeton: Princeton University Press.
Tetlock, P.E., Mellors, B., Rohrbaugh, N., Chen, E. (2014). Forecasting tournaments: Tools for increasing transparency and the quality of debate. Current Directions in Psychological Science. 23 (4): 290–295.
Fundamental Attribution Error and Related Cognitive Error
Deaux, K. & Emswiller, T. (1974). Explanations of successful performance on sex-linked tasks: What is skill for the male is luck for the female. Journal of Personality and Social Psychology, 29 (1), 80-85.
Forgas, P. (2011). She just doesn’t look like a philosopher:…? Affective influences on the halo effect in impression formation. European Journal of Social Psychology, 41, 812-817
Jackson, L.A., Sullivan, L.A., & Hodge, C.N. (1993). Stereotype effects on attributions, predictions, and evaluations: No two social judgments are quite alike. Journal of Personality and Social Psychology, 65 (1), 69-84.
Jones, E.E. & Harris, V.A. (1967). The attribution of attitudes. Journal of Experimental Social Psychology, 3, 1-24.
Lerner, Melvin J. (1980). The belief in a just world: A fundamental delusion. NY: Plenum.
Moore, D.A., Swift, S.A., Sharek, Z.S. , & Gino, F. (2010). Correspondence bias in performance evaluation: Why grade inflation works. Personality and Social Psychology Bulletin, 36(6), 843-852.
Ryan, William (1976). Blaming the victim. NY: Vintage.
Group-Based Schemas and Resulting Biases
Banaji, M. & Greenwald, A. (2013) Blindspot: Hidden biases of good people. Delacorte.
Bertrand & Mullainathan (2004). Are Emily and Greg more employable than Lakisha and Jamal? American Economic Review, 94(4), 991-1013.
Correll, S.J., Benard, S., & Paik, I. (2007). Getting a job: Is there a motherhood penalty? American Journal of Sociology, 112(5), 1297-1338.
Dovidio & Gaertner (1998). In Eberhardt & Fiske (Eds.), Confronting racism: The problem and the response (pp. 3-32). Newbury Park: Sage.
Dovidio & Gaertner (2000). Aversive racism and selection decisions: 1989 and 1999. Psychological Science, 11(4), 315-319.
Eaton, A.A., Saunders, J.F., Jacobson, R.K., & West, K. (2019). How gender and race stereotypes impact the advancement of scholars in STEM: Professors’ biased evaluations of physics and biology postdoctoral candidates. Sex Roles. https://doi.org/10.1007/S11199-019-01052-w
Eberhardt, J. (2019). Biased: Uncovering the hidden prejudice that shapes what we see, think, and do. Viking.
Fiske (2002). What we know now about bias and intergroup conflict, problem of the century. Current Directions in Psychological Science, 11(4), 123-128
Fiske, Cuddy, Glick, & Xu (2002). A model of (often mixed) stereotype content: Competence and warmth respectively follow from perceived status and competition. J Personality and Social Psychology, 82(6), 878-902.
Ginther et al., D.K. Race, ethnicity and NIH research awards. (2011). Science, 333, 1015-1019.
Heilman (1980). The impact of situational factors on personnel decisions concerning women: Varying the sex composition of the applicant pool. Organizational Behavior and Human Performance (26), 386-395.
Heilman (2018). Combatting gender discrimination: A lack of fit framework. Group Processes and Intergroup Relations, 21 (5), 725-744.
Moss-Racusin, C.A., Dovidio, J.F., Brescoll, V.L., Graham, M.J., & Handelsman, J. (2012). Science faculty’s subtle gender biases favor male students. PNAS 109(41), 16474-16479
Nosek, Banaji, & Greenwald (2002). Group Dynamics: Theory, Research, and Practice, 6(1), 101-115.
Rosette, A.S., et al. (2008). The White standard: Racial bias in leader categorization. J Applied Psychology, 93(4), 758-777
Steinpreis, R.E., Anders, K.A., & Ritzke, D. (1999). The impact of gender on the review of the curricula vitae of job applicants and tenure candidates: A national empirical study. Sex Roles, 41(7/8), 509-528.
Ross, H.J. (2014). Everyday bias. NY: Rowman & Littlefield.
Sackett, DuBois, & Noe (1991). Tokenism in performance evaluation: The effects of work group representation on male-female and White-Black differences in performance ratings. J Applied Psychology, 76(2), 263-267.
Sy, et al. (2010). Leadership perceptions as a function of race-occupation fit: The case of Asian Americans. J Applied Psychology, 95(5), 902-919.
Valian, V. (1998) Why So Slow? The Advancement of Women. Cambridge: MIT Press, p. 280.
Steele, C. (2010). Whistling Vivaldi and other clues to how stereotypes affect us. Norton.
Limitations in Judging Merit
Kahneman, D. (2011). Thinking, fast and slow. NY: Farrar, Straus and Giroux.
Sandström, U. & Hällsten, M. (2008). Persistent nepotism in peer-review. Scientometrics, 74(2), 175-189.
Tetlock, P. E. (2005). Expert political judgment: How good is it? How can we know? Princeton: Princeton University Press.
Thorngate, W., Dawes, R.M., & Foddy, M. (2009). Judging merit. NY: Psychology Press.
Ross, H.J. (2014). Everyday bias. NY: Rowman & Littlefield.
Tversky, A. & Kahneman, D. (1974). Judgment under uncertainty: Heuristic and biases. Science, 185 (4157), 1124-1131
Uhlmann, E. & Cohen, G. (2007). ‘I think it, therefore it’s true’: Effects of self-perceived objectivity on hiring discrimination. Organizational Behavior and Human Decision Processes, 104, 207-233.
Wenneras, C. & Wold, A. Sexism and nepotism in peer review. (1997) Nature, 387, 341-343.
Proxies for Excellence
Bertolero, M., Dworkin, J.D., David, S.U., et al (2020). Racial and ethnic imbalances in neuroscience reference lists and intersections with gender. bioRxiv, doi: https://doi.org/10.1101/2020.10.12.336230
Chatterjee, P. & Werner, R.M. (2021) Gender disparities in citations of high-impact journal articles. JAMA Network Open, 4(7):e2114509. doi:10.1001/jamanetworkopen.2021.14509
Clauset, A., Arbesman, S., & Larremore, D.B. (2015, 12 February). Systematic inequality and hierarchy in faculty hiring networks. Science Advances.
Dworkin, J.D., Linn, K.A., Teich, E.G., et al (2020). The extent and drivers of gender imbalance in neuroscience reference lists. Nature Neuroscience, 23, 918–926
Melnikoff, D.E. & Valian, V. (2019). Gender disparities in awards to neuroscience researchers. Archives of Scientific Psychology, 7(1), 4–11. https://doi.org/10.1037/arc0000069
Peters, D. P. & Ceci, S. J. 1982. Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5, 187-255.
Sine, W.D., Shane, S., & DiGregorio, D. (2003). The halo effect and technology licensing: The influence of institutional prestige on the licensing of university inventions. Management Science,49 (4), 478-496
Skitka, L.J., Melton, Z.J., Mueller, A.B., et al. (2021) The gender gap: Who is (and is not) included on graduate-level syllabi in social/personality psychology. Personality and Social Psychology Bulletin, 47(6). 863-872.
Diversity, Homophily and Faculty Networks and their Role in Hiring and Careers
Belle, D., Smith-Doerr, L., & O’Brien, L.M. (2014). Gendered networks: Professional connections of science and engineering faculty (153-175). In V. Demos, C.W. Berheide, & M. Texler Segal (Eds.), Gender Transformations in the Academy (2014). UK: Emerald Group.
Clauset, A., Arbesman, S., & Larremore, D.B. (2015, 12 February). Systematic inequality and hierarchy in faculty hiring networks. Science Advances
Hofstra, B., Kulkarni, V.V., Galvez, S.M., & McFarland, D.A. (2020). The diversity-innovation paradox in science. PNAS, 117(17), 9284-9291.
Kazmi, M. A., Spitzmueller, C., Yu, J., Madera, J. M., Tsao, A. S., Dawson, J. F., & Pavlidis, I. (2021). Search committee diversity and applicant pool representation of women and underrepresented minorities: A quasi-experimental field study. Journal of Applied Psychology, 107(8), 1414–1427.
Nielsen, M.W., Alegria, S., Börjeson, L., Etzkowitz, H., Falk-Krzesinski, H.J., Joshi, A., Leahey, E., Smith-Doerr, L., Woolley, A.W., & Schiebinger, L. (2017). Gender diversity leads to better science. PNAS, 114(8), 1740-1742.
Nielsen, M.W., Bloch, C.W., & Schiebinger, L. (2018). Making gender diversity work for scientific discovery and innovation. Nature Human Behavior, 2, 726-734.
Page, Scott (2007). The difference: How the power of diversity creates better groups, firms, schools, and societies. Princeton University Press.
Page, Scott (2017). The diversity bonus: How great teams pay off in the knowledge economy. Princeton University Press.
Peters, D. P. & Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5, 187-255.
Phillips, K. W. & Lloyd, D.L. (2006). When surface and deep-level diversity collide: The effects on dissenting group members. Organizational Behavior and Human Decision Processes, 99(2), 143-160.
Way, S.F., Morgan, A.C., Larremore, D.B., & Clauset, A. (2019). Productivity, prominence, and the effects of academic environment. PNAS, 116 (22), 10729-10733.
Efforts to Remove Bias
Chan, W. & Mendelsohn, G. A. (2010). Disentangling stereotype and person effects: Do social stereotypes bias observer judgments of personality? Journal of Research in Personality, 44, 251-257.
Goldin & Rouse (2000). Orchestrating impartiality: The impact of blind auditions on female musicians. The American Economic Review, 90(4), 715-741.
Ross, H.J. (2014). Everyday bias. NY: Rowman & Littlefield.