ACM Logo  An ACM Publication  |  CONTRIBUTE  |  FOLLOW    

The future of assessment depends on elevating culturally diverse perspectives

Special Issue: Advancing Beyond Multiple Choice eAssessment

By Susan Lyons / September 2021

Print Email
Comments Instapaper

In this opinion piece, Susan Lyons explores the imperative to amplify diverse voices and perspectives in the field of educational measurement. She identifies two negative effects of the lack of cultural diversity in leadership positions within the field and ends by calling for a higher level of critical consciousness within the profession of educational measurement.

People of color and other minoritized individuals have a long history of leadership and impact on the field of educational measurement. For example, Professor Edmund Gordon, who is celebrating his centennial this year, has been a visionary in our field for more than 60 years—pushing the field to envision a future of assessment that more directly serves teaching and learning. Despite many powerful examples of diverse leadership, those who hold the most influential positions continue to be overwhelmingly White and male. While this is not unique to educational measurement, there are likely specific, undesirable effects on our assessment practices and products when the decision makers are almost exclusively members of the dominant culture. Two of the possible unintended negative effects of a lack of cultural diversity in decision-making at the highest levels of our field include:

  1. Adoption of professional practices and values that reinforce existing social structures; and
  2. Assessment designs that center White culture and privilege dominant groups.

Potential Differences in Professional Practices and Values

Had our field leveraged the benefits of cultural diversity in leadership from the beginning, it's highly likely that differences in accepted professional practices and values would have emerged. Moss, Pullin, et al. [1] remind us, "Beliefs and practices informed by psychometrics have become so deeply engrained in the American educational system that it has become difficult to see them as choices arising in particular sociocultural circumstances or to imagine that things could be otherwise." Forward-thinking scholars are now helping us to see how things could be otherwise. For example, Randall confronts the white hegemony when discussing how our field has traditionally conceptualized issues of construct-irrelevant variance, suggesting that assessment items should be developed to be explicitly anti-racist [2]. Randall argues items that are not actively anti-racist are serving to further marginalize students of color.

In his presidential address to the National Council on Measurement in Education, Sireci makes a similar argument about comparability [3]. Our notion of comparability in psychometrics derives from a positivistic epistemology that privileges a single, dominant interpretation of the construct and demands a single acceptable form of behavioral evidence to demonstrate achievement on that construct. Sireci argues that the hyper-focus on standardization in educational measurement leads to exclusion, "and the goal of educational measurement is not to measure the students who are easiest to measure and who conform to the most dominant culture associated with the measurement enterprise, but rather to obtain the best measure of each and every student's proficiencies." These two papers challenge us to envision how our psychometric values of construct validity and comparability might evolve as we seek to elevate diverse perspectives and develop the assessments of the future.

Potential Differences in Test Design

As with our psychometric practices and values, it's highly likely that involving diverse voices in defining and operationalizing our measured constructs will introduce differences in the types of evidence of learning we privilege in our assessment designs. It is incredibly challenging to develop items that are culturally relevant for all students, especially when those leading test design and development are mostly members of the culturally dominant group. Designers of any kind have to make assumptions, and in the case of test design those assumptions may be limiting the relevance of the assessment for students from non-dominant cultures. Research has shown that culture has a significant effect in how test takers approach standardized tests and on test-taking performance [4]. This research creates an imperative for our field to interrogate fully all of the possible explanations for group differences in test outcomes, and to evolve our assessment design practices to keep pace with the fields of learning and instruction that have shifted to more explicitly account for the role culture.

Where Do We Go From Here?

Luckily, as a field, we are increasingly recognizing the overdue need to amplify diverse voices. Many organizations and leaders are working to address disparities in access to the field and in the promotion and retention of diverse talent. The National Council on Measurement in Education has funded a number of special projects supporting students of color in measurement programs and highlighting diverse perspectives, including the work of Darius Taylor with The Wrong Answer Project. Jennifer Randall, associate professor at UMass Amherst, and Joseph Rios, assistant professor at the University of Minnesota have secured funding from the Chan Zuckerberg Initiative to evaluate recruitment and retention efforts for students of color in measurement doctoral programs. Additionally, late last year I had the privilege of launching a new nonprofit, Women in Measurement, with a group of dedicated colleagues who are passionate about advancing gender and racial equity in educational measurement leadership. Women in Measurement provides structures of support for career advancement through its programs, which include mentoring sessions, fellowship awards, speaking engagements, and networking events.

The structural racism and misogyny that have led to disproportionate access and representation are certainly not unique to educational measurement, but it is going to take the energy of all of us to address the persistent inequities within our corner of the world. Not only must we elevate the voices and perspectives of people of color, but we must all commit to raising the critical consciousness within our field so we can effectively grapple with the most pressing issues we face in educational measurement.


[1] Moss, P. A., Pullin, D., et al. The idea of testing: Psychometric and sociocultural perspectives. Measurement: Interdisciplinary Research and Perspectives 3, 2 (2005), 63–83.

[2] Randall, J. "Color-neutral" is not a thing: Redefining construct definition and representation through a justice-oriented critical antiracist lens. Educational Measurement: Issues and Practice. May 2021.

[3] Sireci, S. G. Standardization and UNDERSTANDardization in educational assessment. Educational Measurement: Issues and Practice 39, 3 (2020), 100–105.

[4] Arbuthnot, K. Reimagining assessments in the postpandemic era: Creating a blueprint for the future. Educational Measurement, Issues and Practice 39, 3 (2020), 97–99.


Susan Lyons, Ph.D. is the principal consultant for Lyons Assessment Consulting. Dr. Lyons works at the intersection of educational measurement and social justice. She partners with clients to provide thought leadership, design systems, lead research, and offer technical advice that leverages the power of assessment to create a more equitable future. Current clients include states, school districts, nonprofits, and testing companies. In addition to her consulting work, Dr. Lyons teaches at Boston College and is the executive director of a nonprofit organization aimed at advancing gender and racial equity in the field of educational measurement, Women in Measurement, Inc.

2021 Copyright held by the Owner/Author.
Publication rights licensed to ACM.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2021 ACM, Inc.


  • There are no comments at this time.