ACM Logo  An ACM Publication  |  CONTRIBUTE  |  FOLLOW    

Closing the Assessment Excellence Gap: Why digital assessments should go beyond recall and be more inclusive
Advances in eAssessment (Special Series)

By Gavin Cooney, John Kleeman / January 2023

TYPE: OPINION
Print Email
Comments (3) Instapaper

Editor’s Note: This article has been adapted from the conference keynote at Beyond Multiple Choice 2021.

Stakeholders in digital assessments have requirements and expectations that technology implementations do not always meet. We call this gap between what society needs and what it often gets the "Assessment Excellence Gap." It is common to use digital assessments to measure recall of knowledge, but assessments that assess higher level skills are generally more useful. Additionally, when assessments are delivered with technology, it’s critically important to ensure it is done so in an inclusive way so everyone has a fair chance to show their skills.

Why Effective, Digital Assessment Matters

It’s worth reminding ourselves why assessment matters.

The Universal Declaration of Human Rights states education and choice of employment are human rights. Since digital assessment can act as a gate for life chances and these rights, it is only reasonable to hold it to a very high standard.

Few readers of this magazine will need to be persuaded that lifelong learning is crucial. This is how Nelson Mandela [1] put it: "Education is the great engine of personal development. It is through education that the daughter of a peasant can become a doctor, that the son of a mine worker can become the head of the mine; that a child of farm workers can become the president of a great nation."

Meanwhile in the world of work, there is a huge focus on skills. According to a Deloitte CEO survey in January 2022 [2], 71 percent of CEOs saw a labor/skills shortage as being one of the top three external issues likely to influence or disrupt their business strategy in the next 12 months. Businesses that want to improve their organization to deal with the challenges of technology, globalization, and/or the pandemic will see skill improvement and learning as a key part of their approach.

Few readers will also need to be reminded that assessment, when done well, is a critical part of learning. According to the recently published ISO 30422 international standard on learning and development in the workplace [3]: "Ongoing assessment of learners’ achievement against their identified learning needs is an important feature of the effective implementation of learning and development. Assessments motivate learners by giving accreditation or validation of learning. Assessments also inform learners and instructors what has been learned and retained and where to direct further learning."

Time spent learning doesn’t always lead to useful skills being learned. For instance, the 2018 World Bank Report "Learning to Realize Education’s Promise" based on detailed, worldwide evidence, advises "schooling is not the same as learning" [4]. The report suggests measuring learning quality by attendance is inadvisable, as attendance alone does not bring skills. When useful skills are acquired, assessment often plays a key part. The report recommends to achieve higher skill levels, we should assess learning better and more often and act on assessment evidence to guide learning practice.

There is a parallel in corporate learning and development. A challenge for Chief Learning Officers (CLOs) is to measure the effectiveness of training and learning interventions. Time and money are spent, but how does this impact the business positively? Digital assessments provide a widely used data point to help measure this (e.g., "level 1" post-course satisfaction surveys and "level 2" knowledge tests) [5].

Assessment Excellence Gap

Modern, digital assessments offer powerful benefits over paper assessments.  Assessment authors can collaborate across geographical distances and get immediate access to results data to improve questions and assessments. We can deliver assessments more quickly, more often, in a more personal and adaptive way and on devices that are close to hand for learners. And we can use technology to derive meaning and value from assessment results. And many learners prefer digital assessments to paper ones [6].

However, despite determined, professional, and focused work by the assessment community, there can be a mismatch between intent and outcome, when assessing online. Simply put, the effective digital implementation of an assessment design is not a given. The already demanding task of aligning learning objectives with curriculum and assessments can be further challenged by an organization’s digital capabilities and resources.   

One factor is that going digital increases user expectations. According to a World Economic Forum report [7], "... customers are forming their expectations of product/service quality around the speed, convenience and ease of use offered by their perceived leaders in customer experience (e.g., Uber, Apple and Amazon). These expectations now transcend traditional industry boundaries with customers expecting similar quality of experiences across the products and services they consume."

Assessment also has unique security needs (to prevent cheating) and can have pressing scalability challenges (thousands of people starting a digital test at the same time). There is a significant challenge in building a digital assessment experience that takes place at the right points in the learning journey, assesses the right things, adapts to get the best from learners, provides data-driven insights, is inclusive and accessible, and is robust, secure, and private.

In the context of the assessment experience, these are interrelated and interlocking factors. Without all these, the ability to truly deliver effective digital assessments is compromised. We use the term the Assessment Excellence Gap to describe the difference between what is required from a learning platform’s assessments and what is often delivered. There can be many aspects of this gap: security, privacy, scalability, robustness, consumer UX, accessibility, inclusivity, and device support are a few. 

There are two aspects of the Assessment Excellence Gap we will focus on:

  1. The first is too many assessments test for knowledge of facts when most real-world skills require a great deal more than just recall. The fact of the matter is recall facts are easily accessible on the fly due to Google. Times have changed and so must our mindset around assessment. This impacts the real-life validity of many assessments.
  2. The second is, given that education is a human right, digital assessments need to be inclusive, i.e., work well for all test-takers, not just most test-takers. Progress is being made but there is room for improvement.

Beyond Multiple Choice or Beyond Recall?

The theme of the conference this article arose from is "Beyond Multiple Choice," suggesting assessment needs to move beyond focusing primarily on multiple choice questions. Although this is a worthy goal, we’d suggest one that is more directly associated with improved validity is to go "Beyond Recall."

It’s easy to write questions that ask for recall of facts. And anecdotal evidence suggests very many assessments contain many such questions. If assessments are to be used to measure real-world skills, they must measure things that are relevant to those skills. For most job roles and other life skills, recalling facts is a useful part of the skill, but it’s not the most important.

For example, when we visit a doctor, we expect the doctor will know different parts of anatomy, but much more important is the ability to evaluate symptoms and make a judgment to suggest a course of action. It is not just doctors who need higher level skills such as judgment. According to the U.S. Department of Labor O*Net database of occupational information, there are 873 occupations that require judgment and decision making—including chefs, drivers and nannies. It’s hard to think of many jobs where recall of facts is the most important aspect of good performance.

A useful way to think about cognitive skills is the revised Bloom’s taxonomy [8], redrawn in the picture below. The lowest level of the taxonomy is remember/recall (also called memorize). Higher level skills include understanding, applying, and evaluating. There is a place for testing recall skills, but most assessments of learning or for job readiness should include questions that test higher level skills.


[click to enlarge]

We cannot put it better than assessment experts Coscarelli and Shrock [9]:

"In general, the single most useful improvement you can make in writing test items is to write them above the memorization level. … the vast majority of test items are written at the memorization level. In contrast, the vast majority of jobs require performance that is above the memorization level. This disconnect between testing practice and job performance is what often leads management to question the value of training and turns testing into a misleading indicator of performance, e.g., 'How come they passed the course but can’t do the job?' is a common summary of the problem."

Questions that ask for higher level skills can be well-written, multiple-choice questions [10], but, especially in digital assessments, it’s often easier to use question types that involve more than simply recognizing a choice—whether text entry, drag and drop, video, ranking or many others [11]. There is also much value in situational judgment questions, which present a dilemma that requires a choice to be made as to the most professional approach to resolve [12].

Almost every digital assessment system out there allows a wide variety of question types—let’s use them to test beyond recall. For most use cases, such assessments will be more valid, and the results will be more useful to make decisions within organizations.

Inclusivity

The second call to action we’d like to make is around inclusivity.

Education and career choice are human rights, and we need to find ways in which all test-takers can participate in the assessment process and show their skill. We sometimes think of assessments as a way to divide or classify people. But another approach is to consider digital assessments as a way for everyone to demonstrate rich skills and to gain feedback on how they can improve those skills.

All test-takers deserve a fair opportunity to demonstrate their skills in an assessment. As well as the obvious issues of being fair in relation to gender, race, and other demographic characteristics, here are seven key aspects of inclusivity for assessments.

  1. All assessments need to be accessible to people who have disabilities. This should not just be in a "tick the box" way but in a usable and effective manner that works for the variety of accommodations needed and in a practical way.  For example, software shouldn’t just support one type of Braille keyboard but allow the user to use their own. Learnosity has published a white paper "Building Accessible Learning Products," which is worth reading if interested in this area.
  2. Most assessments need to work on the majority of widely used devices. Within reason, test-takers shouldn’t be prevented from taking a test because they don’t have the right device, as that discriminates against those without resources. Assessments should use responsive design techniques to work well and fairly on all supported devices.
  3. Some assessment programs also need to consider how to provide access to people who don’t have connectivity to be able to access online digital assessments. Otherwise, they risk equity failures. Several approaches are possible including paper and offline delivery.
  4. Assessments also need to be fair to people who do not natively speak the language of the test. For example, based on U.S. census data [13], more than 20 percent of the U.S. population speaks a language other than English at home. There is an obvious risk of assessments being biased against such non-native speakers—potential approaches include reviewing language complexity, translating/adapting tests or providing more time or coaching.
  5. All assessments need to be designed to avoid cultural bias and stereotyping. Scenarios, abbreviations, and context need to work for all test takers, not just the majority demographic group, see for example the new U.K. Ofqual guidelines on accessible assessments [14]. Effective review mechanisms, guided by technology, for questions being designed are part of the solution to this.
  6. When remote or in-person proctoring is used to help secure an exam, it must be done in an inclusive way, respecting the dignity of the test-taker and any special needs. For example, some female test-takers in some cultural contexts need to be able to request a female proctor [15].
  7. One area that is not always considered is that some people suffer severely from test anxiety. There is evidence that some capable people have difficulties with formal tests, for a variety of reasons including anxiety [16].  We need to make assessments which all test-takers have a reasonable opportunity to learn for and pass.

Inclusivity is a moving target. Devices change, rules change, and organizations are becoming more global. Accessibility needs constant attention. Inclusivity is not an afterthought but needs building in carefully into assessment software. Inclusivity requires investment and focus, but also deserves it.

To quote a U.K. exam regulator on the recent introduction of new U.K. guidance on making exams accessible [17]:

"This isn’t about making exams and assessments easier, but about breaking down the barriers that stop young people achieving their true potential and making sure that exams actually test the things they are designed to test."

Conclusion

Learning and education is one of the most important things in the world.

It is important that excellent, digital assessment is widely available—not just in the best learning and assessment programs out there, but in all assessment programs. Every instructor, teacher, trainer out there who wants to create and deliver assessments should have the technology to realize their ambitions.

Let’s do it as best we can—seek to improve. And let’s seek to move beyond recall and make all our digital assessments inclusive.

References

[1] Mandela, N.R. Long Walk to Freedom. Abacus, London, 1994.

[2] Deloitte. Fall 2022 Fortune/Deloitte CEO Survey. 2022.

[3] International Organization for Standardization. ISO 30422:2022: Human resource management—Learning and development. ISO. 2022.

[4] World Bank. World Development Report 2018: Learning to Realize Education’s Promise.  World Bank, Washington D.C., 2018.

[5] Kirkpatrick, J. D. and W. K. Kirkpatrick. Kirkpatrick’s Four Levels of Training Evaluation. ATD Press, 2016.

[6] Butler-Henderson, K. and Crawford, J. A systematic review of online examinations: A pedagogical innovation for scalable authentication and integrity. Computers & Education 159, December (2020).

[7] World Economic Forum. Digital Transformation of Industries. Jan. 22, 2016.

[8] Armstrong, P. Bloom’s Taxonomy. Vanderbilt University Center for Teaching, 2010.

[9] Coscarelli, W. and Shrock, S. The Two Most Important Things You Can Do to Improve Testing in Your Organization. 2010.

[10] Azer, S. A. Assessment in a problem-based learning course: Twelve tips for constructing multiple choice questions that test students' cognitive skills. Biochem. Mol. Biol. Educ. 31, 6 (2003), 428-434.

[11] Burke, E. and Kleeman, J. Assessing for Situational Judgement. Questionmark. 2018.  

[12] Heffernan M. How to drive digital assessment beyond the MCQ test. Learnosity. Blog. 2017.

[13] Zeigler, K. and Camarota, S. 67.3 million in the United States spoke a foreign language at home in 2018. Center for Immigration Studies. Oct. 9, 2019.

[14] Ofqual. Ofqual Handbook: General Conditions of Recognition. GOV.UK. Oct. 12, 2017. Updated May 21, 2022.  

[15] O’Leary, A. Should a female test taker be able to insist on a female proctor? Questionmark. Blog. June 5, 2020.

[16] Buck, R. and Woods, K. Understanding test anxiety. The Ofqual blog. March 1, 2019.

[17] Ofqual. New Ofqual guidance on making assessments accessible for students. GOV.UK. May 12, 2022.

About the Authors

Gavin Cooney is CEO and co-founder of Learnosity. Cooney is a passionate advocate for driving improvements in the field of learning and assessment. Frustrated by the sluggish rate of innovation within assessment products, he came to understand that APIs had the potential to supercharge the whole industry. This vision has matured into a multinational tech company that powers the learning experience for more than 40 million learners globally.

John Kleeman is the founder of Questionmark. He wrote the first version of the Questionmark assessment system and founded the company in 1988. Questionmark recently became part of Learnosity, where Kleeman is EVP. He is a chartered engineer, Fellow of the British Computer Society, and a Certified Information Privacy Professional/Europe. He has contributed to many assessment standards initiatives and is a director of the Association of Test Publishers and was the 2021 Chairperson.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Copyright is held by the owner/author(s). Publication rights licensed to ACM. 1535-394X/2023/01-tk $15.00 https://doi.org/10.1145/3558395


Comments

  • Fri, 28 Apr 2023
    Post by New Edition Legacy Tour Jacket

    https://www.leatherjacketz.com/product/new-edition-legacy-tour-jacket/

  • Fri, 28 Apr 2023
    Post by David

    Black color is the most popular in the fashion world and this is the reason everyone keeps black color jackets. This year we also see New Edition Legacy Tour Jacket which is the best black color yet stylish jacket this year.

  • Fri, 28 Apr 2023
    Post by David Valentine

    This is a best one

ADDITIONAL READING

Advances in eAssessment (Special Series) This series of articles covers advancements in eAssessment. The series features educators, developers, and researchers from around the world who are innovating how learning is assessed while meeting the challenges of efficiency, scalability, usability, and accessibility.
  1. Going Beyond Multiple Choice
  2. Centering All Students in Their Assessment
  3. Harnessing the Power of Natural Language Processing to Mass Produce Test Items
  4. Getting Authoring Right—How to Innovate for Meaningful Improvement
  5. Closing the Assessment Excellence Gap—Why Digital Assessments Should go Beyond Recall and be More Inclusive