Helping Students with Learning Disabilities Through Video-Based, Universally Designed Assessment

“If assessments that purport to measure student learning are not ... universally designed, those assessments can pose barriers or obstacles for students with learning disabilities, obstacles that interfere with their ability to demonstrate what they have learned” [1]. K-12 students with learning disabilities (LD) can often go undiagnosed because they can’t be easily identified with intelligence tests used to assess students with intellectual disabilities [2], and they don’t have noticeable physical or sensory deficits. As a result, students with LD are often not identified for special education services until after attempts to provide these students with extra help in small group and individualized settings have been unsuccessful [3]. While students with LD are prone to struggling in school, these students with appropriate supports can achieve at levels similar to students without disabilities [4]. Some of the roots of these students’ struggles are related to difficulties with memory and processing, more specifically, with working memory, which is the processing, storing, and integration of multiple pieces of information [5].

Schools are increasingly using computer-based tests as a means for accommodating students with LD because of low-cost features such as read-aloud or text-to-speech functions, dictation, calculators, etc. [6]. “Utilizing text-to-speech with assessments can minimize singling out students with disabilities, diminish students’ embarrassment, and allow students the choice of rereading questions multiple times (Dolan and Hall, 2001).” [7]. Despite these advantages, recent research suggests students with LD aren’t more successful when using computer-based testing accommodations [6]. This may be due to lack of training on how to access these features or students’ lack of fluency with using the computer for the purposes of test-taking [6]. Other research has shown that a read-aloud feature through a computerized voice is not as effective as the use of a human voice [8]. In addition, read-aloud accommodations may only help with certain reading disabilities such as decoding issues but are not sufficient to help students who suffer from working memory issues [9].

Universal Design of Assessment Framework

In order to overcome the barriers that many students with LD face when testing, alternative means to assess students’ knowledge must be developed and the framework of universal design of assessment (UDA) may guide educators and test designers on how to do this. Underlying the concept of UDA is the theory of universal design for learning (UDL), which has its roots in the concept of universal design for architecture but is applied to the design of instruction in order “to improve and optimize teaching and learning for all people based on scientific insights into how humans learn” [10]. This framework takes individual differences into consideration by building in flexibility through three main instructional design principles, which are to “provide multiple means of engagement,” “provide multiple means of representation,” and “provide multiple means of action and expression” [11].

 The first approach to UDA is focused around the third principle of varying the ways students can express themselves. This is achieved by creating an open-ended assignment that can be completed in multiple ways. For example, students might be asked to explain why it is important to control variables in a scientific experiment, and they can choose multiple means in which to demonstrate their knowledge. One student could write a report, while another could do a podcast, and a third could create a visual representation. With this approach, students can select from a wide range of relevant options to express themselves that avoids barriers they may face and best demonstrates their capabilities and strengths as learners.

However, this approach is not suitable when developing standardized instruments. Rather than varying students’ options for expression, the alternative is to provide flexible representations or varying formats for the test material. Although the UDL framework proposed by Meyer et al. [11] provides guidelines for multiple means of representation, these are more appropriate for guiding the design of instruction as opposed to assessment. To supplement these overarching guidelines, more specific frameworks to UDA have been developed, such as one proposed by Johnstone, Thurlow, Moore, and Altman [12]. This UDA framework includes six principles for designing testing items, including (a) measuring what the test intends to measure, (b) respecting the diversity of the population, (c) providing clear text, (d) offering clear visuals, (e) including concise and readable text, and (f) allowing for format change [12]. Following each of these principles can avoid unnecessary barriers to assessment. The next sections introduce ways to incorporate universal design principles to assessments through an example assessment on scientific and proportional reasoning designed for middle school students. Although the examples provided were designed for young learners, the UDA principles can also be applied to adult learners.

Measuring What the Test Intends to Measure

One of the key considerations in creating an assessment that follows the UDA framework is to review that the test measures what it is intended to measure and minimizes the need for knowledge and skills unrelated to the test [12]. One issue for students with LD is that assessments often present multiple pieces of information at one time in the form of written text. These assessments require students to process, store, and simultaneously integrate information. Without supports to alleviate difficulties with working memory, students with LD are likely to struggle with assessments even though they may grasp the content being assessed. These assessments may result in informing teachers more about students’ difficulties with working memory or reading difficulties than their knowledge of academic content [13]. The approach of helping students with LD offload information by temporarily storing in visuals [14] tends to alleviate some of the challenges these students face with working memory and allows them to have better opportunities to succeed on assessments [4].

Simple text-to-speech or computerized read-aloud functions are not sufficient for many students with LD who need additional support with memory and processing skills [9]. For students with LD, researchers have found gestures as simple as pointing to what the students need to focus on at a certain time can have a major impact on their chances of being able to process multiple components of information at one time, determine the next step, and combine the results from that step with what needs to happen next [15]. Students with LD can get overloaded with too much information at once and gesturing can help students keep track of important information and make key connections between different parts of the problem, such as word problem text and the answer choices or information in a chart [15].

 These strategies have been tested in psychology lab settings and, in some cases, with students in K-12 settings, and researchers have found success with using gestures [16]. To provide gesturing in an online test, our recommendation is to include a video of a person pointing to different parts of the problem as it is read. Figure 1 shows a video of a proportional reasoning problem about filling glasses from bottles. In this video, the person gestures by writing out the number of bottles as the problem is read aloud. This video also models the use of scrap paper to offload information from working memory. This best practice of using videos to provide support for students with LD reduces the need to hire individual readers for students and avoids singling out students who need this support.

Figure 1. Video of a proportional reasoning problem with gesturing.

[click to enlarge]

Respecting the Diversity of the Population

A second consideration is that the test items should be sensitive to students from different backgrounds and experiences, and not unfairly advantage or disadvantage individuals based on characteristics such as age, ethnicity, gender, disability, etc. [12]. Johnstone et al. recommended reviewing the test items with experts to ensure the test does not disadvantage specific populations of students [12]. For example, instead of assessing students’ understanding of control variables by using a problem involving pendulums, with which students are unlikely to have personal experience, the problem can be rewritten to be about playground swings, which is more relevant to students and experiences they are more likely to have had. A best practice based on this principle is for test designers to understand their target learners and contextualize problems so they can be better understood.

Providing Clear Text

Another factor to pay attention to when creating universally designed test items is ensuring the font, size, color, contrast, and blank spaces deliver maximum readability [17]. The W3C Web Accessibility Initiative (WAI) has developed guidelines and resources to help make online materials accessible to people with disabilities [17]. There are a number of free tools that can check web accessibility. For example, a free extension for Google Chrome called NoCoffee can be used to test how text will look for individuals with different visual impairments [18]. This tool provides a quick means to double-check that the text has the needed clarity.

Offering Clear Visuals

It is also important to offer visuals that can help students answer the question [4]. Johnstone et al. recommend figures have clearly defined elements, strong contrast, and clear labels [12]. For example, adding an arrow to an image of a ruler to indicate the precise measurement (see Figure 2) for the amount of liquid in a beaker can add much-needed clarity for students in solving the problem. The addition of an arrow functions much like a gesture in that it makes the information easier to process. Students can process things faster and easier with arrows, which has a direct impact on how well they are able to succeed regarding working memory (i.e., processing, storing, and integrating multiple sets of information). If students can process something quickly and easily, they have a much better chance to not lose any information they are trying to remember and connect it to new information successfully [19]. Adding labels to images is a simple practice for reducing barriers for students with LD.

Figure 2. Addition of an arrow to add clarity to the illustration on test..

[click to enlarge]

Including Concise and Readable Text

Another area for consideration when enhancing assessments to fit the UDA framework is to ensure the text is concise and readable. Johnstone et al. recommend using common vocabulary and sentence complexity that are age-appropriate, minimizing superfluous words, avoiding the use of idioms, defining technical terms, and including a clear question [12]. By making these small changes in the problem’s text, one can help ensure that the problem is better understood when read aloud.

Allowing for Format Change

In keeping with the UDL principle of providing multiple means of representation [11], it is important for the test to offer a variety of formats that don’t change the meaning of the question [12]. The universally-design test format gives students the choice to watch and listen to videos of the problems or review a text-based version. The videos also provide students with the option to pause, rewind, and replay the problem as many times as needed. In addition, the videos can be turned off for students who grasp the concepts quickly. See Figure 3 for an example of a universally designed video-based test question that allows for format change. 

Figure 3. Test item in a Universal Design of Assessment.

[click to enlarge]

In this example test items, students have the option to watch a video, examine the visual image without playing the video, or open a text-only version of the problem. Offering students the option to choose the format of the test can help reduce the barriers students face when test-taking, allowing them to demonstrate what they actually know about the topic.

Challenges

Universally designed assessments have many benefits, but can also present challenges—both with the design of the test as well as during implementation. Careful attention is needed for the design of test items that utilize different media elements because, if not designed well, they can actually exacerbate working memory issues due to additional inputs that can distract students’ attention [20]. For example, designers should ensure that material is segmented into small components and images are directly related to the content to avoid cognitive overload. [20]. In addition, it is important, when designing test items with videos, that these videos can be turned off for students who don’t need them. Students who do not struggle with working memory issues may actually do better when directing their own attention [20], and, as a result, may find the gestures in the video unnecessary, distracting, or even annoying. Finally, extensive testing should be used to confirm the reliability of test items when students have the flexibility to change the format of the questions [21].

Implementation of universally designed, video-based assessments can also present difficulties. First, schools must have the necessary technology available for students. Each student will require a device for taking the test along with headphones so the noise of the videos isn’t distracting to others. Second, both teachers and students need training on all the features of the universally-designed assessment in order to benefit from its flexibility [6, 7]. Third, it is important to give students scratch paper if the problems require students to work out solutions to the problems. It can increase cognitive overload if students have to store too much information in their working memory [20]. 

Future Directions

Universally designed, video-based assessments may offer a much-needed solution to assessing students with LD. Standard online test features, such as text-to-speech or computerized read aloud, are not sufficient in helping students with working memory issues fully demonstrate what they really know about a complex topic. Incorporating videos of a narrator reading the problem and including gestures may help address the needs of students who struggle with processing, storing, and integrating multiple pieces of information simultaneously. One area for further development is the incorporation of personalized testing that matches the accessibility features with the specific needs of an individual [22], such that the videos would only be available for students who need them and don’t need to be manually turned off. This would reduce the need to train students and teachers on how to tell whether or not the videos, as well as other accessible features, would be helpful or could actually interfere with students’ ability to demonstrate their understanding. In the near future, intelligent systems may be able to assess students’ needs and provide them with the accommodations needed on-demand and dynamically change to align with those needs.

References

[1] Christensen, L.L., Shyyan, V. Johnstone, C. Universal design considerations for technology-based, large-scale, next-generation assessments. Perspectives on Language and Literacy 40, 1 (2014), 23–31.

[2] American Association on Intellectual and Developmental Disabilities. Definition of intellectual disability. (October 12, 2019) http://aaidd.org/intellectual-disability/definition#.WvXUUYWcHIU

[3] Gresham, F, M. and Vellutino, F.R. What is the role of intelligence in the identification of specific LD? Issues and clarifications. Learning Disabilities Research and Practice 25 (2010), 194–206. DOI: https://doi.org/10.1111/j.1540-5826.2010.00317.x

[4] Marita, S. and Hord, C. Review of mathematics interventions for secondary students with LD. Learning Disability Quarterly 40 (2071), 29–40. DOI: https://doi.org/10.1177/0731948716657495

[5] Baddeley, A. Working memory and language: An overview. Journal of Communication Disorders 36 (2003), 189–208. DOI: https://doi.org/10.1016/S0021-9924(03)00019-4

[6] Gelbart, W. R. Students with learning disabilities and computer-based high-stakes testing 53, 5 (2018), 308–312. DOI: https://doi.org/10.1177/1053451216676796

[7] Bouck, E. C. Online assessments in the content areas: What are they good for? Journal of Special Education Technology 21, 2 (2006), 67–73.

[8] Flowers, C., Kim, D., Lewis, P. and Davis, V. A comparison of computer-based testing and pencil-and-paper testing for students with a read-aloud accommodation. Journal of Special Education Technology 26, 1 (2011), 1–12. DOI: https://doi.org/10.1177/016264341102600102

[9] Ketterlin-Geller, L.R., Yovanoff, P. and Tindal, G. Developing a new paradigm for conducting research on accommodations in mathematics testing. Exceptional Children 73 (2007), 331-347. DOI: https://doi.org/10.1177/001440290707300304

[10] CAST. Universal design for learning guidelines version 2.2. (October 12, 2019) http://udlguidelines.cast.org

[11] Meyer, A., Rose, D. H., and Gordon, D. Universal design for learning: Theory and practice. CAST Professional Publishing. Wakefield, MA, 2014.

[12] Johnstone, C., Thurlow, M., Moore. M., and Altman, J. Using systematic item selection methods to improve universal design of assessments. Policy directions. National Center on Educational Outcomes Technical Report 18, University of Minnesota, Minneapolis, MN (2006).

[13] Ketterlin-Geller, L.R. Knowing what all students know: Procedures for developing universal design for assessment. Journal of Technology, Learning, and Assessment 4, 2 (2005), 4–22. https://ejournals.bc.edu/ojs/index.php/jtla/article/view/1649

[14] Risko, E.F. and Dunn, T. L. Storing information in-the-world: Metacognition and cognitive offloading in a short-term memory task. Consciousness and Cognition 36 (2015), 61–74. DOI: https://doi.org/10.1016/j.concog.2015.05.014

[15] Hord, C., Marita, S., Walsh, J.B., Tomaro, T., Gordon, K. and Saldanha, R.L. Teacher and student use of gesture and access to secondary mathematics for students with learning disabilities: An exploratory study. Learning Disabilities: A Contemporary Journal 14 (2016), 189–206.

[16] Rasmussen, C., Stephan, Michelle., and Allen, K. Classroom mathematical practices and gesturing. The Journal of Mathematical Behavior 23, 3 (2004), 301–323. DOI: https://doi.org/10.1016/j.jmathb.2004.06.003

[17] W3C. 2018. Web content accessibility guidelines 2.1. (October 12, 2019) https://www.w3.org/WAI/standards-guidelines/wcag/

[18] Aaron Leventhal. NoCoffee – vision simulator. (February 9, 2013). Retrieved from https://accessgarage.wordpress.com/

[19] Keeler, M.L., and Swanson, H.L. Does strategy knowledge influence working memory in children with mathematical disabilities? Journal of Learning Disabilities 34 (2001), 418-434. DOI: https://doi.org/10.1177/002221940103400504

[20] Wiley, J., Sanchez, C. A., and Jaeger, A.J. The individual differences in working memory capacity principle in multimedia learning. In R. E. Mayer (Ed.), Cambridge handbooks in psychology. The Cambridge handbook of multimedia learning, Cambridge University Press, New York, NY, 2014. 598–619.

[21] Ketterlin-Geller, L.R. and Tindal, G. Embedded technology: Current and future practices for increasing accessibility for all students. Journal of special education technology 22, 4 (2007), 1–15.

[22] Russell, M. Recent advances in the accessibility of digitally delivered educational assessments. In S. N. Elliott, R.J. Kettler, P. A. Beddow and A. Kurz (Eds.), Handbook of Accessible Instruction and Testing Practices (2018), 247–262. Springer, Cham. 

Acknowledgements

This work is supported, in part, by NSF DRK-12 Grant #1417983.

About the Authors

Dr. Janet Mannheimer Zydney is an Associate Professor in Instructional Design and Technology and the Director of E-Learning for the School of Education at the University of Cincinnati. Dr. Zydney’s research is on the use of technology-based scaffolding in online environments, multimedia programs, and digital games to improve students’ problem solving and critical thinking. She teaches classes in design-based research, learning sciences and technology, universal design for learning, and teaching with technology.

Dr. Casey Hord is an Associate Professor in the Department of Special Education at the University of Cincinnati. His primary research interest is developing mathematics interventions for students with learning disabilities and students with mild intellectual disability. Other research interests include the role of visual representations and strategic questioning in mathematics teaching, the training of pre-service teachers to teach mathematics to students with mild disabilities, and the potential role of mathematics tutors for students with mild disabilities in urban, suburban, and rural settings.

Dr. Kathy Koenig is a Professor of Physics at the University of Cincinnati, with her research in physics education. She has extensive experience in the evaluation of pedagogies and curriculum that support student success in college-level science and math courses. In addition, since 2006, she has been funded by NSF and NIH to investigate the development and evaluation of curriculum that advances student scientific reasoning abilities. This work also involves the development of reasoning assessments for students in various age groups, including middle school through college.