ACM Logo  An ACM Publication  |  CONTRIBUTE  |  FOLLOW    

12 Steps Toward Immersive Learning

By Dov Jacobson / April 2018

Print Email
Comments Instapaper

Learn by being there. It is this sense of presence that powers the promise of immersive learning. That promise-to enhance the acquisition of new knowledge and improve its far transfer-is supported by theory [1, 2]. Research into situated learning reveals the value of coherent frameworks that faclitate exploration and experimental problem solving [3]. Immersive learning embeds the learner in a such a playful framework.

Immersive learning employs technology to intercept the learner's senses with synthetic signals. The consistency of these signals persuade the perceptual system that it is embedded in the virtual environment. The learner is promoted from an observer of the syntheric world to one of its inhabitants.Immersion promises learners new access to experience. Learners can visit worlds that were once forbidden: worlds too remote, too costly, too dangerous, or too consequential. They can step right into a hypothetical. Learners arrive as lucid dreamers, awake to the illusion, and accepting it. Open-minded, they are ready to learn.

This promise isn't new. Renaissance painters achieved immersion by inventing vanishing point perspective. Audiophiles pursue immersion by piling up speakers. Although grainy, jumpy, and silent, early movies were stunningly immersive to their new audiences-people ran from the theater to escape the onscreen image of an oncoming train. Today, we seek to create immersion in the emergent medium of virtual reality (VR).

Different technologies offer different levels of VR. We can consider a spectrum that ranges from the classic stereoscopic viewer to a stadium-scale multiplayer VR experience. At each level, immersion is deeper, as more senses are more deeply engaged and the learner becomes more of a participant. Each level demands its own immersive learning strategy. We can best study this spectrum by deconstructing the individual features that each contribute to the sense of immersion.


One of the most immersive techniques employs the least technology. Whether it is the dark and hushed house at a Broadway play or the built-in blinders of a VR headset, we achieve a lot of immersion simply by masking out the competing signals of real reality [4]. Researchers have measured 36 distinct external classroom distractions [5]. Most of these can be eliminated by the VR headset, regardless of its content.

Stereo Vision

Deep insight always results from the aggregation of differing points of view. That abstract truth, applied literally, explains the principle of binocular vision. Each eye sees a slightly different view of the world, and the brain integrates these differences, yielding depth perception This phenomenon of binocular vision has been exploited since the Victorian era. In the early 1900s, audiences poured pennies into peepshow arcades. Tonight they will line up for tickets to 3-D movies [6].

VR headsets easily deliver excellent stereoscopy. Learning prorofessionals have explored the potential of stereoscopy for the past 100 years. In many domains, it adds only the power of novelty, When that is exhausted, it offers a modest increase in realism that rarely justifies the awkward apparatus required for its display.

However in certain learning domains, depth perception is critical to undertanding. In these domains stereoscopic viewing has been found effective, specifically in the fields of structural chemistry [7] and anatomy [8], especially when applied to the training of surgeons.

360 Vision

When an image contains everything that can be seen from a given point in space, the image is called 360 (also pano, photosphere, surround photo). The learner is at the center of the scene. As in real life, the viewer sees only one sector of the scene at any time, but can turn and tilt the view to see other sectors of the environment. Everyone who has used Google Street View is familiar with 360 vision [9].

Figure 1. A 360 photograph of the author in a VR headset. © 2018 GamesThatWork

360 images are particularly useful in the study of architecture, urban and interior space planning, as well as world culture and history [10]. In any study of the built environment, the traditional frame of a flat image introduces a gross distortion. Through evolution and design, these spaces offer 360 expeiences to their inhabitants. For example, congregants within a cathedral will not only gaze straight down its central axis toward the altar, they will also experience the peripheral rhythmic symmetries of columns and arcades flanking the nave in which they stood.


There are different ways to view a 360 image. When it is displayed on a screen, large or small, the viewer changes the angle of view with the motion of a mouse or fingertip. VR systems, however, introduce the new feature of head tracking. To select the angle of view, learners simply turn to face what they want to see.

This direct, natural control of the field of view enhances immersion. It is associated with headsets, but it can be achieved with a plain cellphone. As you turn your head, you hold the cellphone screen in front of your face-offering a moving window into the virtual world.

Head-tracking has been used to enhance learning in many fields. It is employed extensively by those learning to overcome phobias using techniques such as exposure therapy [11].

360 Video

The 360 image can be video. In this case, the surrounding world is one of moving images. The location of the point of view (the camera location) may also move [12]. While viewers generally cannot control the location of this camera, they can control its angle and tilt.

Recent work has shown the learning effectiveness of 360 video in journalism [13]. Much of the attraction of the new medium is its novelty, but certain subjects get genuine value from 360 video. These include complex and decentralized dynamic interiors like factories and markets, unfocused outdoor scenes such as crowds and cityscapes.

360 Audio

Immersion becomes more profound as more senses are involved. The auditory sense contributes powerfully to the sense of presence, particularly when sounds are located in space and therefore they change as the field of view turns [14]. Audio production is far less expensive than producing visual imagery, yet it can be even more compellingly immersive.

Sometimes known as holophonics or ambisonics, 360 audio can be used to cummunicate acutely first person experience. Current examples are sparse, but these indicate the range of utility by illustrating experiences as diverse as trench warfare and a haircut [15].

Synthetic Imagery

The virtual world becomes far more fluid when it is live, rather than pre-recorded. The viewer sees fresh computer-generated content, rather than static photographs. The view is rendered in realtime, as in a videogame. Now, viewers not only change the angle of view but can also shift their head location to see around objects in the foreground. This effect, parallax, is a key component of depth perception.

Real-time synthesized imagery offers a live world, which can respond to user action. It includes the hybrid technology of photogrammetry, in which the synthetic world is not an artist's model, but the result of scanning a real scene and its contents.

Figure 2. This Sketchfab model shows a highly stylized character, "Claire Blare." Cartoon worlds are generally more immersive than realistic worlds with similar visual fidelity. Visitors in cartoon VR see a self-consistent world, while in more realistic VR, the eye seeks out all the defects. © 2018 GamesThatWork


When the live virtual world includes a live view of the real world, this is mixed reality. This category includes augmented reality, where the system embeds virtual images into the live environment, either by intercepting the video feed of a camera or by reflecting synthesized images into the view seen directly through a visor.

It also includes augmented virtuality, in which the system embeds bits of the live world into its computer generated scene [16].

Figure 3. Paul Milgram, with this "virtuality continuum," offered an early map to the spectrum of virtuality [17]. © 2018 GamesThatWork
[click to enlarge]

In augmented reality, viewers control not only the angle of the camera, but also its location. A geolocated application, using GPS for example, is responsive to location all over the world. Pokémon GO provides a well-known example.

Figure 4. Augmented reality helps Air Force trainees learn by interacting with virtual anatomy superimposed on live standard patients. © 2018 GamesThatWork
[click to enlarge]

In some learning venues, particularly traditional classrooms, the full immersion of virtual reality is suspect. Administrators are loathe to allow learners to escape the classroom, even if they escape to a virtual learning environment. To these administrators augmented reality offers a welcome reduction of immersion. Students can enjoy stereoscopic visualization of interactive 3-D models without the isolation of a virtual reality headset.


High-precision location becomes possible when GPS satellites are replaced by local landmarks, such as the miniature scanning electronic beacons called lighthouses. Alternative "inside out" systems use machine vision to extract ad hoc landmarks from the environment. Virtual reality systems with one of these technologies allow learners to walk around, stand up or crouch to see new things. Free movement adds extraordinary verisimilitude to the virtual scene [18]. Walking around and exploring, rather than sitting and looking, makes the user far more present in the virtual space.

Different systems offer different volumes: HTC's mass-market Vive operates at "room scale" with outside-in lighthouse technology, while emerging inside-out systems work at warehouse scale or larger.

Walking in a tightly registered virtual space makes it feel rock solid. But systems that lack the requisite positioning technology still often attempt some sort of virtual locomotion. Users can move in the virtual world, but their real bodies go nowhere. The dissonance between visual cues of movement and the vestibular sense of stasis frequently results in nausea.

"Simulator sickness" has been a major obstacle to the broad adoption of virtual reality. It can result from many techncal sources-frame rate, latency, tracking fidelity, and horizon drift-but it is most often the result of a misguided design decision to compel a visual sense of locomotion that is unsupported by vestibular sensation.


A system that can precisely track the learner's head can (with more hardware) also track moving parts. In particular, it can track the hands. The virtual world becomes more immersive as the learner reaches out to touch it. Handheld controllers also deliver haptic signals to the hands, engaging more of the sensorium in the virtual world [19].

There is a rich history of scientists who advocate hands-on learning. These include seminal thinkers such as Seymour Papert and Maria Montessori, who promoted the practice of learning with her manipulatives. Montessori wrote "…first in play and then through work, the hands are the instruments of man's intelligence…" [20].


When the player touches the world, the world responds. Virtual worlds can respond to real actions-throwing an object, wielding a weapon, or opening a door. The player is not just present, but participating. The players acts and enjoys the consequences of action. A world that responds offers profound immersion and throws open the doors to exploratory learning.

The arguments for learning based on interactivity and simulation have been made for decades, and center on the phrase "learning by doing" [22]. Entire schools, conferences and careers are devoted to the study of this effect.

Figure 5. In "Kitchen Designer," players perform situated problem-solving in a virtual space. © 2018 GamesThatWork


It is not enough to be present. It is not enough to be interactive. A deeper immersion comes when the learner is motivated to achieve a particular goal in the virtual world. The learner is fully immersed when the world presents challenges that must be confronted to achieve an end. Immersive learning is acheived when these challenges lead players to explore the learning objectives.

The introduction of challenge into the leaning process forms the core of game-based learning [23]. Learning games can be produced many media from simple cards on a table top to a massive, military wargame exercise. Digital platforms for game based learning range from standalone hand-held devices to global networks of specialized hardware. But there is a unique synergy in the combination of learning games and virtual reality. Its adherents (among them the present author) would argue this combination represents the peak of each technology.

Figure 6. This Korean fan site shows "Brush Up VR." It is a vigorous, humorous action game in room scale virtual reality. Children learn about conscientious dental hygiene and tooth surfaces, while having fun. © 2018 GamesThatWork


Visitors to VR worlds can return with new skills and insights that will improve their experience of live reality. The challenge and privilege of this generation of learning designers is to discover how to employ these new componenets of technology and their associated psychological effects to create a new discipleine of immersive learning.


[1] Dede, C. Immersive interfaces for engagement and learning. Science 323, 5910 (2009), 66-9.

[2] Liestøl, G. Situated simulations between virtual reality and mobile augmented reality: Designing a narrative space. In Furht, B. (Ed.) Handbook of Augmented Reality. Springer, 2011, 309-319.

[3] Lave, J., and Wenger, E. Situated Learning: Legitimate Peripheral Participation. Cambridge University Press, Cambridge, 1990.

[4] Lin, J. W., Duh, H. B. L., Parker, D. E., Abi-Rached, H., and Furness, T. A. Effects of field of view on presence, enjoyment, memory, and simulator sickness in a virtual environment. In Proceedings IEEE Virtual Reality 2002. IEEE, Washington D.C., 2002, 164-171.

[5] Tesch, F., Coelho, D., and Drozdenko, R. We have met the enemy and he is us: Relative potencies of classroom distractions. Business Education Innovation Journal 3, 2 (2011).

[6] King, R. D. A brief history of stereoscopy. Wiley Interdisciplinary Reviews. Computational Statistics 5, 4 (2013), 334-340.

[7] Holford, D. G., and Kempa, R. F. The effectiveness of stereoscopic viewing in the learning of spatial relationships in structural chemistry. Journal of Research in Science Teaching 7, 3 (1970), 265-270.

[8] Trelease, R. B. From chalkboard, slides, and paper to e_learning: How computing technologies have transformed anatomical sciences education. Anatomical Sciences Education 9, 6 (2016), 583-602.

[9] Anguelov, D. et. al. Google street view: Capturing the world at street level. Computer 43, 6 (2010), 32-38.

[10] Cecchini, B. T. "The Battle of Atlanta" cyclorama (1885-1886) as narrative indicator of a national perspective on the Civil War. Doctoral dissertation, Rice University, 1998.

[11] Hoffman, H. G. Virtual-reality therapy. Scientific American 291, 2 (2004), 58-65.

[12] Neng, L. A., and Chambel, T. Get around 360 hypervideo. In Proceedings of the 14th International Academic MindTrek Conference: Envisioning Future Media Environments. ACM, New York, 2010, 119-122.

[13] The Daily 360. The New York Times. Accessed July 12, 2017.

[14] Grimshaw, M. Sound and immersion in the first-person shooter. International Journal of Intelligent Games and Simulation 5, 1 (2008), 119-124.

[15] Tudor, J. Five digital marketing trends for higher education in 2017. LinkedIn. Feb. 23, 2017.

[16] Ternier, S., Klemke, R., Kalz, M., Van Ulzen, P., and Specht, M. ARLearn: Augmented reality meets augmented virtuality. J. UCS 18, 15 (2012), 2143-2164.

[17] Milgram, P., and Kishino, A. F. Taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems E77-D, 12, (1994), 1321-1329.

[18] Kohn, J., and Rank, S. You're the camera!: Physical movements for transitioning between environments in VR. In Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology. ACM, New York, 2016.

[19] Stone, R. Haptic feedback: A brief history from telepresence to virtual reality. In Brewster S., Murray-Smith R. (Eds.) Haptic Human-Computer Interaction. Lecture Notes in Computer Science, vol. 2058. Springer, Berlin, Heidelberg, 2001.

[20] Montessori, M. The Absorbent Mind (Vol. 1). The Montessori Series. Montessori-Pierson Publishing Company.

[21] Roussou, M. The interplay between form, story, and history: The use of narrative in cultural and educational virtual reality. In Balet O., Subsol G., and Torguet P. (Eds.) Virtual Storytelling. Using Virtual Reality Technologies for Storytelling, ICVS 2001. Lecture Notes in Computer Science, vol 2197. Springer, Berlin, Heidelberg, 2001, 181-190.

[22] Aldrich, C. Learning by Doing: A Comprehensive Guide to Simulations, Computer Games, and Pedagogy in e-Learning and Other Educational Experience. Pfeiffer, San Francisco, 2005.

[23] Jacobson, D. Constructive failure. Serious Play Conference. 2015.

All illustrations in this article represent GamesThatWork projects.

About the Author

Dov Jacobson is a pioneer in game-based and immersive learning. He leads the Atlanta-based studio, GamesThatWork, to develop interactive mixed realities of virtual, mobile, social and web. They apply game science to help players acquire new knowledge, exercise new skills and form new habits. The player's achievements in-game confer real world benefits . These benefits include oral hygiene behaviors for children, disciplined thinking for U.S. intelligence analysts, design tactics for Boeing's aerospace engineers, market segmentation strategy for entrepreneurs, and more. This work has recently won top awards from authorities such as the National Academy of Medicine, the U.S. Education Department and the Entertainment Software Association. Dov is committed to articulate the principles that his team explores and share the progress they achieve.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Copyright is held by the Owner/Author. Publication rights licensed to ACM. 1535-394X/18/04-3152720 $15.00


  • There are no comments at this time.