ACM Logo  An ACM Publication  |  CONTRIBUTE  |  FOLLOW    

Remote proctoring? Step up your game and put the learner first

Special Issue: Advancing Beyond Multiple Choice eAssessment

By Geoff Chapman / September 2021

TYPE: MANAGEMENT
Print Email
Comments Instapaper

Delivery of proctored exams on-screen has existed for more than 15 years. The pandemic has pushed the technology and its radical service encounter into the mainstream. But new visibility brings new responsibilities and issues to resolve. A gold rush of new and existing suppliers means many learners now have the new exam experience they needed while others have had challenging experiences. This article considers the remote proctoring market, the growth drivers, and how the vendors and exam owners need to improve, through an exploration of contemporary service delivery, such as suggested by Professor Clayton Christensen's Jobs To Be Done theory, and modern digital service encounters expected by learners.

The disconnect between analog invigilation of exams and digital proctoring has disrupted a traditional, socially normed service encounter. The gold rush for remote proctoring has created a double whammy: service delivery concerns exacerbated by digitization, and how quickly digital services evolve of themselves.

There are at least 160 high-stakes exam software and services companies, inclusive of remote proctoring suppliers, generating $8.4 billion in revenue [1]. Half of them have a remote proctoring offer. The U.S. is by far the largest origin country, followed by the U.K., India, and the Netherlands. Big volume, high-stakes, substantial ed-tech expertise. But how did we get here, what's the pandemic's impact, and how should the experience be better?

Grappling and Worrying

Some traditional brick-and-mortar test center businesses have acquired remote proctoring platforms. They've grappled with shifting analogue exam delivery resources while worrying if remote proctoring will cannibalize their test center business. Long leases on expensive prime real estate can be difficult to renegotiate or dispose, and displaced staff can't always be retrained successfully.

Pure-play remote proctoring companies have fewer legacy issues. Their hockey-stick growth shows test volumes akin to the brick-and-mortar companies. Would they have reached the low-to-mid seven-figure test volumes without the pandemic? Definitely not.

But the pandemic has changed the rules of the game. The ed-tech sector is firmly on the global finance radar. The dramatic lowering of the entry barriers for delivery of high-stakes assessment means it's never been easier to get started.

An Outsourcing Bonanza

Where invigilation and proctoring were done internally (in-person) by university faculty, private training providers or delivery specialists, COVID-19 catalyzed an outsourcing bonanza—new opportunities swept up by a vendor market hitting pay dirt. Not a pivot from test centers, but an outsourcing story.

Jobs To Be Done

While the sector works out the post-COVID-19 landscape, many exam owners are trying to understand learner behaviors. They ask questions such as, "Will we see a full return to test centers?" "Are we keeping and extending remote proctoring?" "Do we actually need summative assessment?"

The late Professor Clayton Christensen's Jobs To Be Done theory steers the discussion away from product features and benefits [2]. The theory directly asks of users: "What job are you trying to get done that causes you to hire this service?"

Christensen's work for McDonald's restaurants involved not asking customers what flavors or which cup size they wanted. He instead asked, "What job are you hiring this milkshake for?" Customer responses would range from, "It keeps my stomach full 'til lunch," "It's easier to hold while driving," "It lasts longer than an apple," to, "I don't want the guilt trip of having a Snickers bar."

They want it easy, they want it to last—quicken the morning drive-through line, thicken the shakes.

So, instead of vendors asking test sponsors, "Do your candidates need fully accessible lockers?" "Would they prefer two workstation monitors for their accountancy exam?" ask them, "What job are you hiring remote proctoring for?"

With those answers, solutions need to make that job as effortless and friction-free as possible.

Reduce Friction—Have a Lemonade

The idea of reducing friction in the customer experience, or "snagging" if you're a Brit, is witnessed in many areas. From the insurance-tech world, Lemonade has a headline: "Sign up in 90 seconds, submit claims in three minutes, get paid in eight seconds." Lemonade identifies the user's demands and puts them, not what an insurance broker or a reinsurer thinks, first.

Nuggets of Data

Lemonade only takes data relevant and proportionate to the policy. A clean, pertinent and contemporary process. There's also a value-add: At year-end, the customer's unclaimed post-fee premiums are donated to a charity of the customer's choice. This incentivizes them not to submit fraudulent claims, as they'd essentially be cheating their own charity.

So, why does this matter? Learners haven't always been on-boarded successfully to remote proctoring—there's room for vendors and exam owners to better understand who their customers are, and what their service expectations are. Can a little guilt tripping help prevent malpractice?

Twenty Seconds of Fame

In the contemporary world of selfies and TikTok videos, Lemonade claimants are asked to shoot 20 seconds of phone video, describing in their own words what happened—giving them an outlet that they own. If something goes wrong in an exam, why not let the candidate upload a video and explain themselves?

Business leaders and education leaders often forget that service is now a two-way street—social media and globalization mean that candidate feedback can't be stymied, stonewalled or sat upon.

Also, Lemonade focuses on the first- and second-jobber market: the young people getting a foot on the property ladder, but also finding their career feet. Surely prime users of remote proctoring? These are the service encounters that learners now expect.

Defending Assessment Security in a Digital World

The evidence base for remote proctoring is rapidly emerging alongside that of exam security. Professor Phillip Dawson's new book, Defending Assessment Security in a Digital World, [3] looks at how learners cheat in different contexts, and the effectiveness of different approaches to prevent it. That evidence then informs the development of assessment security standards and metrics, as well as the impact of assessment design techniques.

Shenanigans: Cheating-as-a-Service

Professor Dawson claims there is a lack of independent evidence that supports the notion of remote proctoring actually stopping cheating. Buying cheating services is now in the mainstream. Just as we buy software-as-a-service, learners can buy cheating-as-a-service using cryptocurrency to get a stack of off-the-shelf solutions.

Respecting Well-Being and Privacy

If you're running a lengthy exam, learners will probably have comfort break concerns, medical conditions that need regular medication, or childcare commitments. Remote proctoring at home doesn't take place in hermetically sealed Faraday cages. Having a kangaroo court that shames the learner because they couldn't momentarily control their environment is not a long-term customer strategy. The privacy and well-being of children and minors at home, not just adult learners, needs to be acknowledged and catered for. The notions of equity, diversity, and adversity underpin much of Professor Dawson's contemporary and well-made arguments.

For example, instead of booking a day's absence of work to visit a test center, remote proctoring is currently viewed by learners as more akin to booking and attending a Zoom call—relating to their closest experience.

Learners and candidates are taking a ton more responsibility. Instead of show-up-and-sit down at a test center, they have to prep their rooms, read the rules, and have their behavior monitored, with a kickback in new complaints! This is genuine disruption.

Rapid, on-the-hoof, remote proctoring procurement has caused issues: learners have become upset, there are new pressures on stakeholders and regulation is playing catch-up. Good practices are sun-rising—they're not yet fully embedded or understood by the end users: it's also a heady brew for governance and regulators.

A Bumpy Switch

So, the pandemic has catalyzed a bumpy digital switch, but solutions to determine learner performance have to work for everyone. Candidate and stakeholder expectations are evolving quicker than some business leaders and education leaders can cope with or comprehend.

In the digital world, where Bebo and MySpace were once popular, customer expectations evolve inexorably. If a new service adds value and reduces friction and time, switching costs and efforts are now also lowered.

Is proctoring reaching its Blockbuster moment [4]? Will the test center go the way of the once-beloved video store? Learners want learning and assessment services online with genuine equity and universal access.

For a while, motor vehicles and horse-driven carriages co-existed, until the latter became the preserve of marginalized specialists and period-piece curiosities. Digital providers will (eventually) iron out remote proctoring's wrinkles. Are neutral, brick-and-mortar locations now riding off into the sunset—or harshly shuttered, just like Blockbuster?

With growing investment, user engagement and enhanced access, remote proctoring isn't going away.

References

[1] Private client research paper developed by World Exam Tech. 2020.

[2] Christensen, C., Hall, T., et al. Know your customers' 'Jobs to be done,' Harvard Business Review 94, 9 (2016), 54–62.

[3] Dawson, P. Defending Assessment Security in a Digital World: Preventing E-Cheating and Supporting Academic Integrity in Higher Education. Taylor & Francis, 2021.

[4] Chopra, S., Veeraiyan, M. Movie Rental Business: Blockbuster, Netflix, and Redbox. Kellogg School of Management Cases. Jan. 20, 2017.

Author

Geoff Chapman is founder of Acta Education, and cofounder of the World Exam Tech journal: immersed in computer-based testing and e-assessment within a three-decade career. He also consults to business owners and the ed-tech investment community, for new market entry, commercialization strategies, and investor readiness. A published author on e-assessment, Chapman's website offers free e-assessment resources, presentations, and blogs to help communicate good practice from around the world. He also tweets on e-assessment via Twitter @chappersman. Formerly vice chair of the e-Assessment Association, Geoff is also cofounder, adviser, and subject matter expert to a number of ed-tech start-ups. In October 2020, Chapman's management team sold the Mo University Assistant to UCAS, the U.K.'s university application provider.

©2021 ACM  $15.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2021 ACM, Inc.



Comments

  • There are no comments at this time.