An Alternative Way to Assess the ROI of e-Learning in Training
Part II

In Part I of this article, Lambe provided a brief background on the economics of e-learning. Here in Part II, he looks at some of the most common ways that organizations deploy e-learning to support their strategic objectives, and shows how measurement of viability and impact can be approached within those situations.

The article is adapted from Patti Shank's and my book, The E-Learning Handbook, Past Promises, Present Challenges (Pfeiffer, 2008). —Saul Carliner, PhD, CTDP, Associate Professor in the Graduate Program in Educational Technology, Concordia University

Measuring the Viability and Impact of e-Learning
How might one assess the ROI of an e-learning infrastructure that support a variety of different business objectives?

Here, in Part II of this article, I consider how to assess ROI for e-learning with these six major business objectives in mind:

1. improving productivity
2. improving quality
3. leveraging human capital
4. reducing risk
5. remaining in the marketplace
6. accessing new markets.

In each section, I describe examples of e-learning, measurement questions to consider, and stakeholders who might be interested in these results.

Figure 1 shows a process for approaching the types of issues described in these upcoming sections.

Figure 1. A decision tree can be used to assess the business impact of investments in e-learning.

1. Improving Productivity
When e-learning is intended to support the improvement of productivity, it is ultimately aimed, in economic terms, at ensuring that the output per time or dollar spent is the same or greater than before the e-learning program was introduced. At the least, an effective investment should result in an equivalent level and quality of output. At most, one or both are improved.

For the most part, e-learning in terms of productivity improvement is based on the productivity of the learning staff—not the learners.

Examples. Examples of e-learning in support of productivity improvement include: 1) more people have access to learning for the same budget outlay or less, and 2) the sales force of an organization takes product training by e-learning, thus reducing time spent in classroom and out of the field, away from customers.

Stakeholders. The stakeholders in such an effort are both business owners of product or service being addressed by the learning and participants in the learning program, who are evaluated and rewarded on productivity targets.

Leverage points. Points to leverage such an e-learning infrastructure include standard productivity and performance reporting systems; performance bonuses and commissions; and existing intranets and mobile computing infrastructures.

Questions for measurement. Measurement questions include:

• How is ROI calculated?
• To what degree do stakeholders attribute any productivity gains to the e-learning (pre and post initiative)?
• What alternatives to e-learning exist as an aid to productivity improvement, and how much would they cost (pre and post initiative)?
• Have there been any quality or effectiveness gains or losses as a result of deploying e-learning? If yes, can these be quantified? How?

Improving Quality
When e-learning is intended to support a strategic quality improvement initiative, it is integrated into the workflow and through performance support systems at the specific points in a business process that need improvements in speed, accuracy or consistency, or where vulnerabilities in quality can be mitigated by providing support through e-learning support. This e-learning support can help workers perform more effectively, or can be aimed at customers to help them effectively buy and use complex products and services.

Examples. One example of e-learning in support of quality improvement is a medical clinic that provides healthcare workers with mini-modules about the diagnosis and treatment of a specific conditions, based on symptoms, patient history, and medications. Another example is a vendor of enterprise resource planning software, which provides e-learning on its public site to help small and medium enterprises to understand how its products can support their businesses, and to help them make purchasing decisions appropriate to their needs.

Stakeholders. The stakeholders in such an effort are: business owners of the product or service being addressed; quality audit and control departments, which have overall responsibility for quality in businesses; business process improvement teams, which have similar responsibility; and customers and potential customers of complex products and services, who would be users of the service.

Leverage points. Points to leverage such an e-learning infrastructure include:

• quality audit and control systems
• standard operating procedures and standards such as ISO 9000
• business process reengineering and Six Sigma (a type of quality control) projects
• marketing and customer service processes and initiatives
• Internet website and sales force deployment
• product information and manuals.

Note that some of these leverage points are not necessarily in the technology itself, but comprise other parts of the ecological system that supports e-learning in the organization.

Questions for measurement. Measurement questions include:

• What improvements in speed, consistency, or accuracy does e-learning enable? By what amount and degree of seriousness are customer-facing errors (that is, errors that customers would notice) reduced?
• To what degree do stakeholders associate the quality of their performance or their purchasing decision (if customers) with their e-learning support?
• What alternatives to e-learning exist as an aid to quality improvement, and how much would they cost (pre and post initiative)?
• Have there been any productivity or sales impacts as a result of deploying e-learning, and can these be quantified? If so, how are they quantified?

Leveraging Human Capital
When e-learning is intended to help an organization leverage its human capital, it refers to the ability to attract, develop, and retain talent and experience in employees so that the organization realizes a competitive advantage.

In globally-dispersed organizations, e-learning tools can connect employees and executives through collaboration software, giving them access to common learning resources in support of their work. Some e-learning provides professional development opportunities that might not be otherwise available and is widely valued by employees, such as access to personal development courses like time management and effective negotiations, as well as access to academic courses online.

In other instances, e-learning supports leadership and management development programs. Such programs are usually blended and include face-to-face components for establishing and strengthening relationships across the organization but e-learning components can extend the impact of these sessions and sustain collaborative learning across distances and time.

Leveraging human capital is particularly important in knowledge-intensive industries, especially those in which much of the knowledge is tacit and lies within the minds of the workers.

Examples. Examples of e-learning in support of leveraging human capital include a knowledge management project in a large multinational organization that captures the more challenging leadership experiences of senior retiring executives, packages them in modules as business simulations, and uses them as the basis for online leadership development projects. Another example is a series of modules on common engineering standards that a global engineering firm asked its experts to prepare and make available on its global intranet as a support tool for its project teams that are distributed throughout the world and work virtually. These modules are included in a collaborative project space so that engineers in different countries can discuss their application in different contexts, and each project space has access to the company's subject matter experts.

Stakeholders. The stakeholders in such an effort are human resource directors and senior management team, executives and managers, functional specialists and subject matter experts, and knowledge managers, all of whom play multiple roles, sometimes commissioning pieces of the system, sometimes using the system.

Leverage points. Points to leverage such an e-learning infrastructure include: knowledge management projects, particularly knowledge audits, expertise directories and intellectual capital measurement systems; leadership and management development programs; global intranets; collaborative technologies and tools; virtual team working and project management processes; recruitment, induction, performance review and promotion processes.

As in the last section, note that some of these leverage points are not necessarily in the technology itself, but comprise other parts of the ecological system that supports e-learning in the organization.

Questions for measurement. Measurement questions include:

• To what degree is the reuse of expertise and tacit knowledge of an organization's human capital enabled through e-learning?
• What economic or competitive advantage is reaped by the reuse of expertise through the e-learning system?
• To what degree do employees attribute their personal development and learning to the support provided by e-learning?
• To what degree do stakeholders associate the re-application of knowledge with their e-learning support?
• What alternatives to e-learning exist as an aid to human capital development, and how much would they cost (pre- and post-initiative)?

Risk Reduction
When e-learning is intended to support the reduction of risks throughout an organization, it's used to avoid or mitigate risks in volatile and uncertain business environments, in markets that are subject to rapid deregulation, in capital and infrastructure-intensive industries such as pharmaceuticals, and in industries where the pace of innovation is high.

Because the requirement for authoritative and coordinated knowledge of the inherent risks in the environment is high, e-learning can help by allowing the enterprise to disseminate up-to-date knowledge of the environment and corporate policies that respond to these environmental conditions quickly, widely, and in a way that is customized to particular job roles.

Examples. An example of e-learning in support of risk reduction is an insurance company updating its product training materials on the fly as regulatory changes take effect, and as competition with banks for the same market segments intensifies. The company can use learner records on its LMS to demonstrate that all employees have been trained in compliance with regulatory requirements.

A second example is a pharmaceutical company requires its research and development teams to compile e-learning modules summarizing current research projects and their progress to share across the entire company. This creates greater visibility for risk and success factors in projects, subjects projects to greater professional scrutiny, and fosters cross-fertilization of ideas.

Stakeholders. The stakeholders in such an effort are legal and compliance teams, risk managers, business development teams, directors of research and development, investors, and business owners of products and services addressed by the e-learning effort.

Leverage points. Points to leverage such an e-learning infrastructure include: legal and compliance processes; standard operating procedures and operating guidelines; ISO 9000 processes; training department processes; risk assessment and management processes; corporate intranets; content publishing capabilities; processes for funding and structuring research and development efforts.

Questions for measurement. Measurement questions include:

• What risks have been avoided or mitigated through the timely provision of knowledge via e-learning, and can their impact be quantified? If so, how?
• What disadvantage would have accrued from the absence of e-learning?
• To what degree do stakeholders attribute their avoidance or mitigation of risk to the support provided by e-learning?
• Are close competitors using e-learning to help manage their risks, and what impact does it appear to have on their competitive performance?
• What alternatives to e-learning exist as an aid to risk avoidance, and how much would they cost (pre- and post-initiative)?

Basic Qualification to Remain in the Marketplace
In some fast moving businesses with very rapid product lifecycles, such as telecommunications, the rapid availability of learning material is a basic qualification for competition.

For these businesses, e-learning is not a choice. It essentially provides a ticket to compete in a marketplace. Without e-learning as a way of keeping sales and service staff up to speed on short lifecycle products, there's no chance of staying in the market.

John Chambers of Cisco described it best. When asked if he had calculated the ROI of his e-learning drive, he is reported to have said, "Who calculates ROI on their telephone systems? If we don't have one, we can't do business. E-learning is fundamental to the way we operate."

Examples. Examples of e-learning as a basic qualification to remain in the marketplace include using e-learning to address fast evolving technology and software products which, when combined with large distributed sales forces, require speedy and frequent production and dissemination of product information in a form that learners can easily consume. Another example is using e-learning in call centers to gain advantages of speed, market share, accuracy and consistency, productivity and profitability.

Stakeholders. The stakeholders in such an effort are business owners of the product or service being addressed by the learning initiative, employees who cannot effectively function without easy access to information about products and service information, and executives of strategic planning.

Leverage points. Points to leverage such an e-learning infrastructure include: strategic and business planning cycles; intelligence about competitors; benchmarks of competitors; high reliability intranet and broadband access; content publishing capabilities; and performance support systems.

Questions for measurement. Measurement questions include:

• What is the competitive intensity of our market? How costly is it to compete in this market in relation to the potential profiles?
• Are competitors using e-learning to gain competitive advantage? How many competitors are and what advantage do they acquire?
• To what degree do stakeholders associate their effectiveness with their e-learning support?
• What alternatives to e-learning exist as an aid to functional effectiveness and how much would they cost (pre and post initiative)?
• Have there been any competitive gains or losses as a result of deploying e-learning, and can these be quantified? If so, how?

Accessing New Markets
When e-learning is intended to increase access to new markets, the existing infrastructure of the organization usually possesses sparse knowledge resources and experience in the new area of need, whether that area of need be knowledge about a new geographic area, a new product line, or a new vertical market.

In such situations, learning needs are pervasive across an organization, from design, procurement, and production, to marketing, sales, and service.

E-Learning can help by making available packaged information and knowledge content across the enterprise, presented in forms that are appropriate to the different functional roles.

Examples. One example of e-learning used to support of access to new markets is a consortium of small and medium enterprises in Scotland, who subscribe to a shared e-learning platform hosting resources, modules and case studies focused on acquiring new business in the European Union. A second example might be a large multinational organization that specializes in consumer products and provides a global management development program that includes e-learning modules on multi-cultural awareness, simulations and case studies on doing business in different countries, and online collaborative projects.

Stakeholders. The stakeholders in such an effort are business owners of product or service being addressed, country and regional managers, sales teams and sales managers, business development managers, and partners in any joint ventures.

Leverage points. Points to leverage such an e-learning infrastructure include: business development cycle; legal processes; global intranets; competitive intelligence processes and systems; strategy and business planning cycle; localization systems and processes; recruitment and overseas posting of employees; and new employee orientation, as well as ongoing training and development processes.

Once again, note that some of these leverage points are not necessarily in the technology itself, but comprise other parts of the ecological system that supports e-learning in the organization.

Questions for measurement. Measurement questions include:

• To what degree can the establishment of new offices or marketing campaigns be accelerated or made less costly through the use of e-learning?
• To what degree, and how quickly, is the experience gained by executives who pioneer the entry into a new market, recycled and reused by other executives?
• To what degree do stakeholders associate the speed and effectiveness of their entry into a new market with their e-learning support?
• What alternatives to e-learning exist as an aid to market entry, and how much would they cost (pre and post initiative)?
• Have there been any sales or market share impacts, or any avoidance of risk as a result of deploying e-learning, and can these be quantified? If so, how?

Using these Approaches to ROI
Creating a more discriminating set of tools for measuring the economic value impact of e-learning will help executives warrant their investments with greater confidence and guide their initiatives with greater precision.

The ground covered in Parts I and II of this article maps out the ways in which more closely linking e-learning to business strategy—combined with clear performance and impact targets, recognition of key infrastructure opportunities, and a system for engaging and soliciting measurement data from stakeholders—can cumulatively enhance its economic viability.

Acknowledgment
The author acknowledges the valuable editorial contribution made by Saul Carliner in preparing this article for publication.