Designing and developing e-learning projects
a three-tiered approach

The following is an excerpt from the forthcoming book The E-learning Handbook: A Comprehensive Guide to Online Learning (Pfeiffer, 2008), edited by Saul Carliner and Patti Shank.

One of the problems with Instructional System Design (ISD), the methodology that underlies most instructional design efforts, is that it takes a one-size-fits-all approach to projects. More specifically, as presented in texts and courses (and often, as approached in practice, taught), ISD assumes that all projects are new ones with equal levels of complexity.

In reality, a wide variety of instructional design projects exist. Some are new, some are revisions. Some revisions are simple ones, requiring either a brief update of information or a quick fix to something in error. Some revisions are more complex than new courses, more like historic preservations that are intended to bring a building up to code while preserving the crumbling exterior of the original structure. Some projects are intended for the classroom, some are intended for presentation online.

Of those ISD projects destined for presentation online, a similarly large variety exists. Some are intended for synchronous use (that is, instructor and students are online at the same time) and others for asynchronous use (that is, instructor and students are not online at the same time). Of the asynchronous materials, some are primarily instructional (such as tutorials), some are primarily informational (such as online references), and some are intended to actually perform work for learners (such as smart online forms and wizards). On the one hand, a broad process can probably address what's needed to analyze, design, and develop these materials. But the materials differ so much that no single process without variations can really encompass all of these.

To address the one-size-fits-all problem of ISD frameworks, e-learning entrepreneur Elliot Masie (2005) proposed that organizations classify e-learning projects into one of three tiers of effort and investment. Rather than follow a one-size-fits-all approach to ISD, the underlying idea is that, when organizations launch an instructional design project, such as an e-learning project, they would first identify its tier and use that information to determine how to approach it.

How would that process work? When starting a new instructional design project, the project manager would categorize it into one of three broad tiers—bronze, silver, or platinum—and tailor the work plan accordingly. The following sections provide details on how those plans would be tailored.

Bronze—The Most Basic E-Learning Projects

Bronze projects are the simplest in terms of their scope and, therefore, require the least instructional design effort. Masie (2005) suggested that bronze projects are ones that have a limited impact on the organization, a limited number of learners (often less than 100), or are determined for other reasons to require limited instructional design effort.

In terms of technical skills, a typical bronze project first uses rapid e-learning tools in which the slides from an existing presentation are tweaked for appearance and clarity, followed by the recording of an accompanying narration, and completed by adding both a brief multiple choice tests to quickly assess understanding and a satisfaction survey to assess learners' reaction to the content. In most cases, learning professionals become involved in bronze projects late, after the decision to produce materials has been made, after the deadline has been firmly set, and when little time is available for anything other than completing the program.

Typical bronze projects include:

• Revisions to existing courses that only require simple updates to the technical content rather than a wholesale redesign of content. Examples include a quick revision to a university course for which a new edition of the textbook is now available (but requires few changes other than replacing the words "third edition" with "fourth edition").
• Courses whose primary purpose is to document existing processes that are used by a limited number of employees, such as the manufacturing processes used in a single plant by one or two teams, and for whom access to subject matter experts is easy.
• Courses compiled from noteworthy, one-time presentations by subject matter experts that the organization wants to make available on a "just-in-case" basis—that is, just in case someone would like to view the presentation, but without the expectation that large numbers of learners would take them

Because most bronze projects are often a low priority for the groups that produce learning materials for the organization, bronze projects are often produced by the subject matter experts themselves. In fact, the emergence of rapid e-learning tools, like Lectora and Breeze, easy-to-use Learning Content Management Systems (LCMS) like Force10 and OutStart, and Course Management Systems like Blackboard and Moodle, which let people develop and update content with little training on the software, has facilitated the growth of bronze projects.

Bronze projects are often produced as second- and third-priority projects on an as-available basis (that is, as the person assigned has time), and with limited resources. Most likely, they are produced without much, if any, needs assessment. Although they might have objectives, the focus in terms of development is often "what content needs to be communicated?" rather than "what do learners need to do with the content?" Focusing on the content significantly speeds development, though it probably limits the likely effectiveness of the content. Bronze projects are often published with little or no formative evaluation.

Although instructional designers can take full responsibility for bronze projects, in most cases, they play a limited role. At the least, instructional designers provide some basic instruction for subject matter experts on how to produce effective learning content, then leave subject matter experts on their own to develop the materials. At the most, instructional designers prepare templates and other resources, which guide subject matter experts through a fill-in-the-form-like approach to preparing the learning resources and review the resulting content to make sure that it is reasonably clear and conforms to formatting requirements.

Because bronze projects are produced on the quick, with little or no involvement from instructional designers, many instructional designers look down on such projects. Indeed, the large growth of bronze projects that usurp instructional designers' jobs and de-skills the jobs of those remaining is of great concern to some. David Merrill and Brent Wilson (2006) discuss this concern.

But not all instructional designers look at these projects in this way. In some countries where professional labor costs are relatively low, production of bronze projects represents a significant business opportunity, and is one of the key reasons for the growth of outsourcing in e-learning.

In addition, bronze projects offer benefits beyond outsourcing. Because many bronze projects are low priority ones, they would not have been attempted when the majority of learning was presented in the classroom and technology-based instruction was assumed to require five to 22 times the effort of a classroom course. So another way of looking at bronze projects is that they help organizations to make training available that might not have been feasible in the past.

Indeed, in many cases, bronze projects are not intended to actually be used. They're like many of the reference books that people buy on impulse at a book sale-something that primarily is intended to sit on a shelf but that the reader knows is available should he or she ever want to read it. If there's a high likelihood that the content will never actually be used, why should an organization invest significant resources in developing it?

Silver—The Middle-of-the-Road Basic E-Learning Projects

The next tier of instructional design projects is silver. Silver projects are ones that have either a high impact on the organization or a high volume of learners (between 100 and 1000, but rarely both) and, as a result, require a more significant investment of resources than bronze projects.

In terms of technical skills, a typical silver project might involve some instructional design effort, including a limited needs assessment and the specification of well-articulated learning objectives, and the development of an attractive screen template and graphics. Instructional strategies, however, tend to be basic—silver projects often rely primarily on adaptations of the mastery model of learning adapted from Gagne's (1985) nine events of instruction or games that rely on templates from other sources, like "Jeopardy." Organizations often conduct verification tests to ensure that the programs are producing the expected results among learners, but typically forgo pilot testing with learners because of the time and expense involved. In most cases, the learning professionals become involved in silver projects after leaders have decided that training is needed but with sufficient time to develop the e-learning program.

Typical silver projects include:

• Technical training courses, especially software and systems training, like training for a new customer service system, or for compliance training (where the requirement is that students take a course).
• Conversion of existing classroom courses to online formats, which involve re-conceiving and redesigning content for a new medium but for which most needs assessment, objectives, and assessment are already complete and not re-visited.
• E-learning materials intended for the general public but for which funding is limited, such as a Web-based course on healthy eating that a public health department develops.

In the context of e-learning, because most silver projects involve some level of needs assessment, specification of learner outcomes, description of an evaluation strategy, and moderate to complex instructional design strategies, they typically require that instructional designers play a central role in preparing them. In addition, because most of these projects usually require a more attractive visual appearance than bronze projects and more complex programming, they typically require that the instructional designers not only have strong instructional design skills, a command of the subject matter, and an understanding of the impact of the content on the organization, but also strong production skills, such as skills in using Web authoring tools like Dreamweaver and Flash.

However, silver projects typically have significant limits on resources, either tight schedules or budgets, or strong involvement by subject matter experts whose understanding of instructional design is limited. As a result, instructional designers typically feel constrained in working on them. In some instances, they feel like they could have produced stronger work had they had more development time or resources (like funding for higher quality graphics or animation). In some other instances, instructional designers feel like existing standards and templates, which were put into place to ensure consistency among learning programs and to increase the productivity of instructional designers, limited the effectiveness of the resulting e-learning program. In still other instances, instructional designers feel like their subject matter experts failed to provide them with sufficient development time or dictated too much of the content or instructional strategy for the e-learning program to be as effective as it could have been.

The truth is, none of these issues are unique to e-learning. The same concerns have been raised about classroom training projects. Indeed, concerns about such issues have given rise to the movements within the field of educational technology to emphasize performance rather than learning, and within the field of workplace learning and development to emphasize a tighter alignment with sponsors and corporate goals. The belief is that, in doing so, learning projects might receive more resources and that subject matter experts might better trust instructional designers to produce learning programs, and interfere less in development.

Although bronze projects might, in the future, be the most prevalent e-learning projects, silver ones are typically the ones that will consume most instructional designers' time.

Platinum—The Most Extensive E-Learning Projects

The top tier of instructional design projects is platinum. Platinum projects are ones that have both a high impact on the organization and a high volume of learners (at the least, more than 1000 but often in the tens of thousands). As a result, these projects not only require a significant investment of resources, but the return for such an investment is often so high that it can be easily justified.

In terms of technical skills, platinum projects require the most extensive instructional design and production skills. Because platinum projects often affect large numbers of learners, sponsors have a vested interest in ensuring that the courses meet learners' needs and, therefore, invest extensively in needs assessment, specification of learning objectives, and development of evaluation instruments. Similarly, because platinum projects often address both a high-impact topic and a wide variety of learners, they typically require complex instructional design that captures learners' interests, appeals to a variety of interests and learning styles, and ensures that learning "sticks." In many cases, this involves the development of sophisticated animations, videos, and simulations to enhance the learning experience and immerse students in the application of the content. In many cases, too, this involves the development of a host of supplemental resources to support learners in applying the content, such as the development of job aids that accompany training courses and online tutors for academic courses.

Similarly, because of the large number of learners served by a platinum course and the high need for learners to master the content, organizations not only invest in verification testing but also up-front pilot testing. These pilot tests assess the likelihood that the learning content will achieve its objectives with the designated learners. If these objectives are not met, the tests are meant to specifically pinpoint what's not working so that it can be corrected before the learning materials are formally published.

In addition, because most platinum projects have such a high requirement for success, instructional designers typically become involved in these projects early, when they can not only contribute to the design of the program but, more fundamentally, to the definition of what the project actually entails.

Typical platinum projects include:

• High-volume, off-the-shelf e-learning courses, including courses that are typically required in many academic programs, like calculus, statistics, and Shakespeare; courses that are commonly taught in the workplace, like diversity, safety, and MS Office applications; or common commercial courses, such as courses to teach the alphabet to pre-school children.
• Courses that support high-priority, organization-wide initiatives, like a change management initiative, a major revamping of customer service processes, a new product launch, or a management initiative.
• Courses that address an issue that carries a high-liability cost, such as simulation of safety challenges in an airplane or safety procedures for dialysis staff to follow

Because most platinum projects involve extensive needs assessment, use of creative and inventive instructional strategies, and pilot testing, they typically require the most extensive design resources of the three tiers of projects, often requiring specialists in related fields. For example, in terms of needs assessment, many platinum projects benefit from the involvement of a performance consultant, an expert whose focus is broader than that of an instructional designer. Performance consultants might suggest that learning programs, alone, might not solve the underlying performance issue training and that a combination of interventions (that is, projects) are needed that work together to help sponsors achieve the intended goals of their project (Watkins 2007, Robinson & Robinson 1996). Similarly, because the e-learning programs produced as platinum projects often involve the use of animation, video, and other media elements, these projects typically require production specialists like directors and sound engineers. Indeed, one organization even hired illustrators for a major comic book franchise to illustrate its e-learning program (Driscoll & Carliner, 2005). To ensure the most reliable pilot tests, these projects require the specialized expertise of usability testers, who have the expertise to conduct usability tests, assess the results, and translate them into tangible "action items" for improvement.

As a result for this need for highly specialized expertise, which is often beyond the scope of in-house training groups in corporations and government agencies, faculty development groups in universities, and curriculum development groups in schools, many organizations hire "boutique" shops to handle platinum projects. This troubles some staff instructional designers because they lose the opportunity to work hands-on on some of the most choice assignments. Although these are technically outsourced, because firms are chosen for such projects based on their design expertise, they are not typically included in the discussion of outsourcing in the e-learning industry. Indeed, the cost of these firms is often high by industry standards because they bring unique expertise to the table.

Not surprisingly, many of the projects that receive recognition in the various competitions for outstanding e-learning programs could be categorized as platinum. Indeed, they typically rank as the most common type of project winning awards, even though they are the least common of all instructional design assignments.

Therein lies the paradox of platinum projects. On the one hand, because platinum projects most closely follow the instructional systems design processes described in textbooks and taught in university programs, and involve the most extensive allocations of resources, instructional designers typically believe them to be the ideal. Indeed, many representations of instructional systems design assume that the designer is working on a platinum project.

On the other hand, because they're rare, platinum projects are not typical of everyday instructional design projects. As noted, when platinum projects do arise, many organizations choose to hire an outside firm to develop them because they are not typical. So perhaps these projects are not necessarily the appropriate ones on which to base prescriptions of everyday instructional design practice—which is what we have done.

Implications of the Three Tiers of Projects to the Practice of Instructional Design

Consider that, besides being non-representative of everyday instructional design projects, using platinum projects as the ones on which to base prescriptions of everyday practice also has a more damaging affective effect: it sets practicing professionals up for disappointment. By emphasizing the process followed on the platinum project as the ideal, expectations among instructional designers—especially those with less experience—is that their projects should follow this process. When the projects inevitably do not, instructional designers feel disappointed. At the least, they feel that they should have done more on a project that they did. At the most, they feel that they have been less than competent on the job. The sad truth is that the project never warranted that much effort in the first place.

This idea of matching the work process to the scope of the project has particular implications for e-learning. The rise of rapid design tools and LCMSs has raised expectations that e-learning can be produced—as the name suggests—rapidly. Groups outside of course production groups do not realize that "rapid" merely applies to the process of producing the content, not to scoping it out or designing it. Conducting a full needs assessment and evaluation (processes that cannot be sped up much by rapid e-learning tools) are perceived to slow down release of courses.

In other instances, organizations can afford to invest the time, but courses that either have a short shelf life or a limited audience might not warrant the economic investment in sophisticated design, like certain types of simulation-based experiential activities and games.

Penultimately, in emphasizing analysis and evaluation, the platinum instructional design process also de-emphasizes development and production, even though experience suggests that these account for the overwhelming majority of effort (as much as 75 percent (Hackos 1994)) on most projects. Consider the implications of the 10-phase Dick and Carey model shown in Figure 1, which only allots two phases to development—development and revisions. One could argue that formative evaluation represents a third phase that is allotted to development. Every other phase refers to a form of planning (analysis, requirements, and design) or evaluation effort. This might have been sufficient for classroom instruction, in which the material is presented live and the instructor can make changes while a class is in session, but such an approach is not sufficient for the self-study and distance materials produced for e-learning. For example, because students have no access to instructors when using them, self-study materials must be understandable to all learners on the first "read through." (Even if learners have access to a help line or e-mail address where they request clarifications to the content, chances are learners will not receive an immediate reply as they would in the classroom).

Although the description of the Dick and Carey model suggests that more than one revision might occur, the visual version of the process does not make clear just how extensive the drafting and revision process really is, or suggest how much work it really is. To find out how extensive this process is, consider the number of steps in the publications process in Figure 10-1 (adapted from Hackos 1994). The publications process offers a good comparison because its goal is to produce a piece of usable documentation (Lasecke 2006), a product that is similar to a piece of effective self-study e-learning. In the publications process, seven of the 11 steps are focused on development and production. Indeed, Hackos suggests that, in a typical publications project, only 25 percent to 30 percent of a project is actually spent on analysis and design activities. In contrast, as much as 50 percent of a project might be spent writing a first draft. Research also suggests that most publications go through two rounds of review and revision—twice as much as advocated by the Dick and Carey representation of the model (Carliner, Qayyum, Sanchez-Lozano, & Macmillan, in preparation). In other words, ISD models like the Dick and Carey model provide a flawed view of the actual effort involved in producing learning materials, especially ones for e-learning.

In contrast, the three-tier approach to instructional design projects not only helps project managers develop realistic project plans, they also provide a more realistic view of the effort involved in an instructional design project—and acknowledge that the effort required for each project differs.

This paradox of advocating the methodology for platinum project to all instructional design efforts has always existed, but the lengthy development process advocated for e-learning has exasperated the problem, and caused instructional designers to slowly question the viability of existing approaches to ISD.

References

Carliner, S., Qayyum, A., Sanchez-Lozano, J.C., & Macmillan, S. (In preparation.) The value of technical communication: What technical communication managers track, what technical communication managers report.

Dick, W, & Carey, L. (1995.) Systematic Design of Instruction, (4thnd ed.). Burlington, MA: Addison-Wesley.

Driscoll, M. & Carliner, S. (2005.) Advanced Web-Based Training: Adapting Real World Strategies in Your Online Learning. San Francisco: Pfeiffer.

Gagne, R. M. (1985.) The Conditions of Learning and Theory of Instruction (4th ed). New York, NY: Holt, Rinehart, and Winston.

Hackos, J. T. (1994.) Managing Your Documentation Projects. New York: John Wiley & Sons.

Masie, E. (2005.) Keynote address to the Ottawa Centre for Research and Innovation. Ottawa, ON: September 14, 2005.

Merill, M. D. & Wilson, B. G. (2006.) The future of instructional design and technology. In Reiser, R.A. and Dempsey, J. V. (eds.) Trends and issues in instructional design and technology.(2nd edition). Upper Saddle River, NJ: Prentice-Hall.

Robinson, D. G. & Robinson, J. (1996.) Performance Consulting: Moving Beyond Training. San Francisco, CA: Berrett-Kohler.

Watkins, R. (2007.) Performance Systems Design. Amherst, MA: HRD Press.

Footnotes

DOI: http://doi.acm.org/10.1145/1370000.1361089

Figures

Figure 1. Comparison of the ISD and Publications Processes