ACM Logo  An ACM Publication  |  CONTRIBUTE  |  FOLLOW    

The basics of e-learning
an excerpt from handbook of human factors in web design

By Lisa Neal, Diane Miller / August 2005

Print Email
Comments (1) Instapaper

Introduction

Distance education can provide a richer and more engaging educational experience than is possible within the confines of the classroom. It requires creativity and innovation in the design and development of Internet-delivered materials, especially since materials may have to stand alone, and in the use of delivery technologies. The same issues of quality and effectiveness that exist in the classroom occur in distance education, often compounded by the delivery mechanism and lack of contact between students and teachers and between peers.

Distance education existed long before the Internet, but it has become more prevalent and has changed significantly through technological advances. All sectors, especially higher education, corporate training, and continuing and professional education, want to take advantage of Internet technologies to provide education, training, and collaboration capabilities to geographically dispersed populations to enhance educational experiences and increase enthusiasm for learning.

This chapter defines distance education and delves into many facets of the use of the Internet for distance education; discusses why distance education has become so popular; and describes how to plan, design, deliver, and evaluate a distance education course or program.

The Definition of Distance Education

Distance education is most typically defined as education that takes place independent of location, in contrast to education delivered solely in the classroom, and that may be independent of time as well. In fact, the definition is quite fuzzy and is understood and interpreted differently by different groups and in different contexts. In particular, the definition varies in the inclusion and role of teachers and peers. While traditional education places great emphasis on the role of the teacher, in distance education the teacher can be completely removed from the learning process. Traditional education also delineates a duration for a learning situation, with a start and end date, which is only the case with some distance education.

ASTD, an education-focused professional organization, de- fines distance education in part by how it is delivered, as an Distance education can be characterized as an "educational situation in which the instructor and students are separated by time, location, or both. Education or training courses are delivered to remote locations via synchronous or asynchronous means of instruction, including written correspondence, text, graphics, audio- and videotape, compact disk-read only memory (CD-ROM), online learning, audio- and videoconferencing, interactive television, and facsimile (FAX). Distance education does not preclude the use of the traditional classroom. The definition of distance education is broader than and entails the definition of e-learning." (Kaplan-Leiserson, n.d.)

There are many terms that are used interchangeably, or with subtle distinctions, including distance learning, online learning, Web-based learning, and e-learning. E-learning, as seen in the previous quote, typically refers to education delivered over the Internet while distance education can more broadly include videoconferencing and audioconferencing delivered over phone lines.

Commonly used terms delineate a subset of distance education. Synchronous e-learning encompasses same-time interaction independent of location, while asynchronous e-learning includes any situation where learners are dispersed in time and location. Most distance learning is actually blended learning, which refers to a mix of synchronous, asynchronous, and classroom.

One of the challenges in planning effective distance education is selecting the appropriate mix of synchronous, asynchronous, and classroom activities, as well as determining the role of teachers and peers. There is a considerable range in the sophistication, quality, cost, and development time of distance education curricula. At the same time, outside of the confines of the classroom, there is the opportunity to rethink and redefine learning activities, materials, and courses.

Distance Education Compared to Traditional Education

Traditional classroom education serves a valuable role for children and young adults, where there are many benefits to face-to-face interaction and where maturity and self-discipline are still developing. The classroom is often necessary for supervised practice or for team-building activities that could not readily take place without face-to-face contact. Distance education is different from education delivered in the classroom in many respects besides the lack of proximity. Education in the classroom is traditionally developed and delivered by one person. This teacher may or may not have extensive training in how to teach. In many universities, for example, a professor can be an excellent researcher but have little knowledge about how to plan and deliver a lecture. Distance education separates the development and delivery processes. Development and delivery may be done by different people, or many people may be involved in different capacities such as multimedia experts and technical support people. Even in the case of asynchronous courses, distance education may be delivered on a Web site without human intervention or assistance.

Many of the issues from the classroom still exist in distance education, occasionally amplified by delivery through technology. For instance, cheating has always been a problem for educators. Unless cameras or biometric devices are used, the opportunities to cheat are typically greater online. Other issues are similar; for example, a good teacher structures and teaches a small or large class differently, and the same is true for an online course. Arguably the biggest opportunity in moving away from the confines of the classroom is the ability to redefine the notion of a course: What type of interaction is most beneficial to students? What will help them learn the best? Educators grapple with how to rethink education. As John Maynard Keynes said, "The difficulty lies, not in the new ideas, but in escaping the old ones, which ramify, for those brought up as most of us have been, into every corner of our minds" (Keynes, 1997).

The History of Distance Education

Distance education started in the 1800s with a for-profit school developed by Sir Isaac Pitman for rural residents in Bath, England. Correspondence classes became an alternative for people needing education or training who were not able to attend or did not have access to a traditional program. There have been many notable instances of early distance education playing a major role in people's success; for example, in the 1920s Edwin Shoemaker took a correspondence course in drafting and co-developed the La-Z-Boy recliner, which started an entire industry.

Following correspondence courses delivered by mail, radio became the next delivery vehicle, and it is still commonly used in developing regions where access to the phone or Internet is limited. Instructional television became common in the 1950s and 1960s and, like radio, is still in use. While instructional television never achieved the success anticipated in those decades, arguably the most successful offshoot has been the very popular Sesame Street-type shows and the cable in the classroom programs for children. And certainly, television is still a delivery vehicle in the sense that educational videotapes are still being produced.

As computers became more widespread, computer-based training (CBT) distributed on CD-ROMs became common. Web-based delivery followed in the 1990s. John Chambers, chief executive officer (CEO) of Cisco, predicted in 1999, that "Education over the Internet is so big it is going to make e-mail look like a rounding error." This growth, while not as rapid as predicted, has been fueled by travel reductions, by the increased need for professional development in the workplace, and by perceived cost reductions.

Perspectives on Distance Education

For many years, popular media have created falsely optimistic or gloomy scenarios about distance education. Peter Drucker predicted that big university campuses will become relics. "Cyberuniversities will replace brick-and-mortar . . . . It is as large a change as when we first got the printed book" (Lenzner & Johnson, 1997). Other predictions have involved the demise of the classroom. However, distance education does not remove or eliminate the need for the classroom, in the same way that e-commerce does not eliminate stores, e-mail does not eliminate phones, and information technology (IT) does not eliminate paper. In epidemiology, after a critical mass is reached, a small change tips the system leading to large effect (Gladwell, 2000). In distance education, this tipping point seems to have been reached in the past fewyears. It is now commonly accepted that distance education is a viable alternative to classroom education and, despite spam advertising nonaccredited online programs, much of the d'class' impression left by distance education's origins in correspondence schools no longer lingers. At the same time, there is considerable debate about whether online courses are effective and whether online degrees are valued. To address the former case, in No Significant Difference (n.d.), hundreds of studies since 1928 argue that teaching through remote instruction is as good as traditional methods. To address the latter, there are distance education programs with excellent reputations, but, at the same time, there are many that offer a degree for a fee (Neal, 2003).

There are a number of factors that affect the adoption rates of an idea, practice, or object (Rogers, 1962): relative advantage, compatibility, complexity, trialability, and observability. Looking at each of these factors, it is clear that distance education offers advantages and challenges. Relative advantage asks how much better the innovation is. Distance education at its best can offer education that surpasses most classroom education and, even when not of the highest quality, can offer educational opportunities not otherwise possible. However, distance education at its worst is as bad as the worst classroom experience and, unfortunately, the worst of distance education includes unaccredited programs offering online degrees for a fee. Compatibility asks how consistent it is with what already exists. Much distance education is quite consistent with the classroom but it can go much further, for instance, with simulations and discussions. Complexity asks how difficult is it to understand or use, and the primary difficulty is the skills and technical knowledge that are beyond what is required in the classroom. Trialability looks at how well can it be tried on a limited basis, which is quite easy since many programs offer sample or trial courses. Lastly, observability asks how visible to others the results of using it are, where a well-designed online program should be as good as or better than the classroom equivalent.

The Popularity of Distance Education

Many factors have contributed to the popularity and growth of distance education. Some are organizational, such as the desire to reduce cost or increase reach, and others are societal, including the all-too-pervasive enamorment with technology—just because it can be done it should be done. Another is the desire to improve the quality of education, but there is little data to substantiate this to date (No Significant Difference, n.d.). Cost savings are one of the most common motivators for starting a distance education program. Cost can be saved by reducing learner-related expenses, such as transportation to school, or by reducing institutional expenses, such as those for classrooms or buildings. However, it can be expensive to set up technology, develop courses, and compensate faculty, so the costs may just shift.

Time savings is another often-stated reason for distance education's popularity. Certainly, for the learner, there can be a reduction in time to get to class. For asynchronous e-learning, there is greater flexibility to take a class at optimal times for the learner based on preferences or schedule constraints. While this potentially removes learner fatigue, when a student is in class based on the class schedule rather than personal preference, it can increase time bankruptcy, when what used to be leisure time is now spent on an online course. Time savings also crops up in the reduced time between identifying a need to learn something and finding and taking a course.

Increased accessibility is another factor contributing to the popularity of distance education. Accessibility includes providing more learning opportunities to diverse students independent of location, provided the students have technology skills, access, and support. This includes students with disabilities for whom an online course is easier to take than one offered in a classroom. When an online course includes peer interaction and discussion, all students can benefit from the increased diversity possible. Lastly, online courses can provide increased access to experts, for instance, when an acknowledged expert can easily offer a guest lecture to a course and is more willing to do so because no travel is needed.

Planning a Distance Education Program

Developing a small or large distance education program requires planning in order to elucidate and meet goals. The planning steps are to build a team; make sure the team is educated sufficiently about distance education best practices; and engage as a group in strategic planning, scenario building, and requirements analysis. Even if these steps are done in a cursory fashion, they help with team-building and with informing implementation so it is clearer what is likely to lead to success and what the stumbling blocks might be. The most common mistake of distance education initiatives is to start by evaluating or even selecting and purchasing technology; this should follow the phases of planning.

A planning team should consist of people responsible for decision making, budgeting, and managing and implementing the program, with representation from human resources, IT, and so on as appropriate for the organization. The facilitator may be an outsider, who offers a broader perspective from being involved in overseeing other programs, or an insider, who is knowledgeable about the organization and its politics. Sometimes a participatory approach, involving target students and faculty, is desirable, since their perspectives can be helpful. The team should be educated in distance education and should develop a shared vocabulary.

A strategic plan describes how to bridge the gap from the current to the desired state, and provides a well-structured, systematic view of a distance education initiative (Driscoll, 2001). The planning process is often driven by an external force, such as the expansion of an initiative or competitive pressures, but planning is beneficial in all situations. In the planning process, the team articulates and assesses the current state, looking at the organization's strengths, weaknesses, opportunities, and threats.

It selects a planning horizon, which is typically about 2 years, to implement the plan, but may select an additional, shorter horizon of perhaps 6 months to pilot a few initiatives in order to test the waters and get early feedback. The team defines the desired state it hopes to reach and describes this in mission and vision statements that relate to the institution's strategic direction. It determines the resources, constraints, and budget and develops a realistic schedule and project plan. If the distance education program is internal, it is valuable to determine the readiness of the organization (Readiness Assessment, n.d.).

Market research and a competitive assessment (Competitor Analysis, n.d.). help in identifying best of breed and understanding what has been successful for other institutions—or what has failed and why. They help shape the distance education program and determine the target learner population. A good example of this is the extensive assessment the Massachusetts Institute of Technology (MIT) did that led to OpenCourseWare (MIT Open- CourseWare, n.d.). In the analysis, they discovered that many for-profit online programs were failing to make a profit, which led to the idea of putting courses online for free (Shigeru Miyagawa, personal correspondence, 2003).

Not only have many of the programs they looked at since failed, but MIT ended up with more in grant money to fund their initiative than their most optimistic estimates of the revenue a for-profit program would generate. In addition, the publicity they received is invaluable. Once the target population has been defined, a needs assessment helps in understanding what is likely to work for the target population and can prevent failure based on erroneous assumptions. The very diversity among students that can enrich online learning also presents challenges with respect to designing courses that take into consideration the learner population.

Age, background knowledge, technical knowledge, and comfort and experience using technology are just a few student characteristics that can affect an individual's online learning experience. While no two learners are alike, in traditional classes variations among students are typically easier for an instructor to gauge. While differences in technology savvy may impact to some extent the range of tools a student might use to complete a task in a traditional course, students who are not comfortable with technology can feel intimidated and fall behind very quickly in the online world. These issues, compounded by cultural differences, potential language barriers, and potential wide variation with respect to geographic location (and the resulting time differences), can make defining and designing courses for a target population complex. Careful planning, however, can address many of these issues while meeting additional challenges that may not be so easily addressed in a traditional setting (e.g., leveraging technology to accommodate disabilities).

Scenario building is a facilitated, systematic process to define the experience of the various communities, learners, teachers, support staff, and other people impacted by or impacting a distance education program (Neal, 2000). It does not focus only on what happens when taking or teaching a course, but looks at everything that precedes, follows, and impacts what happens when taking or teaching a course and determines what is needed for members of these roles to be successful in their role. The process involves a number of steps: determining who will be involved in developing, delivering, supporting, administering, or taking courses, and prioritizing these to order the next phase.

Starting with the most important perspective, usually that of the student, the relevant demographics, characteristics, skills, needs, constraints, and motivators of members of that category are outlined. This may include computer and Internet literacy, work or study environment, and other time commitments. The group then determines what happens when a representative of that category, or persona, carries out his or her role, i.e., taking, teaching, or supporting a class. The result of the scenario-building process is a list, which is then prioritized, of the needs and requirements from that role's perspective for the distance learning program. Subsequent perspectives are usually easier to consider after the first, and the entire process leads to requirements analysis and planning for the program.

Media analysis follows scenario building, providing recommendations for the delivery of capabilities that meet the needs of the target learner population as well as the other constituents (IHETS, n.d.). These recommendations may include asynchronous technologies, synchronous technologies, and learning management, possibly integrated with existing educational and Internet capabilities. The media analysis task includes a detailed requirements analysis. Together with the strategic plan, this forms the basis for selecting the right capabilities to meet the vision and needs on a distance education initiative. The process of media analysis requires careful selection of the optimal technology and training mode out of the myriad available. It is not enough to simply incorporate the most recent technological advances into a course - this does not guarantee an optimal learning environment. Media analysis can be performed in prioritized steps, where any program is constrained by cost, deadlines, market, technology or bandwidth issues, and so on.

The first decision is whether face-to-face interaction is needed, and, if so, whether it should be for the entire course, to kick off or end the course, or intermittent. The reasons for having face-to-face interaction are for sensitive topics, developing relationships, or supervised practice. The second decision is what the desired level of supervisory involvement should be and in what capacity. Possible roles include instructor, facilitator, moderator, subject matter expert, tutor, mentor, or coach. A person in one of these roles can provide or reinforce learning, support students or facilitate interactions among students and can do so in an proactive manner or in a passive manner, for instance, by being available to answer questions. This is very different from the prominent role that a teacher plays in classroom education, and self-paced online courses may not provide anyone in a supervisory or teaching capacity.

The third decision is the amount and type of peer interaction. While the classroom is rarely a solitary place, some courses integrate more teamwork and collaboration than others. In an online course, the range is from a solitary, self-paced course to a highly collaborative and interactive course in which group projects and discussions play a major role. The advantages to projects and discussion are the support and context students provide for each other, as well as the enhanced learning that can result. The disadvantages, and one of the reasons that the self-paced style is sometimes preferred, is that the more interactive style requires technology to support collaboration as well as ongoing human guidance and support.

The fourth decision, course delivery, leads directly from the previous one: determining whether this will be a synchronous, asynchronous, self-paced, or blended solution based on how much of and what types of interaction will be supported, as well as scheduling and time zone issues. It is also at this point that it makes sense to determine if courses will have start and end dates or be open-ended and how content will be accessed— online, offline, or both. Some thought may go into what material students retain following a completion of a course—whether synchronous sessions are archived, for example, and students retain access to these sessions. The final step is to determine the course style and components. This includes how information is conveyed, for example, through lectures, discussion, and online materials; what the look and feel and interactivity these materials will have; the extent to which materials will be developed from scratch or repurposed from existing materials; the extent to which courses will be tailored or adapted to learning styles; and what students are expected to do, for example, projects, simulations, and tests, and how these will be conducted. While many aspects of course design may be determined by the nature of the courses, assignments and testing may be determined in part by the type of program, since a degree-granting program requires evidence upon which to assign grades.

Selection, Deployment, and Use of Technology to Support Distance Education

Technology makes distance education work. While initially many distance education initiatives tried to replicate the classroom experience online, Laurillard (1993) offers the perspective that it is better to identify the unique qualities of a technology and exploit those so that one is doing what could not otherwise be done rather than seeking to replicate existing practices. However, technology is only the delivery mechanism and selection process is to not just determine which features are needed and which technologies provide them, but what will be most comfortable for students to learn and use so that the focus is not on the technology itself. Bob Smith, chief operating of- ficer (COO) of Otis Elevator, said "When elevators are running really well, people do not notice them. Our objective is to go unnoticed" (Andersen, n.d.).

Some of the issues to consider in technology selection are the number of students and the need for scalability; any existing software, hardware, platform, or Internet access constraints; the cost to purchase, use, maintain, upgrade, and access a technology solution; security and reliability; and vendor preference, reputation, and the availability of training and support. A good decision is important since this can be a major investment. There are new players in the market as well as occasional mergers, buyouts, and failures, and switching costs can be substantial. There are a number of current standards initiatives, including shareable content object reference model (SCORM) and aviation industry computer-based training committee (AICC). These are important to know about in doing government work and for interoperability between components.

The design and usability of distance education technologies vary considerably, with some tools for similar purposes varying in the number of features and the usability of those features. Students should not be hindered in their learning of the subject matter by the need to spend large quantities of time learning to use, or troubleshooting the use of, a poorly designed technology. Technologies should be easy to use, provide adequate tutorials, and help to enable students to become familiar with using the products to support their learning tasks. Additionally, students should be offered technical support and practice sessions, especially for complex or novel technologies. The technology utilized should also be appropriate for the tasks at hand—that is, it should be useful. In selecting technologies for e-learning, one should determine whether the capabilities provided can support the required level of student-system, student-student, and student-instructor interaction while enabling students to carry out the required coursework. It is important to test tools to determine how features work and how the interface design impacts students' ability to learn. Because of the complexity of some tools, new models for teaching become necessary. For instance, when using many synchronous technologies, a teacher may need a support person to help with use of the technology and monitoring of student interactions.

Asynchronous Computer-based Training and Web-based Training

Computer-based training (CBT) and web-based training (WBT) refer to a course that is distributed on CD-ROM or over the Web for students to take as a self-paced asynchronous course. They are often derogatively called page-turners because a standard layout is used where the mouse can be positioned and the learner can click through screens, often faster than he or she is reading. The advantages of WBT over CBT are that online materials can be updated and hence distributed more easily. The potential exists, since a student is online, for interaction with an instructor and other students. The disadvantages of WBT over CBT are that a student needs Internet access, either to be connected while taking the class or to be uploading and downloading materials. This can be costly, especially for large multimedia files, and prohibitive if a student is traveling or access is expensive.

Many CBTs and WBTs are structured in a linear fashion, where a learner is expected to follow one path through the materials. Some offer more flexibility, so a learner can navigate based on interests or needs. In the most sophisticated, each student follows a path tailored to his or her needs based on testing and progress. Some track what a student does or looks at, requiring, say, 80% of all screens to be looked at for the student to be considered done. CBTs and WBTs have varying degrees of interaction or interactivity, which most often refers to the extent to which a learner is passive or is actively using, say, a simulation or needs to be mousing over text to receive information. While development of CBTs and WBTs can be expensive, the costs can be recouped through broad distribution.

Technologies for Delivering Asynchronous Courses

There are many technologies that can be effectively employed to deliver asynchronous e-learning. These include many collaborative tools that can be brought to distance education including e-mail and discussion forums. Although many learning management systems include the ability to upload and share documents, e-mail is often used for informal, behind the scenes exchanges of resources (e.g., draft versions of documents, Web links) in support of group work. E-mail, in its simplest use here, can provide a speedier version of the traditional correspondence course. More typically, it is used for teacher-student communication and student-student exchanges.

Discussion forums are included in much distance education and provide a mechanism for discussion on specific course topics as well as informal exchanges carried out asynchronously over time (e.g., days, weeks, months). More robust discussion forums might support the ability to attach documents or uniform resource locators (URLs), or send e-mail notifications when new posts are added. Threaded discussion forums are typically organized so that the exchange of messages and responses are grouped together and are easy to find. Common ways to group or sort postings are by date, title, author, group, or by specific topics defined by the instructor or other participants. Often, threaded discussions are expandable and collapsible to allow participants to manage the number of posts shown on their screen at once and to facilitate browsing groups of posts. Discussion forums can be extremely effective since there can be a great deal of sharing of perspectives and insights among students; the challenge is to encourage the right amount and type of participation so that learning is enhanced through discussions.

Many techniques can help, such as modeling appropriate discourse through an initial example dialogue posted at the onset of a course, highlighting interesting student postings, assigning student moderators to summarize a discussion thread, and teacher participation and feedback (Neal & Ingram, 2001). Often, an instructor or a designee will seed the discussion by posing a thought-provoking question or statement to spark dialogue. In academic settings, students are often motivated to participate in discussions when a portion of the course grade is awarded for participation (i.e., frequent, scholarly discussion posts) or when the discussion topics address real-world issues or concerns.

Technologies for Delivering Synchronous Courses

Similarly to asynchronous courses, there are many collaborative technologies that can be effectively used to fully or partially support synchronous distance education. Most of these have the advantage that they need not only be used for real-time interaction, but can also be archived for subsequent review. These include audioconferencing, electronic whiteboards and screen sharing, instant messaging, text chat, virtual worlds, video communication, and Web casting and Web conferencing.

Audioconferencing, using the telephone or voice over Internet protocol (VoIP), allows a group to interact in real time through sharing voice (audio) and other artifacts such as slides or text. In its simplest form, this can be accomplished using phone lines and previously downloaded meeting materials. Audio is a fairly simple and often inexpensive way of supporting lecture and discussion in a course. The biggest issue with effective audioconferencing is quality, since students are generally intolerant of poor quality audio. For telephony, speakerphones with mute capabilities aid participation, and, for VoIP, headsets with microphones similarly make it easier for a student to participate. Sessions conducted with audio should be 1 to 2 hours in length, since it is harder for a student to sit through even a great lecture and discussion online than in the classroom. It is hard to replicate the classroom experience online, even with high quality audio, and it is generally most effective to make material that would otherwise be in a lecture available as prework for class and then put a heavier emphasis on discussion during a session. This also helps keep students accountable for their work outside of class. Audio sessions, just like the classroom, vary depending on class size, and what works for a small group is unlikely to work for a large group. Since people cannot see one another, simple protocols smooth interaction, such as prefacing a remark with one's name.

Audio works best when supplemented by other technologies to support information sharing and exchange. Whiteboard tools typically consist of an electronic version of a dry-erase board which can be used by a group of people in a virtual classroom, either used alone or overlaid on a shared application. They are used for freehand writing and drawing in the former case and pointing to or highlighting information in the latter case. These range from very simplistic shared graphical editors to fairly sophisticated shared applications incorporating audio, slideshows, or applications through remote desktops. Some whiteboard tools may support graphing, polling, group Web browsing, and instructor moderation. Screen and application sharing allow a teacher to share an open application on his or her desktop with a class. Many Web casting tools bundle application sharing, whiteboard, chat, a participant list, polling, and feedback indicators such as hand-raising with audio and video capabilities to provide a virtual classroom.

Probably the most frequently used form of synchronous interaction occurs via Instant Messaging (IM) and text chat. These tools provide the ability for synchronous conversations between people over the Internet by exchanging text messages back and forth at virtually the same time. IM typically involves pairs of individuals, while chat tools involve larger groups (sometimes whole classes). IM and chat provide a means for a teacher to hold online office hours. IM allows students to see when a teacher is available for questions and is quicker and easier than the phone or e-mail. It allows students to easily find when another is available, say, for collaboration on a project. It has the benefit that students can see who else is online, even if they do not communicate, which is helpful since online students may feel isolated. "One of the surprising properties of computing is that it is a social activity. Where I work, the most frequently run computer network program is the one called 'Where' or 'Finger' that finds other people who are logged onto the computer network" (Kiesler, 1986). IM and chat tools can vary widely—some are simple, allowing the exchange of text messages with little else; others provide an ability for private messaging, ignoring specific participants, sharing files or URLs, or providing some structure for interaction (i.e., for students to ask questions and instructors to provide answers). Chats used within learning environments should be persistent (i.e., chat histories should remain available for review by students and instructors throughout the life of the course). Instructors should be able view chat logs (time-stamped) for student assessment.

Virtual worlds take IM and chat into a visual realm, where avatars, or representations of people, move in a two- or three dimensional world and talk to each other. Virtual worlds have been used successfully for teaching languages and architectural design, where, in the latter case, students can construct buildings that others can tour. Virtual worlds are also useful for the informal social interaction that happens more readily on campus (Neal, 1997).

Videoconferencing extends the capability of audioconferencing by including video. Such services enable instructors to either stream video from within the system or else enable videoconferencing, between instructors and students, between students, or between multiple classrooms. While this seems appealing, fast connections are necessary to avoid debilitating delays or poor quality (i.e., choppy) video. For multipoint videoconferencing, all participants must have access to video cameras—a requirement that may be unrealistic for courses where students are logging in from home. Streaming video is becoming more common and is often replayed rather than live.

The all-too-common talking head generally does not provide useful information; often a high quality photograph of the presenter and high quality audio are more effective since they require less bandwidth and the student can focus attention on the supplemental materials such as PowerPoint slides. However there are instances when it can be quite riveting when done well and when the visual information is of well-known people or of objects so that useful information or authenticity or credibility is being provided (Draper, 1998). Web casting or Web conferencing couples many of the above synchronous technologies into one package, using either the telephone or VoIP for audio. A few products even allow either to be used, providing maximum flexibility depending on location and phone charges. These technologies vary considerably in quality and cost, aswell as feature set. As complexity increases, the number of capabilities becomes too much for one person to manage when teaching and an assistant or moderator becomes necessary, especially if a text chat is used since it is very difficult to talk to a class and monitor a chat at the same time. In addition, extensive preparation and scripting is needed for a well-run session. For instance, if a polling tool is used, questions need to be entered in advance. Learning Management and Learning Communities Learning management systems (LMS) and learning content management systems (LCMS) provide the registration and administrative processes for distance education. They can aid a student in determining learning needs through a skill-gap analysis and personalized learning plan. They track student progress so that a teacher or manager can view what a student has accomplished.

Typically they provide multiple views and levels of security based on roles, including administrator, faculty, and student views. Many provide additional capabilities, such as authoring tools to create content. Authoring tools can be separate from an LMS, especially when used to develop multimedia. Learning portals and learning communities are a means of bringing informal and formal learning together, using a virtual classroom for live sessions and knowledge management techniques to facilitate informal knowledge locating and sharing. Communities of practice similarly support informal learning. Sometimes this coupling of formal and informal learning is offered through a personalized learning portal. When students take a course, arguably much of the learning takes place following completion when they are applying what they have learned and providing context that is often lacking in the physical or virtual classroom. There are many opportunities, especially when students are already used to communicating online, to facilitate sharing of the learning that follows a course in an online learning community.

As an example of a learning community, CTA Resource, www.ctaresource.com, is an interactive community of practice dedicated to those interested in the field of cognitive task analysis (CTA). CTA is a specialized field of practice with a relatively small number (a few thousand worldwide) of researchers and practitioners. Initially established and launched through funding from the U.S. Office of Naval Research, this not-for-profit Web site is intended to facilitate interaction among the CTA community worldwide. CTA Resource offers free membership and provides access to a variety of online resources including an extensive digital library of CTA literature and technical reports, a contact list for members, a directory of companies and academic institutions working in the field, and tools to enable members to communicate with each other. Interaction can occur at any time through the use of discussion threads or e-mail between members.

Unlike a structured academic or training course, there is no explicit, shared motivation for members to visit the CTA Resource site. Members need encouragement to visit the site in groups and interact with each other. To this end, online seminars have been offered as a means to educate members while motivating large numbers to visit the site at the same time and interact (Neal, Miller, & Anastasi, 2002). Survey responses from members indicate that there is perceived value in the online offerings: "The on-line seminar is an excellent method of sharing information in a participative manner without the expense and time of attending a physical conference."; "Actually, I like it better. It is a little more self-paced, and easier to get your questions asked." Members have also indicated the value of such events for those living and working overseas, who may not have many opportunities to network and collaborate with colleagues and experts in other countries. Another benefit of the virtual seminar format is the availability of seminar archives for those who could not attend live or wish to review the materials later.

How to Design, Develop, and Structure Online Courses

Most instructional development evolves by trial and error, where the instructor learns over the years what type of instruction seems to be most appropriate for his or her students without realizing that he or she is adjusting the educational instruction to accommodate such issues. Too often, in planning distance education, the focus is on technology rather than on the needs and constraints of the learner and other stakeholders such as instructors, managers, and support personnel. In an effort to produce something different and highly interactive, technology designers sometimes forget that the focus is learning. Instructional designers and technology experts must work together to minimize mismatches between instructional objectives and technology solutions. Research may suggest potential benefits in use of a particular approach or technology for instruction, but feedback from actual use may reveal little or no benefit in a particular context or situation. Further probing of stakeholders, for example, instructors and students, might indicate that problems inherent in the design of the technology itself caused frustration and prevented instructors and learners from reaping the potential benefits of the technology.

In contrast to the development of classroom-delivered courses, distance education is almost always developed by teams. While sometimes the teacher is the developer, a common model is for development and delivery to be done by separate people. Often one or more subject matter experts are queried by instructional designers, who then work iteratively with the subject matter experts and media experts to develop a course. At the extreme, this process can be likened to movie production in terms of the number of roles involved. These roles include: project manager—oversees planning, schedules, and purchasing; subject matter expert—determines and validates content; instructional designer—ensures that learning takes place; graphic designer—uses authoring programs to implement instructional designer's storyboards; programmer— enters course content and builds interactive exercises; quality assurance specialist—tests and evaluates; interface designer— determines look and feel to fit student needs; multimedia developer —incorporates audio, video, and animation; writer or editor—creates text and verifies grammar and style; Web master—handles technical support and maintains hardware and network connections; librarian—provides resources; instructor—ensures the material can be taught effectively; and institutional stakeholder—must be satisfied with course quality.

Martinez (2001) found that students did best, in terms of satisfaction, intentional learning performance, and learning efficacy, in environments that best suited their learning orientation; time management was also a factor. Some of the key learning theories that are commonly applied to distance education are implicit learning (intuition)—for complex tasks; explicit learning—for salient noncomplex tasks; objectivistic—learn by being told; constructivistic—learn by doing with the learning situated in an authentic or realistic context; problem- or scenario-based, in which a student is given a problem to solve and, in the process, has to delve into many aspects of the subject matter but with learning self-directed; and activity theory—learning shaped by physical. These theories lead to a style of instruction and course design.

Course Templates, Development Plan, and Costs

Course templates can help instructors create the initial structure for an online course and can promote consistency in the design of course artifacts within and across online courses. Instructors can use templates to create announcements, course content, descriptions, objectives, policies, registration, syllabi, assignments, discussion forums, participant biographies, and postclass feedback (Designing Web-based Training, n.d.). Templates typically include a "what you see is what you get" (WYSIWYG) content editor. Course content can be created from scratch using the editor or linked to or uploaded from existing files or a content repository. Instructors can typically copy and modify default templates or create new templates to meet the needs of a specific course.

The types of templates need to be carefully planned to maximize effectiveness of a course. Because online courses generate unique challenges, it is important to look at the social, not just educational, needs of students. For example, when dealing with different cultures it is imperative that the learners share personal information about themselves so the other learners can assign an identity to the electronic communication (Klobas & Haddow, 2001). LeBaron and Miller (2004) describe the use of expanded student biographies and an ice-breaker exercise to promote student interaction and initiate a sense of community within a graduate-level online academic course. Personal pro- files or homepages with photographs are an increasingly common way that students can get to know each other and their teacher and feel connected despite the distance.

A course development plan to analyze, design, build, and evaluate a course should be developed (Hall, 2000). The typical steps are to analyze the learning problem, the organizational issues, and identify learner characteristics, knowledge, and skills; design what the instruction will look like: instructional goals, teaching/learning activities, and how the learning will be evaluated; develop materials to the design description and test; implement or plan for how a class will be delivered, received, updated or revised, and maintained and any training and piloting; and evaluate throughout the design process and during implementation to make sure class is effective for the learners. Iterative testing, even of prototype or storyboard, should be done to ensure that the course will be effective, and a course should be piloted with representative audience in as close as possible to the actual setting. Evaluation is discussed in depth in a later section. Another part of course development plan is to determine the shelf life of a course, that is, how frequently it needs to be checked for accuracy (including working links) and how often it should be updated.

It is difficult to estimate the time and cost for producing an online course. ASTD uses very general ranges of 40 hours of development for 1 hour of classroom instructions and 200 hours of development for 1 hour of finished computer-based training. Some of the variables impacting time and cost are the length of the module, since a longer module will cost more; complexity of the topic and resulting material to teach it; testing; graphics and multimedia, with audio, video, and animation being the most time consuming and expensive to create; the simplicity or complexity of navigation since the more options included; the more complex and costly the development will be; and content, whether content already exists or needs to be developed or updated.

Instructional Design

Distance education typically relies heavily upon textual materials. Some data indicates that a continuum exists of lecture, reading, seeing, hearing, seeing and hearing, collaboration, doing, and teaching others where lecture offers the lowest retention and teaching offers the highest. Gagne (1985) provides a nine-step model to help ensure effective learning. His events of instruction include:

  1. Capture learners' attention with a thought-provoking question, interesting fact, or visual information
  2. Inform learners of objectives by initiating the internal process of expectancy and helping motivate them; objectives should form the basis for assessment and evaluation
  3. Stimulate recall of prior learning by associating new information with prior knowledge, which facilitates learning; ask about previous experiences or build upon an understanding of previously introduced concepts
  4. Present the content,which should be chunked and organized meaningfully, explained, and then demonstrated, and to appeal to different learning modalities, use a variety of media (text, graphics, audio narration, and video)
  5. Provide learning guidance to help learners encode information for long-termstorage with examples, nonexamples, case studies, graphical representations, mnemonics, and analogies
  6. Elicit performance; such that learners practice a new skill or behavior to confirm correct understanding, and repetition increases the likelihood of retention
  7. Provide feedback as learners practice and provide specific and immediate feedback of performance
  8. Assess performance with posttest or final assessment without receiving additional coaching, feedback, or hints; mastery of material is typically granted after achieving a score of 80-90% correct
  9. Enhance retention and transfer to job through repetition of learned concepts

These items have led to the principles commonly used for instructional design; Keller (1987) developed the ARCS model of motivational design, a systematic model describing four factors (attention, relevance, confidence, and satisfaction) intended to aid in designing instruction that is motivating to students. The ARCS model categorizes the four factors as follows: attention— arouse curiosity and sustain interest; relevance—make tasks relevant to learner's needs and interests; confidence—build on success and gain self-efficacy; and satisfaction—provide extrinsic and intrinsic rewards.

New models more fully leverage the structural and communication potential of the Internet. Engagement theory (Kearsley & Shneiderman, 1999) carries these notions further into online learning, stating that students must be meaningfully engaged in learning activities through interaction with others and worthwhile tasks, facilitated and enabled by technology. These learning activities occur in a group context, are project-based, and have an authentic focus. "The difference between engagement and interactivity reflects the shift in thinking about computers in education as communication tools rather than some form of media delivery devices" (Kearsley and Shneiderman, 1999). However, this requires a group of students taking a course at the same time, under supervision, which not all distance education includes. Even though the name engagement theory implies a more engaging design result from application of the theory, a student needs to be motivated for even the most engaging course to teach him or her.

In implementing these theories, online course designers should keep in mind how the course will realistically be used by students. Many of the factors that need to be considered arise from scenario building, such as low levels of comfort with technology and blocks of time to study that may be short or interrupted. Online courses can be designed to increase motivation by challenging students and providing tasks that are fun to accomplish. However, "fun and pleasure are elusive concepts" (Norman, 2004) and there is no consensus on how to design enjoyable experiences (Monk, Hassenzahl, Blythe, & Reed, 2002). Norman discusses the importance of emotion in design and that "you actually think differently when you are anxious than when you are happy . . . . devices and software should be designed to influence the mood of the user; they will be more effective because they are more affective" (Gibbs, 2004). Engagement is accepted as important in distance education but is similarly elusive.

Usability Principles

Usability principles increase the likelihood of designing a course That works from the learner's perspective. The five aspects of usability (Constantine & Lockwood, 1999) are learnability, rememberability, efficiency in use, reliability in use, and user satisfaction. Since learners typically do not tolerate extensive training before taking an online course, it is important to keep these principles in mind during design to ensure that learners can focus on learning, not on using the learning application. A successful design facilitates learners reaching a flow state (Csikszentmihalyi, 1990), in which learners are so focused on what they are doing that they are unaware of the passage of time. Inability to, for instance, navigate to the next screen easily disrupts flow. It is also important to remember that, when a course is delivered over the Internet, learners may have developed habits of skimming rather than reading material or quickly click to the next screen. Information in an online course needs to be designed to accommodate how people read online and make judicious use of highlighting, fonts, and formatting to increase readability. An appropriate balance is needed between boring and flashy in order to capture attention and enhance learning without detracting from their focus.

Progress indicators are a basic component of a well-designed interface and are especially important to help pace a student, set expectations, and offer feedback on progress. Some indicators to include are how long something should take to do or play, which should be exact for audio or video and an estimate for a lesson or exam to set expectations and allow good allocation of time, and clear indications of where a student is in the course, what the student has accomplished so far, and what he or she should do next. To instill learner confidence, it is important to inform learners about learning objectives, course requirements, and assessment criteria and to provide feedback on their accomplishments (Keller, 1983).

Another aspect of interface design that is critical for many online courses is appropriate internationalization and localization. Internationalization means designing the interface and writing for a global audience. Localization means making modifications to fit the language and culture of students. This includes being sensitive to the differences between, say British English and American English, providing local references and examples, and understanding the differences between reactions to and acceptance of humor. Although it could also include the use of different instructional styles to accommodate students, it rarely does.

Similarly, there are other student differences that impact design; the most common to accommodate are age, gender, background, or disabilities. In some countries or institutions, it is essential for legal reasons to make courses accessible to people with a range of disabilities. The key ones are visual, hearing, motor skills, and neurological disabilities. However some of these, such as color blindness, are fairly common and others may be important because of the environment a student is in, such as one that is noisy.

Selection of Media

The heart of a well-designed online course is appropriate selection and use of media. While the technologies to support the variety of media are described above, a designer needs to decide which media to use and then which technology to use to support it. Sometimes, in a constrained environment or within an established program, there is no choice of technology, only the choice of which components to use and how to use them most effectively within the course.

Visual language refers to the tight integration of words and visual elements, including animation, cartoons, and diagrams (Horn, 1998) and is important in distance education because the online materials are crucial to student learning and there may be limited if any support available as a student uses these materials. A standard has developed in online courses that is often referred to as electronic page-turners, where a course consists of screens of material that a student clicks through. Each screen has some text and a graphic, with some optional interactive elements such as animations. An important principle for distance education design is that graphics and interactive elements should be purposeful and aid in learning. At worst, they can be distracting, much like banner ads. But visual representations can be very powerful at conveying information (Tufte, 1990, 1992), as can audio and video. The challenge to an online course designer is to determine how to most effectively convey the materials to students, given the nature of the topic and the demographics of the students, and to determine how to best use text, animations, graphics, audio, video, exercises, discussions, and projects to teach. The Media Selection Matrix (n.d.) is based on learners' access to and familiarity with distance education technology (Roblyer & Ekhaml, 2000) and offers a rubric for assessing interactive qualities of distance learning courses that can be used to design an interactive course. Moss (2002) provides some guidelines and suggestions to design to increase learning, for instance, giving an example of how different exercises engage different (and additional) mental processes.

One of the most difficult aspects of course design is determining the type and design of testing. Although there are some examples of distance education in which completing the course requires merely clicking on all or a high percentage of screens, most use some testing to gauge learning. Self-assessments give immediate feedback to a student, generally from multiple-choice questions. While multiple-choice questions offer the advantage of immediate feedback to the student, it is often difficult to develop questions that require deep thinking. Open-ended questions typically elicit more thoughtful answers and can more accurately gauge learning, but require grading and individualized feedback. The best approach is a function of what is possible resource-wise and the value to the learner.

Before and After Distance Education

The above discussion pertains primarily to the design of courses. A last point on course design is the role of pre- and post-class activities. Before a class commences, students can become comfortable with any technology they will be using, can engage in pre-reading, either optionally or to help them acquire knowledge so that students have a level playing field in class, and can introduce themselves through profiles or a discussion forum. An introduction, for first time students, on how to be a successful online learner can be helpful, especially since many of the skills are different from what is needed to succeed in the classroom. Following an online course, there are opportunities for students to continue discussions. Often the same tools used for class can be used to develop what are sometimes called communities of learners or communities of practice. This may be especially valuable in environments in which students are learning and using skills and can reflect upon and provide context to fellow learners.

Techniques and Best Practices for Delivery of Distance Education

Teaching online can be quite different from teaching in the classroom. Although not all distance education has a teacher—self-paced courses being an example—when they do the teacher needs to be adept at using the delivery technology and at interacting with students effectively without the physical presence and visual information that are so abundant in the classroom. It is not enough for an instructor to be expert in the content being taught—instructors must be adequately prepared to ensure their online courses will be a positive experience for students. This reaches beyond familiarity with the technology itself—an understanding of how teaching practices and behaviors need to be adjusted to succeed online is required. For example, adjustments might be needed regarding the number of assignments and due dates, given that text-based interaction and coordinating coursework and feedback between peers and the instructor often take longer than classroom-based situations. Berge (1995) defines the roles of an online instructor to be pedagogical, social, managerial, and technical and points out that these may not all be carried out by the same person; in fact, they rarely are. Each type of medium (e.g., print, audio, video, Web based) requires specific instructional skills for effective use. For example, those teaching in video-based environments must learn on-camera behaviors and adapt to lack of learner feedback, while those using Web-based technology must adapt to the absence of nonverbal cues and learn to interpret online communication nuances (Schoenfeld-Tacher & Persichitte, 2000). It is crucial for instructors to understand how to use technology, apply it to the discipline, generalize it to learn new applications, and guide students in applying the technology. Ideally, m instructors should try to identify opportunities to improve learning via technology by developing a variety of teaching strategies and understanding how technology shapes new teaching roles and student roles (Coughlin & Lemke, 1999). This understanding and ability to leverage technology in teaching is a learned skill—it does not come naturally to most people.

Schoenfeld-Tacher and Persichitte describe areas in which instructors might require training for competence in online instruction including (a) Time to prepare/teach, (b) communication methods (using technology), (c) build on experiences of other teachers, (d) strategies to increase student-student and student-teacher interaction, and (e) organizational details. One cannot assume a simple transition from a face-to-face course to an online setting. It is not simply a matter of an instructor digitizing existing course materials and then coasting through the duration of the course while periodically checking student progress. Such an approach typically results in dry courses that are not engaging. Instead, existing course materials can form the basis for creating learning tools enhanced by technology including help, video and audio presentations, animations, and interactive models. These learning tools provide many benefits, not the least of which is a more engaging online learning experience. In addition, they provide an opportunity to reach more students. Other techniques that are effective in the classroom, such as storytelling, can be brought to online courses with great benefit to students but with some effort and planning on the instructor's part (Neal, n.d.).

Conveying information in multiple modalities can assist those with certain physical or learning disabilities as well as those who learn best in less traditional ways. Gardner's multiple intelligences (MI) theory, for example, states that every learner possesses certain dominant learning styles (or a single style) by which he or she best responds to learning situations (Gardner, 1993). These styles include linguistic, musical, logical mathematical, spatial, body-kinesthetic, intrapersonal (e.g., self awareness), and interpersonal (e.g., social skills). Leveraging technology in the development of a diverse set of online course materials can help meet the needs associated with a variety of learning styles.

A successful instructor-led online course does not consist merely of well-designed stand-alone course materials. Care must be taken to embody a course with personality—enlivening the learning experience through frequent and meaningful student- instructor and student-student interaction to facilitate a sense of course community. This is particularly important in online learning environments. For example, a perceived lack of feedback or contact with the instructor may impede a student's ability at self-evaluation, while a lack of interaction with peers can lead to feelings of alienation, isolation, and the lack of a sense of scholarly community (Galusha, 1997). These outcomes, in turn, can lead to lower levels of learner motivation and engagement. It can be challenging to personalize online courses when technology can limit implicit communication (e.g., facial expressions, gesturing) and make it more difficult to convey and interpret emotions (Norman, 2004).

There have been attempts to build ways that students can express emotion or that emotion can be detected, but none have even approached the success and timeliness of facial reactions and body language in the classroom. Yet instructors can compensate for those limitations through frequent dialogue with students, timely responses to questions and concerns, seeding discussion forums to encourage a scholarly dialogue among students, and providing meaningful assessments and feedback on students' work. In a nutshell—when teaching online it is important to maintain a steady presence by being there for the students. The ARCS model (Keller, 1987) identifies some key factors impacting learner motivation including the confidence level of the learner and sense of satisfaction derived from the learning experience. Learner satisfaction can be supported through extrinsic rewards such as positive reinforcement and motivational feedback.

Online grading tools help instructors track grading and provide feedback on student work. If an automated grading scheme is established, the grading tool can update the grade book whenever a new assignment, quiz, or test is completed. Grading tools should be flexible, allowing instructors to manually edit grades, assign partial credit for certain course activities or student answers, and add grades for assignments that require manual assessment. Online grade books allow the instructor to manage and view grades in a variety of ways including by assignment, by student, and for all students on all assignments. Instructors can also search the grade book based on specific criteria. If an assignment is added to the course, it is automatically added to the online grade book. The software can assist with analysis of grades by automatically calculating the minimum, maximum, and average assignment grade. Often, instructors can export versions of the online grade book to a spreadsheet or other data analysis application. Students can be informed of grades for individual assignments through an automated message or via more personalized feedback from the instructor.

Many learning management systems include automated testing and scoring tools that allow instructors to create, administer, and score objective tests. These testing tools allow instructors to create test questions in a variety of styles including true/false, multiple choice, multiple answer, fill-in-the-blank, matching, short answer/essay, and calculated answer questions. Instructors can typically create feedback messages to be displayed when varying conditions occur. Many tools provide the ability for instructors to create personal course-specific test repositories as well as tap into system-wide repositories, and then use that stored content to create tests for students. The system can randomize the questions in a test, and some tools can randomize on a per-student basis to reduce the chance of cheating. Instructors can control access by assigning dates when tests can be accessed or setting time limits on a test. Some instructors allocate a portion of a course grade to student participation. Tracking software allows the instructor to track the usage of course materials by students. Robust tracking software can report individual usage as compared to overall class usage of various materials and resources. However, often more important than the number of times or length of time that a student spends accessing course content is the quality of the time spent (i.e., the scholarly contributions a student makes to the overall course). Assessing the quality of each student's participation in a course is a challenge for online instructors—one for which tracking software offers little assistance.

Assessing the quality of online participation is very subjective, and a student's perception of his or her own participation may vary from that of his or her peers or the instructor. It is important for the instructor to describe the criteria upon which an assessment of participation is based, particularly if the assessment consists of more than a mere count of the number of posts each student makes. Example criteria defining high quality student participation might include the level of reflection, thoughtfulness, depth of dialogue, research quality, and consistency of participation in course discussion threads or within the context of a specific interactive exercise. It is often helpful for the instructor to model quality dialogue by providing an initial example (e.g., some discussion posts or chat interaction) with a teaching assistant or student volunteer. The instructor's ongoing interaction within the course environment—to the class as a whole as well as with individual students—also sets the tone for the expected level of interaction within the course. When the instructor identifies students falling below the threshold of optimal course participation, it is important to express that concern to the student and reiterate expectations in a timely manner so that the student has enough time remaining in the course to remedy the situation. This is important not only for the benefit of the student in question, but for the overall benefit of the class as the lack of participation on the part of one or more course participants can be discouraging and demotivating to the rest of the class.

With respect to all of the assessment methods described above, it is crucial for instructors of online courses to clearly describe the structure of the class and associated learning tasks and activities in the course syllabus. In addition, a detailed rubric describing major course assignments and breaking down how each factors into the overall grade should be provided. Ambiguity can be a source of anxiety in traditional classroom settings. In distance learning situations, the perceived distance a student might feel between his or herself and the instructor can exacerbate feelings of anxiety stemming from ambiguous course objectives or instructor expectations. That said, merely summarizing each assignment and stating its weight in the overall course grade may not provide enough information for students to feel comfortable and prepared to tackle each assignment. Whenever possible, instructors should try to provide supporting information with respect to course assignments. Examples include more detailed descriptions of assignments, links to related resources, examples of exemplary work, a breakout of allocation of grade points to various aspects of the assignment, frequently asked questions (FAQ) about the assignment, and so on. The trick is to think about each assignment, try to anticipate all of the questions and concerns students might have about it, and be proactive. Provide the answers using a combination of techniques such as those listed here. This approach can help students feel empowered to succeed while minimizing confusion and heading off a deluge of questions later on.

Effective online teaching that promotes engaged learning requires a large investment of time and effort on the part of the instructor. Facilitating active communication and community-based construction of knowledge in an online learning environment can take substantially more time and effort than comparable traditional efforts. For example, a face-to-face version of a graduate-level academic course might consist of a weekly 3-hour course session supplemented with a handful of office hours on a weekly basis for students to use as needed. An online version of the course would require much more of the instructor's time, albeit not necessarily on campus, to be successful. Providing timely, substantive feedback as well as providing an instructor presence for a group of online students on an ongoing basis, along with grading major course assignments, requires a significant instructional time commitment. Enlisting a teaching assistant—perhaps a graduate student—to assist in developing online course materials, moderating class discussions and chats, and managing ongoing course activities can yield a high level of responsiveness to students while allowing for collaboration on the design of course exercises and assessment materials. Based on their own positive experiences as an instructional team, as well as the perceived quality of instruction by their students, LeBaron and Miller (2004) recommend a collaborative instructional approach when feasible—particularly for novice online instructors.

Evaluation of Distance Education Programs

An evaluation plan for a distance education program is essential to determine if the course meets the stated goals and, if not, to determine what to revise. Evaluation provides the data needed to determine the effectiveness of a program so that stakeholders can decide whether to accept, change, or eliminate any or all of its components. Good evaluation looks beyond the surface to inform decisions regarding content topics, organization of content, delivery methods, and so on. It involves specifying a set of criteria to be evaluated, identifying appropriate measures to inform us about the criteria, and analyzing or otherwise examining the resulting data with respect to the criteria. Evaluation is not always done when developing distance education, or it is done in a cursory fashion to eliminate glaring problems. In many corporate environments, there is more of a focus on measuring return-on-investment (ROI) than in evaluating the effectiveness of a program. Certainly quantitative and qualitative methods of collecting and processing data have their place in evaluation, and a preference for one over the other is largely based on how the data will be used.

Evaluation can be defined as the formal determination of quality, effectiveness, or value of a program, product, project, process, objective, or curriculum. Evaluation includes determining standards for judging quality, data collection, and applying the standards (i.e., criteria) to determine quality. Evaluation is the means for determining whether a program meets its goals and whether the instructional inputs match the intended or prescribed outputs. Evaluation can be a broad and continuous effort to inquire into the effects of utilizing content and process to meet clearly defined goals. All of these definitions have a common theme—evaluation can only be conducted after defining a set of criteria or standards based on underlying the educational goals and objectives.

Evaluation can take on a variety of forms and should ideally occur at various stages throughout the life cycle of a distance learning program. The first two definitions of evaluation in the preceding paragraph imply a summative evaluation—one which seeks to determine the level of quality or success after implementation (i.e., is the course as delivered working?). Formative evaluation, on the other hand, entails an approach which seeks to inform the design of a program as it evolves over time. Data are collected during design and development to fine-tune the program before it is implemented in a distance education program. Evaluation, however, is situation-specific and often takes place in a naturalistic setting.

Ornstein and Hunkins (1997) summarize five types of value questions regarding curriculum evaluation:

  1. Intrinsic value pertains to goodness and appropriateness, as planned and as delivered; it asks whether a program incorporates the best thinking to date on the content, arrangement of the content, and its presentation. Intrinsic value is tied to philosophical and psychological views.
  2. Instrumental value asks what the curriculum is good for (does it link back to the stated goals and objectives) and who the intended audience is (will students benefit from the program? Is it appropriate, given the context?).
  3. Comparative value is concerned with determining whether the new or revised program is better than the original. Other factors include delivery, cost, necessary resources, and how it fits into the organization.
  4. Idealization value is not concerned so much with evaluating expected outcomes, but with how to continuously improve and fine-tune the program.
  5. Decision value reminds us that the value of the decisions made regarding curriculum need to be assessed as the curriculum is being delivered.

It is important to maintain a historical perspective, capturing the key decision points and documenting the "whys", so those decisions can be revisited once data and experiences are available to ensure that the right path was followed.

The Evaluation Process

While the specifics involved in an evaluation (i.e., evaluation criteria, methods of measurement, analysis) vary depending on purpose of the evaluation and the philosophies and values of the stakeholders, evaluations must be systematic in order to provide useful results. That is, after all, the point—to inform the design or revision of a product. The following series of steps is common to many forms of evaluation:

1. Evaluation criteria—identify the phenomena to be evaluated and the design to use for the evaluation
2. Data collection—identify information sources and the means of collecting necessary information
3. Organize data—transform the data (e.g., via coding, organization) into interpretable
4. Data analysis—interpret the data
5. Reporting—frame the results in meaningful fashion for the final audience by summarizing it, interpreting the findings, and providing recommendations
6. Refining—use the results to provide iterative feedback regarding the evaluated product in order to continually fine tune it.

The importance of appropriate reporting cannot be overemphasized. An evaluation can be based on a sound set of criteria and include meticulous data collection and thorough analysis, but if the results are not documented in a meaningful way for the target audience (i.e., stakeholders), it is all for naught. A thorough report should include not only a summary of the results and analysis, but an interpretation of the implications for the distance learning program. Whenever possible, a set of recommendations should also be provided for consideration by the stakeholders, keeping in mind that the type of and level of detail of the recommendations differ depending on the recipient's perspective and role.

Usability Evaluation

Usability evaluation, as part of formative evaluation, looks at how well designed a course is from the learner's perspective. The five aspects of usability discussed in the section on How to Design, Develop, and Structure Online Courses are learn ability, remember ability, efficiency in use, reliability in use, and user satisfaction. It is important to test for these during evaluation to ensure that learners can be successful. For instance, if text is difficult to read or a learning application is difficult to navigate, learners may become frustrated and drop out. A consideration in technology evaluation and selection might be its appropriateness for both novice and more expert users (i.e., those with varying degrees of technical savvy). A well-designed, robust technology application should be usable and useful for learners with a diverse set of technical skills.

Evaluation During and Following a Course

Feedback can be obtained during and following a course using both formal means, such as online surveys and data gathering, and informal techniques. The informal include using e-mail, phone calls, and office hours that are for student assistance as a way to seek feedback. Online survey design needs to balance the desire to receive feedback with student willingness to spend time filling in a form. Multiple-choice questions are quicker to answer and compile but offer far less information than open-ended questions. Some online courses require a survey to be completed before the student receives course credit. While that ensures a high completion rate, students may be less willing to respond honestly and carefully under those circumstances. Capturing data through logs is usually easy and supported by many LMS. At a very low level the data tend not be useful for improving a course or understanding how students are participating. Some data, however, can be quite helpful, such as learning which parts of a course students spend the most time on and when they access the course and for what duration. Kirkpatrick (1994) provided the standard scale for measuring training effectiveness, which is used following an online course.

m Level 1 asks how the student reacted; level 2 asks whether learning took place, typically determined by a comparison of pre- and post-tests; level 3 asks whether behavior has changed as a result of the training, that is, whether the student is able to perform his or her job better; and level 4 asks whether the training contributed materially to business results. It is easier to measure impact on productivity of a salesperson than a knowledge worker. Additional ways of measuring training effectiveness include tracking enrollments as an indicator that training is attracting students who perceive that it will meet a need and tracking completion rate to indicate that courses were effectively designed. In such cases, assumptions are often made that require supporting data.

The topics, adapted from Evaluation for Distance Educators (n.d.), that are valuable to receive formal and informal feedback on to improve an online course or distance education program include:

1. Use of technology—student familiarity, problems, positive aspects, attitude toward technology, usefulness of technology skills learned in class to other venues
2. Class formats—effectiveness of lecture, discussion, question and answer; quality of questions or problems raised in class; encouragement and support given to students, pace of class
3. Class atmosphere—conduciveness to student learning, social aspects
4. Quantity and quality of interaction with other students, with instructor, and with any support staff
5. Course content—relevancy, adequate body of knowledge, organization, pointers to supporting resources
6. Assignments—usefulness, degree of difficulty and time required, timeliness of feedback
7. Tests—frequency, relevancy, sufficient review, difficulty, feedback
8. Support services—facilitator, technology, library services, instructor availability
9. Student achievement—adequacy, appropriateness, timeliness, student involvement
10. Student attitude—attendance, assignments submitted, class participation
11. Instructor—contribution as discussion leader, effectiveness, organization, preparation, enthusiasm, openness to student views

Some open-ended questions that provide in-depth feedback include:

1. What were the strongest aspects of this course and why?
2. What were the weakest aspects of this course and why?
3. What would you suggest to improve this course?
4. What were the most important things you learned in this course?
5. What would you have liked to have learned in this course that you did not learn?

ROI may not directly help improve a course or program, but helps justify the funding and support for distance education. Typically ROI compares course development and delivery costs with student and teacher travel, classroom construction costs, and any additional expenses such as food. Some corporate ROI calculations include student salaries for the time they would have been in class, but that may be a flawed measure since, with distance education, a student still needs class time, it is just in smaller blocks and may be taken from the student's personal, rather than work, time. Measuring ROI necessitates assigning a cost structure to each objective and understanding effect on savings and impact on performance, and many are difficult to calculate, such as the value of an employee who is retained longer through better educational opportunities. Some of the more quantifiable measures include the decrease in production costs to update and distribute materials; the relative ease of tracking, monitoring, and measuring student performance; and the reduction in travel and lodging costs.

However, for all the evaluation techniques discussed, and without discounting the importance of the many measures, the most essential aspect of evaluation is to determine whether learning took place. This is commonly done through testing throughout a class and at the end of a class, by comparing the results of pre-and post-testing and by evaluating student assignments. This is, of course, important for all education, not just distance education, but the distance can make it more difficult to use more subjective measures such as seeing the spark of enthusiasm in a student or seeing a student have an "aha" moment in class. Without those visual opportunities, students can still report their subjective satisfaction and understanding, and their online behavior can indicate their enthusiasm for and understanding of a course topic. Longitudinal testing can be used to measure retention.

Support for Learners

"Teachers and students come to conventional higher education having already learned well-defined roles through years of common educational background and experience in the formal education system. During online learning this background and prior experience are less relevant to the context, which can invoke feelings of anomie" (Anderson et al., 2001). Students need help in advance to prepare to be successful distant students as well as support during a course. Students know what to do and how to behave in a physical classroom, for example sit down, look attentive, and take notes. Online students, however, need to be taught what to do, how to act appropriately, and the importance of time management and distraction control.

Students who are more familiar with technology and distance learning are more likely to do better in a distributed instructional setting that requires computer use (e.g., see Gilbert & Werge, 2000; Irani, 2001). Students should feel comfortable enough with the technology that it does not become a barrier to successful learning. In addition, students need to acquire other necessary skills to support a positive learning experience including communication, collaboration, research, critical analysis, problem solving, and self-management. Students working online often work very independently—the lack of regularly scheduled classroom time may result in online students feeling less accountable for their work. It can be very easy for students to fall behind and very difficult for them to catch up later. Students must be highly motivated and able to work independently successfully.

Several of the barriers to distance learning described by Galusha (1997) pertain to the lack of student preparedness and support. With respect to online learning, first-timers have special needs. Study materials must reflect fact that many students have little or no experience with online learning. Technical barriers must be made a non-issue. Selection of intuitive and flexible technology tools, coupled with technical training, can reduce technical barriers. However, similar to the challenges instructors face, students need to acquire more than a basic understanding of the use of the technology itself—they need to understand how to use it efficiently to support their learning tasks. For example, the Internet provides access to a wealth of resources with varying degrees of reputability. Students need to learn good research skills, including good Internet search skills. Due to the sometimes overwhelming amounts of data available, they also need guidance on where to look first (e.g., recommended data repositories) and to develop the ability to critically assess the information they find. A tutorial with an affiliated library service can help students hone these skills, and embedded links to reputable and relevant data resources can point students in the right direction for research projects.

Scaffolding can also help train students in use of technology tools. For example, LeBaron and Miller (2004) describe the design and implementation of an icebreaker exercise for an online course. This loosely structured exercise was carried out using the teamwork capabilities provided by the learning management system used for the course. While this group exercise was intended to promote a sense of community among students at the onset of the course, it had the added benefit of training students in the use of the teamwork capabilities early in the course, as these same tools would be used later on in a more comprehensive mid-semester group exercise. Initial training such as this is advocated by Lin (1999), who found that practice time to acquire skills needed in an online course had positive impacts on self-efficacy, interest, and commitment.

Group work presents another challenge to online learners, who need to collaborate in novel ways to achieve group goals and work effectively. The use of technology tools changes the nature of interaction and group work. Students need to understand the dynamics of online interaction, including understanding "netiquette" (the etiquette of online behaviors). Students involved in group projects need to respect the accountability they have to their work partners as well as to the instructor. It can be more difficult to chase or prod an unresponsive virtual partner. Coordinating work on joint artifacts presents another challenge. Although in today's workplace such coordination is not uncommon, it cannot be assumed that all group members have similar technical skills or access to compatible versions of software tools (e.g., word processors, spreadsheets, data analysis software, presentation editors). Team members must learn how to coordinate and execute their group work.

Studying online requires a high level of motivation and an ability to work well independently. Students need to learn how to manage their study time, particularly when there are no face-to-face classes to reinforce a sense of accountability to the instructor and the class. In the online world, it can be very easy to let other responsibilities or distractions take precedence and, once one has fallen behind, it can be very difficult to catch up. Course schedules integrated with academic planners can help, as can weekly course updates and reminders from the instructor. An instructor who presents an organized syllabus, provides timely feedback, and shows an overall commitment to the course can act as a role model demonstrating effective work practices.

The physical separation of students in programs offered at a distance may contribute to higher dropout rates (Rovai, 2002). Separation has a tendency to reduce the sense of community among students, giving rise to feelings of disconnection (Kerka, 1996), isolation, distraction, and lack of personal attention (Besser & Donahue, 1996; Twigg, 1997), which could affect student persistence in distance education courses or programs. Course design, incorporating instructor and peer interaction, and providing adequate support for students, especially first-time online learners, can increase the sense of community, leading to greater student success as well as a more successful course or program.

Conclusions

Distance education exists in many forms, from self-paced asynchronous courses to highly interactive courses delivered using videoconferencing and Web conferencing technology. The commonality, of course, is the lack of physical proximity, which is what prevents the familiar teaching styles and interaction that occur in the classroom. The range of existing distance education also differs in the types and amount of support available to students. Although the distance inherent to an online course does not per se necessitate that a student takes a course without a teacher or peer support, this is sometimes the case. Yet technology can facilitate rich discussion that takes place synchronously or asynchronously, and, ultimately, the selection of technology and media to support learning at a distance is dictated by technology access, availability and cost, student demographics and needs, and the objectives and constraints of the institution offering the courses. Even within a highly constrained situation, it is possible to offer high quality online courses that meet students' needs and provide engaging learning experiences. The support of teachers and peers can enhance many learning experiences and can help students with the low points— when they are stuck or confused—as well as with the high points—when they are excited and enthusiastic—about what they are learning. Students can benefit not just from access to educational opportunities but from the increased technology skills and digital literacy that result from being online learners. It is important in planning, designing, or delivering distance education for the focus to remain on learning: how can this topic be most effectively conveyed to the target learner population? This chapter has covered many aspects of technology and course design, delivery, and evaluation in support of this objective.

Exercise: Evaluate Online Course

Many free online courses are available. Try to take at least two or more for the sake of contrast. Answer the following questions for each course:

  • What is the target audience and the topic?
  • What works and does not work about the overall design; the interface, navigation, and usability; the content; and the assignments, activities, and tests? Be detailed in your response.
  • Consider the delivery technology and media (audio, video, text, etc.)—what works and does not work for the target audience and topic?
  • How would you evaluate the effectiveness of the course?
  • How would you redesign the course and what improvements would you expect as a result?
  • What have you learned from contrasting the different approaches?
  • What evaluation criteria do your answers to the above questions imply?

References

1. Andersen, E. (n.d.). Infrastructure: The things we take for granted. In ACM Ubiquity. Retrieved March 6, 2004, from www.acm.org/ubiquity/views/e andersen 3.html

2. Anderson, T., Rourke, L., Garrison, D. R., & Archer, W. (2001). Assessing teacher presence in a computer conferencing context. Journal of Asynchronous Learning Networks, 5(2), 1-17. Retrieved March 6, 2004, from www.sloan-c.org/publications/jaln/v5n2/v5n2 anderson.asp

3. Berge, Z. L. (1995). Facilitating computer conferencing: Recommendations from the field. Educational Technology, 35, 22-30.

4. Besser, H., & Donahue, S. (1996). Perspectives on . . . distance independent education: Introduction and overview. Journal of the American Society for Information Science, 47, 801-804.

5. Competitor analysis. (n.d.). Retrieved February 17, 2004, from www.hostserver150.com/usabilit/tools/competitoranalysis.htm

6. Constantine, L., & Lockwood, L. (1999). Software for use: A practical guide to the essential models and methods of usage-centered design. Reading, MA: Addison-Wesley.

7. Coughlin, E. C., & Lemke, C. (1999). Professional competency continuum: Professional skills for the digital age classroom. Santa Monica, CA: Milken Family Foundation. Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York: Harper and Row.

8. Designing web-based training. (n.d.) Retrieved March 6, 2004, from www.designingwbt.com

9. Distance education at a glance, Guide #4, Evaluation for distance educators. (n.d.) Retrieved February 18, 2004, from Engineering Outreach at the University of Idaho Web site: www.uidaho.edu/evo/dist4.html

10. Draper, S. (1998, February). When is video good for instruction? Retrieved February 17, 2004, from http://staff.psy.gla.ac.uk/~steve/vidInstr.html

11. Driscoll, M. (2001, August). Strategic plans from scratch. Learning Circuits. Retrieved March 6, 2004, from www.learningcircuits.org/2001/aug2001/driscoll.html

12. Gagne, R. (1985). The conditions of learning. (4th ed.). NewYork: Holt, Rinehart & Winston.

13. Galusha, J. M. (1997). Barriers to learning in distance education. Interpersonal Computing and Technology., 5(3-4), 6-14.

14. Gardner, H. (1993). Multiple intelligences: The theory in practice. New York: Basic Books.

15. Gibbs, W. W. (2004). Why machines should fear. Scientific American, 290, 37.

16. Gilbert, S., & Werge, I. (2000). Education, technology, and change: Queries. (IESD report). Retrieved March 6, 2004, from www.tltgroup.org/gilbert/QuakerQueriesIWswg2-19-01.htm

17. Gladwell, M. (2000). The tipping point: How little things can make a big difference. Boston: Little, Brown.

18. Hall, B. (2000, May). Resources for enterprise-wide e-learning initiatives. E-Learning Magazine.

19. Horn, R. E. (1998). Visual language: Global communication for the 21st century. Bainbridge Island, WA: MacroVu, Inc.

20. IHETS. (n.d.). Guiding principles for faculty in distance learning. Retrieved March 10, 2004, from www.ihets.org/progserv/education/distance/guiding principles/

21. Irani, T. (2001). If we build it, will they come? The effects of experience and attitude on traditional-aged students' views of distance education. International Journal of Educational Technology, 2(1).

22. Kaplan-Leiserson, E. (n.d.). E-learning glossary. Retrieved February 17, 2004, from www.learningcircuits.org/glossary.html

23. Kearsley, G., & Shneiderman, B. (1999). Engagement theory: A framework for technology-based teaching and learning. Retrieved March 6, 2004, from http://home.sprynet.com/~gkearsley/engage.htm

24. Keller, J. M. (1987). Strategies for stimulating the motivation to learn. Performance and Instruction, 26(8).

25. Kerka, S. (1996). Journal writing and adult learning. Columbs, OH: Eric clearinghouse on Adult Career and Vocational Education. (ERIC Document 399 413).

26. Keynes, J. (1997). The General Theory of Employment, Interest and Money. Amherst, NY: Prometheus Books.

27. Kiesler, S. (1986, January). Harvard Business Review.

28. Kirkpatrick, D. L. (1994). Evaluating training programs: The four levels. San Francisco, CA: Berrett-Koehler.

29. Klobas, J., & Haddow, G. (2000). International computer-supported collaborative teamwork in business education: A case study and evaluation. International Journal of Educational Technology, 2(1).

30. Laurillard, D. (1993). Rethinking university teaching: A framework for the effective use of educational technology. London: Routledge.

31. LeBaron, J., & Miller, D. (2004). The teacher as agent provocateur: Strategies to promote community in online course settings. In T. Latomaa, J. Pohjonen, J. Pulkkinen, & M. Ruotsalainen (Eds.), eReflections—Ten years of educational technology studies at the University of Oulu. Essays contributed by the network builders. Oulun yliopiston kasvatustieteiden tiedekunnan elektronisia julkaisuja 3. Lenzner, R., & Johnson, S. (1997, March 10). Seeing things as they really are. Forbes.

32. Lin, C.- J. (1999). The effects of self-efficacy and task values on students' commitment and achievement in web-based instruction for Taiwan higher education. Dissertation Abstracts International Section A: Humanities & Social Sciences, 60(6-A), 1905.

33. Martinez, M. (2001). Mass customization: Designing for successful learning. International Journal of Educational Technology. 2(2). Media selection matrix draft. (n.d.). Retrieved February 18, 2004, from www.cde.psu.edu/de/id&D/media selection matrix.html

34. MIT OpenCourseWare. (n.d.). Retrieved February 18, 2004, from http://ocw.mit.edu/index.html

35. Monk, A., Hassenzahl, M., Blythe, M., & Reed, D. (2002, September-October). Funology: Designing enjoyment. SIGCHI Bulletin.

36. Moss, C. (2002, January 7). Finding balance: The vices of our "versus". First Monday, 7(1). Retrieved March 6, 2004, from www. firstmonday.dk/issues/issue7 1/moss/index.html

37. Neal, L. (n.d.). Storytelling at a distance. eLearn Magazine. Retrieved on March 6, 2004, from www.elearnmag.org

38. Neal, L. (1997, November 16-19). Virtual classrooms and communities. In Proceedings of ACM GROUP '97 Conference, Phoenix, AZ.

39. Neal, L. (2000, June 28). Scenario building to design a distance learning program. In Proceedings of ED-MEDIA 2000-World Conference on Educational Multimedia, Hypermedia & Telecommunications, Montreal, Quebec, Canada.

40. Neal, L. (2003, December 16). Degrees by mail: Look what you can buy for only $499!!! ACM eLearn Magazine (column), Retrieved March 6, 2004, from www.elearnmag.org

41. Neal, L., & Ingram, D. (2001). Asynchronous distance learning for corporate education. In K. Mantyla and J. Woods (Eds.), 2001/2002 ASTD distance learning yearbook. New York: McGraw-Hill.

42. Neal, L., Miller, D., & Anastasi, D. (2002). Launching a community of practice through an online seminar series. In Proceedings of EDMEDIA. 2002, Denver, CO.

43. Norman, D. (2004). Emotional design. New York: Basic Books. No significant Difference. (n.d.). Retrieved February 18, 2004, from http://teleeducation.nb.ca/nosignificantdifference

44. Ornstein, A. C., & Hunkins, F. P. (1997). Curriculum: Foundations, principles, and issues. Boston: Allyn & Bacon.

45. Personal correspondence with Shigeru Miyagawa, 2003. Readiness assessment. (n.d.) Retrieved February 17, 2004, from www.mediapro.com/html/resources/readiness.html

46. Roblyer, M., & Ekhaml, L. (2000). How interactive are your distance courses? A rubric for assessing interaction in distance learning. Online Journal of Distance Learning Administration, 3(2). Retrieved February 18, 2004, from www.westga.edu/~distance/roblyer32.html

47. Rogers, E. M. (1962). < i> Diffusion of innovation.. New York: The Free Press.

48. Rovai, A. (2002, April). Building sense of community at a distance. International Review of Research in Open and Distance Learning. Retrieved March 6, 2004, from www.irrodl.org/content/v3.1/rovai.html

49. Schoenfeld-Tacher, R., & Persichitte, K. (2000). Differential skills and competencies required of faculty teaching distance education courses. International Journal of Educational Technology, 2(1). Tufte, E. (1990). Envisioning information. Cheshire, CT: Graphics Press.

50. Tufte, E. (1992). The visual display of quantitative information. Cheshire, CT: Graphics Press.

51. Twigg, C. A. (1997). The promise of instructional technology. About Campus 2(1), 2-3.



Comments

  • Wed, 02 Dec 2015
    Post by caio fabio

    discusses why distance education has become so popular; and describes how to plan, design, deliver, and evaluate a distance education course or program. caio fabio

ADDITIONAL READING

    Diane Miller
  1. Online learning and fun
  2. Lisa Neal
  3. Is it live or is it Memorex?
  4. The Value of Voice
  5. Predictions for 2006
  6. Five Questions...for Christopher Dede
  7. Five Questions... for John Seely Brown
  8. Five questions...for Shigeru Miyagawi
  9. "Deep" thoughts
  10. 5 questions... for Richard E. Mayer
  11. Designing usable, self-paced e-learning courses
  12. Want better courses?
  13. Just "DO IT"
  14. Five questions...
  15. Formative evaluation
  16. Senior service
  17. Blogging to learn and learning to blog
  18. My life as a Wikipedian
  19. Five questions...for Elliott Masie
  20. The stripper and the bogus online degree
  21. Five questions...for Lynn Johnston
  22. Five questions...for Tom Carey
  23. Not all the world's a stage
  24. Five questions...for Karl M. Kapp
  25. Five questions...for Larry Prusack
  26. Five questions...for Seb Schmoller
  27. Do distance and location matter in e-learning?
  28. Why do our K-12 schools remain technology-free?
  29. Music lessons
  30. Learn to apologize for fun and profit
  31. Of web hits and Britney Spears
  32. Advertising or education?
  33. Five questions…for Matt DuPlessie
  34. Back to the future
  35. Serious games for serious topics
  36. Five (or six) questions...for Irene McAra-McWilliam
  37. Learner on the Orient Express
  38. Talk to me
  39. Q&A with Diana Laurillard
  40. Do it yourself
  41. Degrees by mail
  42. Predictions for 2004
  43. "Spot Learning"
  44. Q&A with Saul Carliner
  45. When will e-learning reach a tipping point?
  46. Online learning and fun
  47. In search of simplicity
  48. eLearning and fun
  49. Everything in moderation
  50. Predictions For 2003
  51. How to get students to show up and learn
  52. Q&A
  53. Blended conferences
  54. Predictions for 2002
  55. Learning from e-learning
  56. Storytelling at a distance
  57. Q&A with Don Norman