Evaluating the development of online course materials

If online courses are to become a permanent feature of higher education—not merely a fad of the dot-com era—college faculty must believe that developing online materials has academic value. In addition, such online education requires certain resources to be in place. Resources technological in nature are easy to identify: computers with browser software, Internet access, servers, and so forth. Other resources are less obvious: computer skills of both students and the developers of online materials, and, an essential component: the resources needed to sustain that development. As content and concepts in college courses evolve, so must online materials. And as technology changes, so do the delivery methods for online materials. Faculty must be motivated to update and renew their course materials and keep up with advances in their delivery. Is faculty motivated? And just how do the perceptions of academics in the US compare with the views of their counterparts abroad?

I conducted a study to examine the similarities and differences between U.S. and non-U.S. faculty in their perceptions of developing online course materials, taking into account such factors as resources and demographics. The results demonstrate that U.S. and non-U.S. faculty share similar views on the academic value of Web-based materials despite varying demographics.

The efficacy of online course materials has already been established (Alvi et al, 1997, Benbunan-Fitch, 2002, Hiltz and Turoff, 2002). Faculty continues to develop and deliver online course materials even though they realize it may hinder their academic career (Schell, 2004). Faculty seems to be devoted to the development of online course materials even after the initial incentives have been discontinued. The motivation for faculty to develop and deliver online materials appears to endure beyond the monetary and professional reward systems.

There is, however, little research regarding the similarities and differences among faculty that develop these materials. The U.S. currently accounts for the majority of Web use, however, non-U.S. countries are increasing their Web use and the amount of content posted on the Web. It is useful to examine non-U.S. views about online materials because the Internet knows no borders.

I developed a survey to collect information about individual faculty members, their schools, and each member’s perception of the academic value of developing online course materials. For this analysis, schools in the United States and non-U.S. schools were compared. There were 481 observations from U.S. schools and 48 observations from non-U.S. schools. Of these 48, 26 countries were represented: from Australia to Saudi Arabia; Mexico to Turkey; the United Kingdom to the Ukraine. It was important for this article that non-U.S. schools represented many different countries—no more than six observations came from a single non-U.S. country.

Responses from Canada (36 observations) are not included in this analysis. It was felt that while Canadian schools are certainly distinct from schools in the United States, Canadian faculty might not be viewed as distinctly non-U.S. because of the close proximity of the two countries, the shared conference experiences that result, and faculty training in each other's universities.

All those surveyed have developed and used online course materials so that they would be able to express their views based upon first-hand experience. Only responses from colleges offering at least a bachelors degree were included in the analysis. Again, the focus of the survey is to determine the academic value placed on developing online course materials.

Survey Says

It’s important to note that an analysis of variance was performed on questions from the survey. The results were informative in situations where significant differences were found and also where no significant difference was found.

The distribution between public versus private institutions were very similar. The sizes of the academic institutions were also similar: The most frequent response was a school size less than 5,000 students (about ten percent of respondents reported more than 30,000 students). Doctoral programs were offered frequently at responding U.S. and non-U.S. schools. The length of time the respondents had been teaching was similar and their computer expertise/experience was also similar.

Overall, non-U.S. faculty places a higher academic value on developing online materials than U.S. faculty. The reason may be that non-U.S. faculty members generally have a more favorable opinion of the effectiveness of the online learning experience. Faculty was asked "How would you compare the effectiveness of a Web-based learning experience to a traditionally taught learning experience?" Their responses are shown in Table 1. The differences did not prove statistically significant but the trend of the last two responses implies non-U.S. faculty believes more deeply in the effectiveness of Web-based learning.

The terminal degrees attained by respondents were markedly different: 48.9 percent of non-U.S. faculty had master’s degrees while 44.7 percent had doctorates. In contrast, U.S. respondents reported 22.9 percent had master’s degrees while 74.8 percent had doctorates. The distribution of job titles reflects the differences in terminal degrees. Table 2 shows titles from adjuncts to full professors. (Note that the rank of the respondent was evenly distributed among the categories of professorship for U.S. and non-U.S. faculty.)

To rank the importance of teaching and research in regards to promotion and tenure, a scale of zero to ten was used—a value of zero meant no importance while a value of ten meant critical importance. Fifty percent of the non-U.S. respondents' reported a nine or ten value for research while only 19.1 percent reported nine or ten for the importance of teaching. However, only 31.1 percent of U.S. respondents reported values nine or ten for research importance and 38.9 percent for teaching. The non-U.S. respondents clearly respect research as an academic endeavor more than their U.S. counterparts while U.S. faculty is more concerned about teaching.

In earlier studies (such as Schell, 2004) a negative relationship was found in U.S. schools between the value of research and the value of developing online materials. However, non-U.S. schools show a positive relationship. This may be due to their higher opinion of an online learning experience. On the other hand, the survey shows that U.S. faculty uses online materials significantly more than non-U.S. faculty. These intuitively conflicting results appear to be caused by a fewer resources available to non-U.S. students.

While few schools from either group required their students to own computers (6.4 percent for non-U.S. and 8.1 percent for U.S.), the computer lab resources available to students differed significantly. While only 5.9 percent of U.S. respondents reported their resources were either "not enough" or "poor,” 29.8 percent of non-U.S. schools associated their lab resources with these categories. In fact, 40.1 percent of U.S. respondents reported their lab resources were "excellent." While non-U.S. faculty may have a higher regard for online materials, their responses show non-U.S. students have less access to the resources required to consume online materials.

Both U.S. and non-U.S. faculty note a significant increase in the effort required to incorporate technology into their courses. While the responses between U.S. and non-U.S. faculty were not statistically different, the increase itself is significant. Over 25 percent of respondents from both groups reported a value of nine or ten (labeled "increased dramatically") when asked if their efforts had increased.

Both groups were asked their views of the academic value of developing online materials as it affects the promotion and tenure decision. The respondents' believed there was more value in development of online materials compared to the beliefs of their colleagues, chairs, promotion/tenure committees, or administrators. The U.S. and non-U.S. responses were not statistically different. However, in one respect the non-U.S. respondents were more optimistic than their U.S. counterparts.

The scale of choice was from zero (“no academic value”) to ten (“critical”). Looking at the responses of the faculty members' views across all values from zero to ten, there was little difference. However, we can estimate the momentum of the views by comparing the optimistic responses (the value is critical) to the pessimistic values (no value to develop online materials). Differencing the percentage of respondents reporting a value of nine or ten versus respondents reporting zero or one will provide a measure of the optimism of the respondents.

The difference of optimistic (nine or ten reported) compared to pessimistic (zero or one) was 15.2 percent compared to 2.2 percent for non-U.S. faculty. U.S. faculty had a 10.8 percent optimistic view versus an 8.2 percent pessimistic view. Non-U.S. faculty is clearly more optimistic in the view of developing online materials as an academically valuable effort.

The Similarities Are Evident

While there are differences between non-U.S. faculty and their US counterparts, there is much common ground. When it comes to believing in the value of developing online course materials, both groups agree there is academic value. Both groups also respond that their colleagues, chairs, and administrators attribute less value to developing online materials.

It appears that non-U.S. faculty place higher academic value on developing online materials. They feel more strongly in the value of developing materials to help them achieve promotion and tenure. And while both groups rate the efficacy of online course materials higher compared to traditional face-to-face teaching, non-U.S. faculty believes more strongly that online course materials are more effective.

These results are surprising because non-U.S. faculty reported that their students have less access to the technology needed to utilize online materials. Personal ownership of computers, as well as the access to computer lab resources, is lower for non-U.S. respondents.

The survey did not contain questions that would explain the oddity of non-U.S. faculty believing more in the value of online course materials even though they have less resources. Perhaps that is the answer itself: Those less affluent with resources tend to regard their worth higher.

The bottom line is that both U.S. and non-U.S. faculty believe in the academic value of developing online course materials – Non-U.S. faculty even more than their counterparts in the U.S. As the utilization of Web resources shifts towards greater and greater non-U.S. users, we can only expect that the development and use of online course materials will grow.

References

1. Alvi, M., Yoo, Y., and Vogel, D. (2000). Using Information Technology to Add Value to Management Education. Academy of Management Journal, December 1997, 1310-1333.

2. Benbunan-Fitch, R. (2002). Asynchronous Collaboration Around Multimedia Applied to On-demand Education. Journal of MIS, Spring 2002, 117-145.

3. Hiltz, S. and Turoff, M. (2002). What Makes Learning Networks Effective? Communications of the ACM, April 2002, 56-59.

4. Schell, G. (2004). Universities Marginalize Online Courses. Communications of the ACM, July 2004, 53-56.