ACM Logo  An ACM Publication  |  CONTRIBUTE  |  FOLLOW    

User experience in the first ARISE distributed classroom

By Paul W. Smith, Kelly A. Lyons / March 2004

Print Email
Comments Instapaper

The following is a description of our experiences in providing a live distributed classroom in which students from an IBM software development lab and two university campuses participated in a graduate seminar course. Our experiences were part of an initiative called Advanced Research Initiative for Software Excellence (ARISE).

As the emphasis on lifelong learning and personal and professional development increases, spurred by the growing demand for skills in today's workforce, so does the need keep growing for innovative instructional delivery mechanisms. Studies show that the ability to upgrade skills and to take technical training surpasses salary in importance as a measure of job satisfaction among information technology (IT) workers. Highly skilled, highly educated software developers desire ways in which they can participate in leading-edge learning opportunities such as graduate-level activities at universities.

These motivations prompted the University of Toronto, the University of Waterloo, York University, the IBM Toronto Lab, and Canada's National Research Council's Institute for Information Technology to form a consortium. This initiative is ARISE. In this article, we present details of the first ARISE course, an advanced-level graduate course, offered to multiple sites using video-over-IP technology. We provide information about the user experience of those participating in the course (professor, university students, and industry students). We identify many lessons learned from this first ARISE course offering and provide a comprehensive set of goals and objectives for future courses.

A great opportunity exists in the partnering of industrial software research and development and academic software research expertise. The ARISE vision is to create an advanced software technology research and educational initiative that leverages synergy among universities, research labs, and industry with a focus on cross-disciplinary research and academic activities. ARISE aims to bring together university students, professors, researchers, industry developers, and users to study specific topics in software through one or more of the following coordinated activities:

  1. Advanced research projects on topics such as e-business, data management, software development tools, Web services, and e-learning
  2. Intensive three to five day workshops that bring together leading experts from around the world to work on aspects of a specific topic or problem
  3. Advanced-level courses delivered using distance-learning technology, such that courses may be attended and/or taught simultaneously from all partner sites

A research project is already under way. It focuses on enhancing the collaborative experience in distributed learning and distributed software development. It uses audio conferencing, voice technology, voice-to-text processing, and information visualization via large visual displays. Tools are being created for recording, visualizing and searching of synchronous voice interactions, and blending transcripts of these synchronous interactions with the products of asynchronous communications such as e-mail.

Overview and Motivation

Many employees at IBM want to participate in university activities while at work. In addition to participating in seminars at universities and in projects through the IBM Toronto Lab's Centre for Advanced Studies, some employees would like to formalize their involvement and receive a Master's degree with IBM's support while continuing to work for IBM. Within IBM Canada Ltd., there are two ways that employees can do this. The first way is by participating in the ConGESE (Consortium for Graduate Education in Software Engineering) program, in which professors come into IBM (and other companies) to teach the course part of the Master's degree. In the second way, employees participate in a part-time Master's program, taking (usually) one course per term at a nearby university. In the first case, the employees experience the convenience of taking their course at their workplace without having to travel to a university and back once per week; however, they miss out on the opportunity to interact with and learn from non-IBM employees during the course. (Similarly, the non-IBM university students taking the course could benefit from interacting with people from industry.)

Synchronous Learning

Many distributed learning practitioners, particularly academics, question whether the benefits of synchronous courses are worth the cost of struggling with the technology. These practitioners have focused on asynchronous forms of eLearning. Yet for many people in workplaces such as IBM, virtual meetings are a daily occurrence. Synchronous remote collaboration is the norm. Also, IBM's goal is to deliver 40 percent of employee education through distributed learning, even though much of current formal IBM employee education takes place in the classroom. As both corporations and institutions of higher learning are discovering, replacing the classroom with asynchronous education is not always effective. Synchronous distributed learning will be necessary in both the workplace and advanced education settings. ARISE is bringing together workplace and higher education through synchronous distributed graduate seminars. Because of their highly interactive nature, graduate seminars require a synchronous approach.

Course Mechanics

The first ARISE course was offered in the fall of 2002 in the area of Web-based Data Management. There were three sites, connected live via video-over-IP. There were students from the University of Waterloo, the University of Toronto, and the IBM Toronto Lab participating in the distributed graduate seminar course. Since the professor was from the University of Waterloo, this institution was considered the host site. The strategy was to use available technology and pedagogy with a goal to identify areas for improvement and research.

The format of the course consisted of seminars led by students from their home site, which were then followed by group discussion across sites. The presentations were made available in advance and there was an official discussant for each presentation. We will discuss the students' experiences as presenters and learners; the professor's experience; the interactions among students; the interactions between students and the professor; and the relative importance of audio, video, slides, and off-line discussion groups.

The effectiveness of our course delivery mechanisms were evaluated through observation, questionnaires, and interviews. We describe the lessons learned and provide recommendations for future uses of this format. Analysis of feedback from course participants and observation of the seminars points to issues that require research and to innovative approaches for advanced collaboration over distance, described below.

Technology

The goal of the Exploratory Stage (the first stage) of the ARISE initiative is to experiment with and explore the program using existing space and equipment in order to understand what kind of equipment and infrastructure is required prior to making any significant investments. The IBM Toronto Lab, the University of Toronto Multimedia Centre for Learning in the Humanities, and York University Instructional Technology Centre had existing Polycom equipment that is used for video conferencing and to deliver video and audio for some classes. The University of Waterloo had compatible equipment through The Centre for Learning and Teaching through Technology (LT3). For the actual course discussed in this article, students participated from three sites: the University of Toronto, the University of Waterloo, and the IBM Toronto Lab.

A design point of the configuration was to use the Internet protocol for communication of the video and audio as well as for sharing the slides. For the first part of the course, the University of Waterloo and the University of Toronto were on the ONet network and the IBM Toronto Lab was on its regular Internet connection. Each site connected to a multipoint conferencing unit (MCU) at the University of Waterloo LT3. The MCU took the three feeds and mixed them into a four-quadrant screen, with the camera view from each site in one of the quadrants, and sent that view back to each of the three sites. Because there were only three sites, one of the quadrants was blank.

In the second half of the course, all three sites were on the ONet network and each site connected to an Avaya MCU at the University of Toronto. Again, the four-quadrant view was created and sent to each of the sites.

The slides were shared using a laptop computer at each site that connected to a Web site that provided Web conferencing technology (http://www-125.ibm.com/team/emeetings/stcenter.nsf), powered by IBM Lotus Sametime. A Web conference provided shared visuals and text chat, and was used in conjunction with the video/audio stream. Any participant could present slides, demonstrate an application, or modify a document with input from other attendees. Anything that participants could view on their own screens could be shared with other attendees. It was easy to switch among participants and sites.

At times, issues and questions came up that were of interest to the entire class, or there were discussions taking place during the classes that needed to be cut-off because of time restrictions. For these situations, Webboard was used for asynchronous collaboration. It allowed the professor to communicate information to all students and provided a mechanism for students and the professor to communicate with one another outside their synchronous classroom participation.

For each site, one person was available to troubleshoot problems. At the University of Waterloo there was also a Teaching Assistant who helped set up the cameras and microphones, and the computer to share slides. At the University of Toronto and the IBM Toronto Lab, the camera and microphones were handled by the class participants or the moderator.

Physical Set-up

The distributed classroom consisted of three physical sites with a different number of students at each site: There were twenty students at the University of Waterloo, three students at the University of Toronto, and five students at the IBM Toronto Lab.

During a presentation, the presenter's site captured and sent video of the presenter. The other sites captured and sent video of their entire groups of students. During the discussion, every site captured and sent video of their groups of students. At the University of Toronto and the IBM Toronto Lab, switching between presenter and discussion mode was done by manually manipulating the one camera to focus on either the presenter or the group of students. At the University of Waterloo, there were two cameras: one was always focused on the presenter podium and the other was focused on the group of students. Switching between presenter and discussion mode at Waterloo was done by switching feeds between the two cameras.

The IBM site and the University of Toronto had a similar physical set up but differed in that the IBM site had two monitors: one showing what the local camera was capturing and sending, and the other showing the four-quadrant view that was sent by the MCU. At the University of Toronto, the single monitor displayed the four-quadrant view. At the University of Toronto and the IBM site, the slides were projected onto a screen beside the monitor, which was at the front of the room. At the University of Waterloo, the project slides and the presenter were at the front of the room and the projected four-quadrant view was at the back of the room. In this way, the presenter could see the local site as well as the two remote sites without turning around. When the discussion took place, the students at the University of Waterloo had to turn to view the projection of the remote sites.

The rooms used for this course are used for other purposes throughout the year, and were adapted with little changes for the purpose of this course. One of the goals of the ARISE initiative is to understand how the rooms should be designed to deliver these kinds of courses most effectively to distributed classrooms.

User Experience

Because this was a distributed seminar course with presentations originating from all three locations, all students were confronted with some remote experience. However, because the professor and 20 of the 28 students were in Waterloo, these students were significantly less affected by the remote experience. There was inevitably some sense of isolation or second-class status for the eight more remote students. These students stated: "I assumed Waterloo students had more access to the prof"; "I don't know if the professor had office hours"; and "I didn't have the feeling I was meeting the remote people in this course."

In this course, the professor did a number of things to make himself available and to encourage cross-site interaction during class and outside class. Thus, combating the sense of isolation may be more a matter of managing perception than of making substantive changes to the pedagogy. However, we do not want to solve the problem of feelings of isolation by making local students feel second-class. We need to find tools to help both local and remote participants. For example, Web site signup for office hours works well for both.

Presenters typically did not tailor their presentations to remote participants. It seems that when there is both a remote and local audience, people present to the local audience and pretty much ignore the remote audience. One presenter from the host site admitted, "When I presented, I did not worry about the remote audience."

Despite the value that could derive from collaborating with people from other institutions, only one of the 14 class projects was done by a cross-site team. Gary Olson and Judith Olson summarize this phenomenon well: "When people attempted to work closely with remote team members on difficult problems, over time the remote technologies were used less and less. Work was reorganized so that people did not have to rely on tight collaboration." ("Distance Matters," Human-Computer Interaction, Volume: 15 Number: 2 pp. 139 - 178, Lawrence Erlbaum, 2000.)The class size itself discouraged interaction, particularly at Waterloo. Therefore, the effects of a distributed classroom were compounded by the number of students involved. The general feeling was that class size was a bigger factor than the remhote component. One participant said: "No problem asking questions, but hard to debate." With multipoint video conferences such as ours, there are problems unique to "fat points" (or nodes that are disproportionately large) such as at Waterloo. Seeing and hearing one of 20 participants is difficult and tends to lead to reduced interaction. Audio difficulties were compounded by the presence of English-as-a-Second-Language-participants, exaggerating any difficulties in understanding what was being said.

Some students felt that they should hold their questions until polled by the professor. "We waited until the prof asked for questions from U of T...it was difficult to jump in to ask," said one student. Some students felt disinhibited as a result of being remote from the professor. They felt more anonymous and therefore there was less risk attached to asking questions. Some technologies are deliberately used to create distance. Instant messaging is sometimes used in this way when chosen instead of the telephone. Perhaps remote classrooms also have this characteristic.

Conclusions

Satisfaction/Performance Mismatches

Students indicated that there were three aspects of our distributed classroom that affected satisfaction, but not performance. First, video was considered desirable, but audio quality was considered essential. Second, the remote experience was considered less desirable than the face-to-face experience, but not an impediment to educational goals. (The familiar "no significant difference" effect.) Lastly, classroom-to-classroom communication was preferred to desktop-to-desktop contact only. Students emphasized that they preferred to gather in a classroom at their respective sites though there was no suggestion that this was more efficient.

Synchronous/Asynchronous Hybrids

Simply emulating face-to-face collaboration is an illusory and less-than-ambitious goal (cf. Hollan and Stornetta. "Beyond being there" in Proceedings of ACM Conference on Human Factors in Computing Systems [CHI'92], pp. 119-125, ACM Press, 1992). A better goal is to create tools that provide benefits that are typically not available in traditional classrooms. Synchronous/asynchronous hybrids are one way of accomplishing this. Accessing a Web-based forum during the synchronous component of the course is one example. Another example is persistent chat, where records of chats are available in an online archive.

If students brought laptops to the classroom, they could use them for public and private chat, to access Web forums, to control course slides independently, to annotate slides, and to customize the presentation. Of course, such integration can also be useful in a non-distributed course. One indicator of a good collaboration tool is that it is used when distance is not an issue, e-mail being the obvious example.

Recommendations to Support Participant Interaction

We have identified a number of potential improvements to future ARISE courses:

  • Limit each classroom to fewer than six students.
  • Try breakout groups as an alternative; they would generate more discussion. They would also reduce the effects of rooms with more than six participants.
  • Invest in community building. Exchange bios and pictures. Place student Web page addresses on the class Web site.
  • Encourage public and private chat pre-, during, and post-class.
  • Support informal interactions beyond chat, such as community events or unstructured conversation time by video conferencing.
  • Automate the audio and video tracking of speakers.
  • Add software that enables electronic hand-raising.
  • Support cross-site project collaboration. Include asynchronous and synchronous tools dedicated to project collaboration.
  • Provide office hours by means of teleconferencing or video conferencing.
  • Enable interactive viewing of archived sessions.
  • Make video images of remote people life size.
  • Encourage presenters to regularly poll remote participants (to create more interaction).

Opportunities for Further Research

A number of potential improvements to distributed classrooms require research to determine whether and how they should be implemented:

  • A universal complaint about collaboration tools is the difficulty of setup and configuration. Anything that could be done to reduce this frustration would be welcomed.
  • We need to know more about the cues people rely on in face-to-face gatherings to engage in rich interaction. Is the impediment to debate technology or pedagogy?
  • Is private chat too disruptive for use as a community-building tool?
  • There will be problems specific to distributed classrooms whenever there are both local and remote students. Currently, presenters focus on their local audience exclusively…what is the right balance?
  • What is the best way to split fat points? Should a site of 20 be structured as 4four points of 5five each? Is it more effective to use individual Web cams within the classroom setting?
  • Remote technical support would be a great boon to any collaboration-at-a-distance situation.
  • Automated recording of sessions is becoming more common. How do students use such recordings?
The views expressed in this document are those of the authors and not necessarily of IBM.

Trademarks

IBM, Lotus, and Sametime are registered trademarks of International Business Machines Corporation in the United States, other countries, or both.

Other company, product, and service names may be trademarks or service marks of others.



Comments

  • There are no comments at this time.