Measuring success
raid on Deerfield revisited

Can a website influence people's opinions of a controversial event? Can it change perspectives and increase tolerance for different points of view? We think the answer is "yes." But how can success of this kind be measured?

In May 2004, eLearn Magazine published "Raid on Deerfield: Telling an Old Story in a New Way" an article describing how a history website tells a story from the perspectives of the five different cultural groups that were involved in the 1704 raid on Deerfield-the pivotal event in the town's history, memorialized and reinterpreted by the Pocumtuck Valley Memorial Association (PVMA). The findings of a summative evaluation report prepared for the Institute of Museum & Library Services (IMLS) suggest that hearts and minds indeed can be changed by a website.

In a detailed case study called "Evaluation for Effective Web Communication: an Australian Example," authors Patricia Gillard and Anne Cranny-Francis state that most website evaluations narrowly focus on an outdated model of usability testing (assigning defined tasks and scenarios) and fail to focus on visitors' experience with the website, the meaning they make of its content, and how they engage with it. Keeping this criticism in mind, we launched an evaluation in early spring of 2005 and completed it by November 30, 2005. We sought to assess the degree to which the website achieved these goals:

• Reach a large and diverse audience
• Increase visitors' knowledge of the event
• Contribute to a greater awareness of, understanding of, and/or appreciation for, the various perspectives that defined the historical event
• Help other institutions develop their own multi-perspective websites

To make this assessment, we used both quantitative and qualitative measures. Quantitative methodologies included Web server statistics that measured how the site was used for 21 months, an instrument for measuring the number of different cultural perspectives that were visited in any given session (in order to evaluate the extent to which users actually explore multiple perspectives), and an online survey. Qualitative methodologies included focus groups, an online survey, and an online game that yielded two months' of data about respondents' understanding of multiple perspectives.

This article, based on statistics from thousands of visitors to the site and questionnaire feedback from more than 100 visitors who responded to our inquiries, focuses on goals two and three: The impact of the website on visitors' knowledge of the event and the various perspectives surrounding it, and on visitors' own perspectives about the controversial topic.

Multiple Perspective Exposure

In developing a website that depicts a controversial subject from a variety of perspectives, we wondered if visitors to the site would be interested in exploring each of the perspectives, or if they would be partial to just one or two perspectives. Would they be partial to the European viewpoint that has predominated for hundreds of years, or would they explore competing views? Although we do not know the backgrounds of our visitors, we do know something about their behavior on the site:

• Visitors to the attack scene viewed each culture's tab in roughly equal numbers.
• Most of the visitors who viewed each of the non-English cultural tabs, also viewed the English tab: 1,211 of the 1,367 who viewed the French tab viewed the English tab; 1,104 of the 1,272 who viewed the Kanienkehaka tab also viewed the English tab; 997 of the 1,174 who viewed the Wendat tab also viewed the English tab; 942 of the 1,068 who viewed the Wobanakiak tab also viewed the English tab.
• 866 visitors viewed all of the cultural tabs.

This data suggests that many of the visitors to the website were indeed open to viewing multiple perspectives. But did they learn anything about the event and its competing viewpoints, and did they change in any way as a result of their experience?

Knowledge Impact

Both quantitative and qualitative data from the viewer survey suggest that the website had a significant impact on visitor knowledge. Fully 82 percent who responded to the survey said they learned new information. Respondents judged their knowledge of the raid before visiting the website as a 4.8 on a 7-point scale, with 1indicating great knowledge and 7 indicating no knowledge at all; they judged their knowledge of the raid after visiting the website as a 2.61.

Comments made in the survey also indicate that visitors believed they gained knowledge, as indicated by these comments:

I learned…"a lot about the complexities of this issue; for most of us, this was our first encounter with the 1704 story."

"I learned an important lesson of culture and human existence."

Some visitors learned an important historical lesson-that local historical events are windows into larger issues, both in the past and up to the present day, as these comments suggest:

"I knew nothing about this conflict on the local level. I had no idea that such an event could resonate into the present day."

"I feel that young people can get deeply involved in history while being challenged to make contemporary connections."

Multiple Perspective Impact

Both quantitative and qualitative data from the viewer survey suggest that the website had a significant impact on visitor perspectives, as well as knowledge; 53 percent of respondents said that they felt more strongly about something after visiting the website. While 45 percent said they left the website more interested in the subject than when they first came to it, 60 percent said they gained a better understanding of other people's ideas. When queried about the single most important thing that they will remember about the website, the most common responses indicated the multiple perspectives (and non-judgmental nature of the site), the informative content (reasons for the raid, personal stories, struggles), and the multimedia aspects of the site (graphics, color, animation, audio, etc.).

In rating their awareness of different points of view both before and after visiting the website, respondents judged their awareness before visiting the website as a 4.96 on a 7-point scale, with 1 indicating great awareness, and 7 indicating no awareness at all; they judged their awareness after visiting the website as a 2.53. The survey also showed that 45% developed an increased interest in something they knew little about before visiting the website, and 50% said their feelings and emotions were engaged by parts of the website. Finally, 19% said they better understand the community in which they live, and 18% said they learned new things about themselves.

Erased Stereotypes

When asked what they learned from the website, we can see that some stereotypes were erased, as indicated by these comments:

"I did not know that there were French people with the Indians in the raid. I had only heard the 'savage' Indians had attacked."

"I didn't know slaves and such were so involved in the Deerfield massacre, and I, being African-American, found it very interesting."

Most important, and tied to our primary objective of developing sensitivity to multiple points of view, we received comments that suggest that while the website does indeed promote an appreciation of multiple viewpoints among visitors, it also helps teachers use it for that purpose:

"Your site promotes open minds."

"As an educator, I feel more strongly than ever the need to present information about an event from more than one perspective…I wish more Americans could see your website."

Unchanged Minds

A very few comments indicate that even with exposure to the 1704 website, some people's views are not going to change:

"My heritage is English settler. No matter how sympathetic I might be for the plight of the Indian, my people and their situation still come first and it is very difficult to see, considering everything, how the settlement of the "New world" could have been much different."

"This website, which pretends to give multiple justifications for genocide is unfortunate. I believe it part of the effort by liberals to give us a moral lesson on an historical event to justify diversity issues on the present."

"Please stop this insanity to rewrite history. Have you never heard that the victor writes the history books?"

New Directions

We are gratified that the Raid on Deerfield website has had a positive impact on its visitors, increasing their awareness of multiple points of view, and in some cases, changing opinions and erasing stereotypes. In an article written for Perspectives in the year 2000, Dr. Eric Foner, then president of the American Historical Association, spoke to the continuing need for museums and historical societies to do a better job of presenting diverse perspectives of historic events. We know that museum exhibits can play a powerful role in defining a group's identity and history, especially when the groups represented have a strong voice in how their experiences are depicted. And online museum exhibits have an advantage in this, making it possible for a visitor to move rapidly and effortlessly from one perspective to the other, appreciating the points of similarity and difference, without being required to remember one before learning the next.

We believe Raid on Deerfield pioneered a new and effective online format for presenting multiple points of view. We supported this format with new scholarship and interpretation as well as character narratives and essays based on new original research. We told in-depth stories of people's lives and described the African-American and women's experiences-as well as the complexities of religion, and local and worldwide economies in Colonial New England-not only through character narratives but essays, historical scenes, and artifact descriptions.

One of the evaluation questions asked "Is the website successful in helping other institutions develop their own multi-perspective website?" We have some evidence that the 1704 website is serving as a model to other organizations, helping them work toward multi-perspectives in their own website development. For example:

From a Mississippi museum: "Wow! This site is incredible! Congratulations on a job well done! Any chance I can get a copy of your budget for this project? I'm trying to do community histories -not all of them are relevant to blues history but focus more on civil rights, agricultural history etc. I'd like to see how much a project of your size cost-and how long it took you to complete it."

From a college History Department: "I attended the American Association for State and Local History conference this past week where Lynne Spichiger presented the terrific Deerfield Web project. I am working with museum partners on a Web exhibit as well. The project management system which she shared with us looks like it would answer one of our largest problems-how to manage the pieces of the project and make these available for collaborative work by a project team dispersed geographically."

Perhaps the best description of what we have tried to do in the Raid on Deerfield website is embodied in a quote which ends the Introduction to the website. It's a portion of the essay "Who Owns History," which was written by Barry O"Connell at Amherst College, and which appears in its entirety on the website:

The end to be sought is not to get something 'absolutely right,' but to make it come alive in all of its uncertainties.

The more we can multiply perspectives from many different kinds of people, the better able we are to ask useful and specific questions out of which can come the fullest sense both of what did happen in the past, and how we might understand and judge it.

…It is our task, as students and teachers, writers and citizens, to bring everyone and everything out of the mist, so we might hear their voices, follow their actions, and respect each person, past and present, as a maker, as well as a subject, of history.

The Raid on Deerfield website was funded by both the Institute of Museum & Library Services (IMLS) and the National Endowment for the Humanities (NEH). The summary of information presented in this article is based on data collected and reported to IMLS in an evaluation report, which is available to the general public. Because IMLS named the website as a model for other organizations, its programming is also available to any non-profit organization that wants to use it. If interested, please contact Lynne Spichiger at the email addresses below.