A Practitioner's Dilemma: How can I calculate the value of communities of practice?

By Julia Storberg-Walker / May 2012

Print Email
Comments Instapaper

On these pages of eLearn Magazine, many authors have written about the challenges of calculating the return on investment in technology and human capital development. For example, Dan Kossman suggested the phrase "return on value" as one way to calculate the added value of learning or training. Kossman wrote, "ROV broadens the analysis of ROI to include financial costs and hard returns as well as intangible benefits like having a scalable business, or increasing employee competence and customer satisfaction." Another author, Patrick Lambe, argued the economics of eLearning was different from the economics of typical training; consequently, how one assesses the value of elearning will be different. Lambe suggested "…metrics such as 'contingent valuation' and 'outcomes-based evaluation' are increasingly used for measuring the impact of other intangible, knowledge-based interventions, such as public library services or knowledge-based ecological systems." These authors, and others like them, wrestle with the difficulties of understanding how much input (in terms of dollars) produces how much output (in terms of value add to the organization). We all intuitively know that investing in certain employee development interventions generate positive returns of the organization, but proving the return can be difficult.

This struggle—to convince CEOs that investing in "soft" skills or that developing and maintaining online communities generates positive returns—continues. In 2011 the father of communities of practice (CoPs), Etienne Wenger, along with colleagues Bev Trayner and Martin de Laat, jumped in the fray to weigh in on the issue and published "Promoting and Assessing Value Creation in Networks: A conceptual framework." This article translated Kirkpatrick's evaluative taxonomy to the online community environment, and offered a framework for connecting community outcomes (for example, meeting a new contact) to positive returns (for example, a new sale is made as a result of the new contact) for an organization. Wenger et al's ideas are still in the percolating stage, but I believe they are on the right track. And, I know they are guided by the demand from practitioners to help them "prove" that communities do indeed contribute to the organizational bottom line.

I spent a week last summer at Etienne and Bev's home in the foothills of the Sierra Mountains, at an event they call Betreat. With me were community of practice leaders from around the world, and they represented a diverse variety of organizations—from big oil to non-profit, from professional association to volunteer organization. What we all had in common was the need to understand more about how to measure the value of CoPs, and I think many of us were blown away by what we learned during that one intense week-long conversation.

We learned it is possible to imagine a framework that illuminates the journey from creating and nurturing a CoP, to evaluating and demonstrating the value of the CoP to the organization. The framework can be downloaded at Etienne and Bev's website, which is chock full of interesting and current information about communities of practice. There are also links to relevant blogs and websites, for example, Nancy White's blog. Nancy calls the framework "lovely" and she has created a remarkable six-plus minute video that you will want to see. Not only does she provide a primer on CoPs for the layperson, she brings out the real value of the framework itself.

Let's first look at how the framework is designed. It is built from the premise that there are three pieces of a CoP:

  1. domain (WHAT is the CoP about);
  2. community (WHO is sharing the domain); and
  3. practice (what the community DOES together).

These three pieces combine to define a CoP, and the evaluation framework is able to assess the value of each piece-if you do it right. Like all evaluation models, it is recommended that when you initially start to design your CoP you want to take into consideration how to evaluate it. How often have you been asked to design and deliver a program, and tagged evaluation at the end in a rush because of time or resource constraints? You know you don't get the maximum benefit…likewise, if you don't consider how to evaluate the ROI of a CoP when you design the community, you won't have the maximum benefit that the framework can provide.

Figure 1. Wenger et al's Framework for Assessing Value in CoPs.
Wenger's Framework
Figure from Wenger, E. and Trayner, B. used with permission.

The framework itself has five levels-four that were adapted from Kirkpatrick's model, and one that was added on top of pile (see Figure 1). The first level—related to the satisfaction level—is called "immediate value" and it assesses what just happened, for example, in a webinar. Did community members like it? Learn from it? Enjoy the facilitator? The second level is called "potential value," and I like to think of this as the new knowledge or understanding that is lying latent but ready to be put to use in the future. This level looks at change: Did the webinar change participants? Did it change how they related to each other? Did it provide them with new resources? Questions like these point to potential use, but do not measure actual use. The third level does this, and it is called "applied value" and this is where the model starts to become interesting to CEOs and others who sponsor CoPs. The applied value level captures evidence about how CoP members actually used the new tool, or collaborated with a new colleague, or applied the new knowledge/skill in service of the job.

You're saying yes, but, I'm still not getting hard metrics like reduced development time, improved efficiencies, or financial returns. The fourth level in the framework provides this, and the level is called "realized value." This is like Kirkpatrick's fourth level—it measures the difference made by the CoP in the work of the organization. Did the use of the tool (as measured in level three) produce an increase in efficiency? New sales? Reduction in cost? This level will be the most helpful because it produces evidence that the CoP is (or is not) producing a benefit—or value—to the organization.

The beauty of the first four levels is that they illuminate the path from attending a webinar, to learning a new skill, to applying the skill, and to measuring the change as a result of the skill. Each step along the way can be captured and measured by using the framework, and each step can be easily understood and described to decision makers. There is a huge amount of face validity to the framework, which helps decision makers to diminish their natural resistance to evaluation schemes that rely on inference and conjecture. In this framework, it is easy to see and accept the value creation stream that a CoP can create.

The fifth and final level may or may not be of interest to decision makers, but the level will be important to CoP members who care about the community. The fifth level is where the community changes as a result of the activity occurring in the first four levels. At this highest level, the framework examines changes in the community—norms, standards, practices, and thought leadership—that has occurred as a result of activity within the community. While there may not be an immediate financial reward connected to this level, the changes in the community will likely impact all other levels and change what members learn, do, and practice.

Combined, the five-level framework is a huge step forward in our attempts to evaluate, measure, and understand the impact that CoPs can have on people and organizations. Go visit Etienne and Bev's website, read the article, and watch Nancy's short video for more information. I believe you will be able to take away great ideas for how to measure your own community of practice.

About the Author

Julia Storberg-Walker served in a variety of positions at Deloitte & Touche, LLP and Deloitte Consulting from 1985-1999. After working in the West Coast offices of Los Angeles and Seattle, she assumed leadership positions in Deloitte's National Office, and then in Deloitte Consulting's Global headquarters. In 1999 she returned to school and obtained her Ph.D. in human resource development in 2004. She is currently an Associate Professor at North Carolina State University, an award-winning researcher, and was recently inducted to the Academy of Outstanding Faculty Engaged in Extension for her consulting work with business, non-profit, and public organizations.

References

Wenger, E., Trayner, B., and de Laat, M. (2011). Promoting and assessing value creation in communities and networks: A conceptual framework. The Netherlands: Ruud de Moor Centrum.

© 2012 ACM 1535-394X/12/05 $15.00

DOI: 10.1145/2207270.2222516



Comments

  • There are no comments at this time.