ACM Logo  An ACM Publication  |  CONTRIBUTE  |  FOLLOW    

Chasing Down the Elusive Credits for Facts and Fictions in Learning and Improvement

By Brett Christensen, Guy W. Wallace / February 2012

Print Email
Comments Instapaper

Not everyone in the performance improvement field is a formal researcher and able to cite the research off the top of their head. But all should know the major "takeaways" of the research. All should know which of the well-established "myths" of our profession are most prevalent so that they and their clients can avoid them.

But it is very tricky to keep up with the facts versus fiction. And sometimes we are hoisted on our own petards—for "our crowds" sometimes provide us with both wisdom and folklore that gets repeated and repeated and repeated until we all believe it—except those hard-headed skeptics who seem to believe nothing and demand to see the proof.

Such is the case with variations of: Only 20 percent of performance issues are rooted in the individual versus the system (or environment).

Science or Snake Oil

Recently the authors were involved in an Internet exchange across several social media channels in an attempt to sort the wheat from the chaff of a rather famous (or infamous) statistical claim, which some believe came from Gilbert, others claim Rummler and Brache, or Mager, or Deming, and some credit/blame Pareto.

Some variations of this claim include:

"Confusion between common causes and special causes leads to frustration of everyone and leads to greater variability and to higher costs, exactly contrary to what is needed. I should estimate that in my experience most troubles and most possibilities for improvement add up to proportions something like this:

  • 94 percent belong to the system (responsibility of management);
  • 6 percent special.

…Good management and good supervision require knowledge of the calculations that will separate the two kinds of cause" [1].

"…we have found that about 80 percent of performance improvement opportunities reside in the environment. Usually 15 to 20 percent of the opportunities are in the Skills and Knowledge area" [2]. Roger Chevalier cites this example specifically in his book A Manager's Guide To Improving Workplace Performance [3].

Robert Mager said "…clues that indicate a situation that might be described as 'plenty of skill but not enough will.' You can be pretty sure that to influence the individual to perform as desired, you must change the environment around the performer in some way rather than try to teach new skills" [4].

The Pareto Principle also referred to as "The rule of the vital few and the trivial many" or the "20/80 rule" has no direct correlation to this discussion and would only serve to further muddy the issue.

You've undoubtedly heard this all before. And you may have heard it for so long that you now take it is a fact and a basis for moving forward in your efforts with due consideration of this fact. It may be something you have even quoted to clients and colleagues when seemingly appropriate.

We, the authors and our helpers (Dale, Jeanne, Dick, and Rich) are from the HPT (human performance technology) school of thought. The 2004 International Society of Performance Improvement (ISPI) Task Force's Report to the Board defined HPT as "an integrated systems approach to improving human performance." They went on to develop a set of four criteria to judge applications of HPT as follows [5]:

  1. Is focused on valuable, measured results;
  2. Considers the larger system context of people's performance;
  3. Provides valid and reliable measures of the effectiveness of those applications; and
  4. Clearly describes applications grounded in prior research or empirical evidence (or are not discouraged by either one) so that they may be replicated under the conditions and by the means for which they were recommended.*

*When stated this way, intuition and respected practice are permitted and encouraged (provided they meet the first three criteria) without scientific evidence provided that there is no research evidence that it may not work under the conditions or by the means where it is being recommended.

So, where is the evidence? Or does the 80/20 claim fall within the "respected practice arena" described above?

Who Said It First?

When the first call went out on LinkedIn in the ISPI Global Group for the proper attribution it got a few immediate responses. The request was worded thusly:

I'm looking for any cites quoting Gilbert's claim that 80% of performance issues are found in the environment, not the individual; especially interested in where the claim was first published. Thanks!

Then the question and search socially hopped, skipped, and jumped over to Facebook:

To all my HPT Friends: Do you have a source for the statement (loosely worded) that 80% of performance problems are not related to training?

Then it was taken right into the lion's den of a small Google Group of mostly long-time ISPI leaders—80 or so folks who should know this quote and its source, as it is a phrase that has been bandied about within the physical and virtual halls of The Society for Performance Improvement since they were the halls of Performance and Improvement—and perhaps when the they were the halls since the last days of Programmed Instruction.

Neither of us were there "back in those early days" to recall that clarion call being part of the scene—but one could imagine it so.

The question posed to them, after Fred Nichols untwisted the twisted wording of the first:

Where did the statement "80% of performance gaps are caused by other than Knowledge/skill deficits" come from?

Where's Your Data? And Data is Plural!

This one-two punch was, back in the day, a popular taunt and reminder from the audience at NSPI, then ISPI, conferences. Those in the know knew to be prepared for it. Newbies to The Society were often caught off-guard and upon their return the next year—if they dared—knew to never be in that vulnerable position again of not having their data ducks in a row.

Then in an attempt to respond to conference goer feedback, the leaders of The Society attempted to make the place friendlier—and such overt challenges were discouraged. We seemed to be a prickly place—because we were—but placing "accurate" over "friendly" was no way to grow a Society.

Or was it?

Some would say ISPI got friendlier and more accepting and acceptable—and accessible to others. Some would say we got soft on facts and encouraged fiction and invited foo foo. That debate continues.

The Internet inquiries netted numerous responses. Contributors from the ISPI Global Group on LinkedIn offered Geary Rummler claimed that 90 percent of performance problems are tied to the environment not the performer, while W. Edward Deming asserted 80 percent of the problems relate to the system, not the people. Respondents also noted that while the claims are similar, they are also significantly different. The "system" is only one component of the environment and thus presents a narrow view of the problem. Dr. Harold Stolovitch offered an answer to a reader of his website with the same question, pointing him to Gilbert, Pershing, and Rummler and Brache as a research starting point on the factors that influence how people perform [6]. Dr. Stolovitch said what emerges from these references is a consensus that around 75-80 percent of the factors that influence performance are environmentally rather than individually based.

Dr. Jim Hill shared that his organization has conducted approximately 4,000 surveys in the past eight years that use Gilbert's behavior engineering model (BEM) as the framework to categorize primary (root) cause issues. Their research shows rather than a single figure being assigned to each BEM category, there is a range. He explained "For example, we've found that Gilbert's 'Data' (which we call 'expectations and feedback') primarily accounts for 28-35 percent of all performance issues depending on various respondent characteristics. It's also important to understand that while we might say it's a-c percent of data, d-g percent for resources, etc., complex performance issues typically have multiple cause(s) requiring multiple solutions (or a solution 'set' a la Rummler and others)."

In the Facebook discussion thread, Dr. Roger Chevalier pointed the readers to Rummler and Brache and W. Edward Deming. Dr. Ryan Watkins offered similar information in the ISPI Google Group, indicating that he could not find the research Deming used to develop this theory but surmised that he must have based it on some type of statistical analysis. He also pointed out that it is important to remember Deming was only referring to skill, not knowledge. Jim Hill, active in this group as well, expanded on Watkins caution, advising, "We need to be careful with our characterizations of common and special cause. As Ryan notes, Deming apparently *estimated* that 94 percent of problems (or possibilities for improvement) lie within the system (see Out of the Crisis, MIT, 1982) and can be classified as common causes. Six percent are special causes (singular events—and not improvable). What 94/6 does not mean is systems versus people. Nor is 94/6 meant to indicate a breakout between 'non-training' and training factors."

Did we find the elusive empirical answer to our inquiry?

No.

We found a tangled morass of thoughts on quote variations and credits for them. It stumped our panel of stars in the closed ISPI Google Group. It bogged down with the other more open Internet crowds and their lack of wisdom on this shifted from "where did this come from and where is the evidence" to "I think…."

Back at the ISPI Google Group ranch the question/topic shifted as well, from "Where is the data?" to "What is the right question to be asking in the first place?"

Wisdom Emerges

Then the ISPI "gurus" of our crowd (the wisdom of our crowd because they are mostly evidenced-based) and their points about the irrelevance of the actual numbers—and the need to follow the science of HPT/ISD (instructional systems design)…

Participants in the different online conversations indicated in their own experiences, the numbers used by Deming and Rummler rang true. Then Fred Nickols turned the discussion in an entirely new direction when he wrote "One hundred percent of the solutions to problems of human performance will involve and make good use of training in one way or another," and continued on with "whether people are or are not at the root of a performance problem it is safe to say that they are always part of the solution."

Dr. Dale Brethower reinforced Nickols comments by pointing out that the question itself is unanswerable as the number of variables impacting performance and how each one influences the other, which Gilbert termed "the diffusion of effects" [7], is impossible to compute as the performance environment is one of continual change. Brethower proposed that a better question to consider is "What must be done to manage the variables influencing the performance gap we are concerned with?" Reminding us that these questions will be situation specific and as such, the information gained in each situation cannot be generalized for the same reason. Brethower's response brought many cheers from the group and led the authors (mostly the younger ones) to realize the importance of knowing how what we say is backed up (or not) by empirical data (science?).

Where's the Evidence?

The importance of understanding how what we say is backed up or not by science…

When first joining the field, the models, theories, and ideas that are presented come fast and furious and it is difficult to sort the wheat from the chaff. It is also challenging in the early days to identify the scientists amongst the snake oil salesmen. Depending where the HPTer is in their cultivation period as a critical thinker [8], the practitioner may not have the individual skills necessary to make these differentiations.

Early in the journey to become a performance improvement practitioner one of the authors was presented with "Dale's Cone of Experience"—and being new, impressionable and not a very critical thinker, absorbed it as gospel. A year or so later, while searching for the source of Dale's Cone for a colleague, he came across a blog entry by one of our very reputable scientists in the field, Dr. Will Thalheimer [9], who provided him with his first lesson in the importance of knowing the veracity of the information you use. The author was surprised and another of his colleagues was disappointed to learn that their initial training had presented them with a flawed model. The colleague had planned to use this information as the basis for a proposal to undergo a major training methodology change in her organization. She still made the proposal, but she did so with the understanding that the numbers associated with Dale's Cone weren't valid, raising her credibility.

Another colleague's question on the source of the 80/20 environment versus performer figure (or 85/15 or 96/4 as discussed above) sparked the author to go hunting again, raising the question in Facebook and then landing in the other ISPI groups, which was the impetus for this article.

Conclusion

If we accept Stolovitch's proposition that a consensus has emerged from the work of the most prevalent leaders in the field, then the statement "80 to 94 percent of performance problems are related to the environment" can be linked back first to Deming who said "In my experience…" and then Rummler and Brache who wrote "We have found that about…." We have not been able to locate any specific research to demonstrate that these claims are evidence based, only assumptions.

Of greater significance are the points raised by our peer reviewers:

  • Every performance problem is situation specific (Pearlstein);
  • Our task is to identify and manage the variables necessary to improve performance (Brethower);
  • It is almost certain that both workplace variables and performer variables will be involved so we should consider both categories (Brethower);
  • Solutions will not always involve training (since for example, motivation is a largely unexplored cause and a key component of the solution to many performance problems) it is essential that we put people in the forefront of all performance issues (Clark).

Dr. Farrington summed it up eloquently stating, "In the world of solution choices, it doesn't matter, really, what causes the highest average percentage of performance problems (if such an average number were really available to us outside of educated estimates). What matters is, for this particular issue that we are trying to fix, we are looking past pre-determined notions about what's causing said issue. We'll apply the set of solutions that get us to where we need to go—whether that's training, or not."

Acknowledgement

We would like to thank Dale Brethower, CPT, Ph.D., Jeanne Farrington, CPT, Ed.D., Richard E. Clark, CPT, Ed.D., and Richard B. (Rich) Pearlstein, Ph.D. for their review and guidance.

About the Authors

Guy Wallace has been an external performance improvement consultant since 1982 specializing in performance-based instruction, curriculum architecture design, ISD processes, qualification/certification performance testing, recruiting and selection criteria, and other non-instructional interventions. He is the recipient of ISPI's Honorary Life Member Award (2010) and an inaugural member of ASQ's Influential Voices to Raise the Voice of Quality (2010). He is the co-founder and current President of ISPI Charlotte, and has served as President of ISPI International (2003-2004). He has been a Certified Performance Technologist (CPT) since 2002, and served as a Director on the ISPI International Board (1999-2001). He has been a speaker at more than 35 conferences (since 1985), and has published more than 12 books and has been published more than 80 times in numerous professional publications since 1984. You can connect with him via his website or via email.

Brett Christensen has been in the learning and development field in the Canadian Forces since 1993, shifting to performance improvement in 2006. He is currently a Canadian Forces Warfare Center Training Development Officer. His most recent qualifications and achievements include the 2010 and 2011 Chapter of Merit for the Armed Forces Chapter (AFC) of ISPI, during which he was the chapter president. Christensen is a Certified Performance Technologist (CPT). He holds a Master of Science in instructional and performance technology from Boise State. His work has been published in PerformanceXpress and the Performance Improvement Journal. Christensen's specialties include: performance improvement, instructional design, eLearning, application of virtual worlds to training (Second Life), needs assessment, project management, learner assessment, instructional systems design, instructional development, and program evaluation.

References

1. Deming, W.E. Out of the Crisis. MIT, Cambridge, 1982.

2. Rummler, G.A., and Brache, A.P. Improving Performance: How to manage the white space on the organization chart. Jossey-Bass Publishers, San Francisco,1990.

3. Chevalier, R. A Manager's Guide To Improving Workplace Performance. AMACON Books, New York, 2007.

4. Mager, R.F. Preparing Instructional Objectives. Part 2. Pitman Management and Training, 1984.

5. ISPI Presidential Initiative Task Force "Clarifying HPT: The report to the board." 2004

6. Stolovith, A.H. (N.D.) "Ask Harold" [Web log comment] Retrieved from Ask Harold

7. Gilbert, T. F. Human Competence: Engineering Worthy Performance. Pfeiffer, 2007

8. Paul, R. and Elder, L. Critical thinking: Concepts and tools. 5th Ed. Foundation for Critical Thing Press, Tomales, CA, 2008.

9. Thalheimer, W. "People remember 10%, 20%...Oh Really?" [Web log comment]. May 1, 2006. Retrieved from Will at Work Learning

© 2012 ACM 1535-394X/12/02 $10.00

DOI: 10.1145/2129230.2134288



Comments

  • There are no comments at this time.