What Do Training Awards Tell Us?

People around the globe love ratings, from "Car of the Year—Slovakia" and "Los Grandes De Nuevo Mexico Music Awards" to Cook's Illustrated's ratings of catsup.

Corporate trainers are no exception. We pay attention to awards contests for training products and often make choices based on them. The contests can serve a valuable function. They help make sense of the sprawling training industry, which is awash in products but has few well-established authorities to guide buyers' choices.

Contests are also popular with companies that run them. The reason is simple: they make money.

Spurned Award
Take a recent contest to identify the most influential individuals in training. Anyone could vote on the web, and the controls were so lax that companies could stuff the electronic ballot box for their CEO.

It wasn't a surprise that some of the individuals who made the list worked for companies that are paying sponsors of the organization running the contest. An award is an effective way to ensure a sponsor's continued loyalty.

I am not saying the people who won weren't worthy. I'm saying the contest wasn't worthy of them. There was no transparency, and the results had the appearance of a conflict of interest. In fact, one of the more prominent individuals the contest recognized refused the award.

Annual Product Awards
The corporate training industry has a variety of product award contests and events, many focused on e-learning. Understandably, people tend to pay attention to the results, not the process that led to the results. When you do take a close look at the evaluation process, you encounter red flags that make you wonder how credible the competitions are.

Vendors usually pay a fee for each product they submit. Because award competitions attract attention, vendors feel pressure to participate. To enter multiple products, a single vendor can easily run up a tab of thousands of dollars. With many vendors participating, the entry fees alone can make the contests lucrative for the sponsoring company. You may notice that contests tend to have quite a few categories. These may correspond to well-defined uses of technology, but they also invite more submissions and thus more money for the contest vendor.

It's an irresistible business model; there's only one catch. If vendors pay a significant sum of money to enter, they tend to expect something in return. The contest vendor has to walk a fine line between keeping one set of customers (vendors) happy while not alienating the other set of customers (training professionals). Contests don't disclose the number of submissions (a trade secret?) so you can't determine how selective the awards are. It does seem, though, that contests are generous with the number of products they recognize. That may be the implicit quid pro quo between contest vendors and product vendors.

To have a profitable contest, you have to keep overhead low. The biggest potential cost is judging. If you are going to run a competition with many entrants, you're going to need a large number of judges. The preferred solution is volunteer judges who do the work for free.

There is no perfect solution, but the downside of volunteers is that you have no idea how competent they are or how uniform their application of the contest criteria is. Those uncertainties plus annual turnover in judges raise the question of how comparable the contest results are year to year.

In addition, the evaluation criteria can be very broad because they have to be to cover all the different types of products in a contest. Judges are given wide discretion in interpreting the criteria so inevitably you're going to get differences in what they think the standards mean. That would be less of a problem if the decisions were explained. I haven't seen a training industry contest in which the judges explain in any detail why they chose a product for an award.

And I have yet to see a critical word about winners. Every technology product involves tradeoffs and compromises that suit some users but not others. You don't get that kind of information from contests.

Another judging model is for the contest vendor to use in-house people. This could be an excellent solution, assuming that employees have the requisite expertise. It also seems a better bet to ensure year-to-year consistency. You're going to pay the employees anyway so there's only an opportunity cost to worry about.

One well-known competition uses this model. The red flag is that the company does consulting for companies in the industry, and there's no Chinese wall between the consultants and the contest judges. They are the same people. In a recent year, six out of 10 award-winning companies also appeared on a list of consulting clients of the contest vendor. In another category over 50 percent of the winners were clients. Note: the publicly available client list was partial; it's possible that some of the winning companies are clients but aren't listed.

A third approach is to dispense with judges altogether and run the contest as a poll. In this approach, the contest sponsor has no role in deciding the winners. The weakness, of course, is how representative voters are of the population relevant to the product being voted on. There all kinds of bias possible in a pool of self-selecting voters. Is it more likely that people who love a product will take the trouble to vote than people who aren't crazy about it? Do you know if voters have much hands-on experience with a product—or none at all?

What Is Actually Assessed
Have you ever wondered how contest judges evaluate a large number of complex e-learning products in a short period of time?

In the case of one leading contest, the answer is that judges don't evaluate the actual product. The contest rules state that vendors submitting products can furnish a recorded demo, document, PowerPoint deck, or a combination to judges. In other words, the contest is actually a marketing competition!

To be fair judges can test drive trial versions of an application. Still, trial versions often disable key features. It also doesn't seem quite fair to have some judges using trial software for one product while other judges view a slick Flash demo of another. For this reason, if I were a vendor I wouldn't take a chance on evaluators drawing their own conclusions—I would submit a marketing presentation that furnishes conclusions for them.

You also have to wonder how many unpaid judges with busy schedules take the marketing pitches as the path of least resistance. I'm not criticizing judges who do that because I can see myself taking that option.

Worthwhile Results?
I don't begrudge people making a buck, and many product contests make a lot of them, year in and year out. I'm not charging contest operators with running rigged competitions.

I've heard vendors complain in private that they feel compelled to participate in some high-profile awards programs even though they're queasy about the process and are well aware of the conflicts of interest. I sympathize with them, but they do have the option of not entering contests they're uneasy about. If they didn't enter one year, would they begin to lose sales? I'm skeptical that the contests have such power in the market.

Some training product contests are poorly designed and allow potential conflicts of interest and bias. The training profession deserves better. Training contests provide information that might be valuable or might not be. The trouble is that you and I have no way of determining the value. Transparency is a well-traveled word these days, but it is what training competitions need.

Models exist that could make them more useful. CINE, a nonprofit film and TV trade group with a professional staff, has been giving out awards for more than 50 years. It relies on volunteer panels that look at specific types of productions, and there is a two-level process to guard against bias and poor judgment.

The key to change is the training profession. As long as we're uncritical about awards, the status quo will prevail because contest vendors have no incentive to change. If we don't demand transparency, then we probably aren't going to get it.