ACM Logo  An ACM Publication  |  CONTRIBUTE  |  FOLLOW    

the only way to know if a particular LCMS is right for you is to test it yourself

By Michael Feldstein / May 2002

Print Email
Comments Instapaper

Some months back, I had the great misfortune of having to work with a learning content management system (LCMS) that will go unnamed here. This product has all the right bullet-points in its marketing brochures. It is backed by impressive-sounding case studies. It has received rave reviews from many of the big-name gurus in the business. (On the home page of their web site, the makers of this product have " 'Best LCMSs Short List'" emblazoned in big letters near the top of the page.) Despite all this, I found it to be completely unusable. Many of the "features" listed in the marketing were unusable because they were undocumented or because they were designed so poorly that they caused more problems than they solved. Even worse, in some cases the marketing claims were just plain false.

In the end, the system did not make any of our work easier and, in some cases, actually made the work much harder. Between the time we spent hunting down information about how the LCMS worked (or, as it turned out, didn't work) and the time we spent trying to figure out ways to get around limitations that we wouldn't have had to put up with had been using more traditional tools, we ended up running substantially behind schedule and over budget for the project. The limitations proved so insurmountable that we eventually had to recommend to the client that we not use this very expensive piece of software he had recently paid for.

Caveat Emptor…But How?

Out of curiosity, I did a Google search for reviews of the product. I wanted to see which reviewers had seen the flaws in it and which ones were taken in by the glossy marketing brochures and whizzy vendor demos. Sadly, I was not able to find a single review that called attention to the problems I had encountered—problems that seem like they should be glaringly obvious to anyone who tried to actually use it to solve a real training problem. If you are in the market for an expensive LCMS, there is apparently no place you can go to get thorough and honest product reviews based on extensive hands-on testing.

If this news shocks you, then you have probably never been involved with the purchase of an expensive server-side IT system before. Take, for example, the challenges of purchasing a content management system (CMS), the less complicated forbear of the LCMS. Here's how author David Walker describes the situation:

"As Internet technology advances, new things become suddenly possible. Take content management. Just five years ago, it was almost impossible to waste a million dollars building a web site. But modern, Twenty-first Century Internet technology means that any medium-sized organization with Web ambitions can now pour a seven-digit sum of money straight down the hole almost instantly. And one of the easiest and most efficient ways to do this is to buy the wrong Web Content Management System, or CMS."

Walker goes on to report horror stories of companies run by smart people who bought big, expensive systems that didn't work. Any veteran IT manager in a reasonably large company will have horror stories to tell, if not about a CMS, then about CRM, ERP, or some other three-lettered, seven-figured acronym. The fact of the matter is that buying complex server-side software is hard, and good advice about it is very hard to find. Training managers and Chief Learning Officers better get used to it, or they will find themselves learning some very expensive lessons.

Trust But Verify

Unfortunately, there is only one really good way for you to get accurate information on the quality of server-side programs, particularly for new product categories like the LCMS: You have to do extensive hands-on testing, performing the kinds of tasks that you will really be doing in your own production environment. Get somebody who has experience with both e-learning and IT products. If you don't have the right person on staff, then hire one. Figure out exactly what you want the software to do. Then have your expert try to do it with an evaluation copy. Don't let the vendor do it for you; they already know the bugs in the system and can work around them. Do it yourself. There is no other way to lower the risk that you will buy a useless product.

Equally important is for you to decide just what you want the software to do for you. The badly implemented and poorly documented "features" in these server-side systems are just a symptom. "Featuritis" happens when product quality (or lack thereof) is not recognized by "expert reviewers." As a result, vendors spend their development dollars adding poorly implemented new features that nobody really needs rather than fixing the basic functionality that could actually be useful to their customers. Consumers of these products can force reform by judging products only by those features that offer a credible likelihood of producing a return on their investment. In conjunction with this column, I have written a new article about what's important in an LCMS. Unfortunately, all I can give you are some tips. There are no shortcuts to due diligence. And in this case, the price of not doing it can be very high indeed.


  • There are no comments at this time.