Making a Certifiable Cake – Part IV

Tips for New Certification Organizations:
Making a Certifiable Cake from Start to Finish
Installment 4 – The Icing (Vanilla or Pickle Flavor?)

by Cheryl L. Wild, Ph.D.
President, Wild & Associates, Inc

I find that I am often called to help organizations that find themselves in some sort of “pickle.” What kind of a “pickle” can a new certification program create if they have followed the advice in earlier articles (benchmarking, planning, and using appropriate measurement techniques)? The answer is many. Given their experiences, certification professionals can offer a number of examples of situations that have required creative problem-solving. Some of those I have encountered include:

  • After you have awarded credentials to a new group of certificants, you discover that a printer error has resulted in the use of old test questions in the exam instead of new, improved questions the item writing committee developed.
  • Your tests contain embarrassing typographical errors.
  • You discover certificant records have not been updated due to staff turnover. Even worse, you discover no one on the current staff knows how to update the database.
  • Five copies of the test are lost at various test centers and you do not know if a security breach has occurred or if it is the result of sloppy handling of test books by test center supervisors.

Dutifully, I advise organizations on how to extricate themselves from these pickles. A more effective solution, however, is to educate program sponsors on basic tactics to avoid these problems in the first place.

Today’s Tip: Develop general practice procedures that:

  • Prevent problems rather than require quality control to detect and handle problems;
  • Are documented and easily available to employees;
  • Include measures to ensure effectiveness; and
  • Are reviewed, updated and improved regularly.

There is a huge difference between prevention and contingent actions. Every doctor I know encourages regular check-ups, vaccinations, healthy diets and exercise. None say, “Don’t worry; you have health insurance, so eat whatever you want.” Health insurance is a contingent action we all need to have, but not one we want to exercise.

Testing is no different. Quality control can help identify problems with your test, but more importantly you want to prevent the errors from occurring at all. How can you avoid basic problems? An example of a client situation I recently encountered exemplifies some basic tactics. (The client uses an item type that presents a complex situation with multiple questions based on the situation.) Different versions of the test contain minor revisions of the situation with multiple questions based on the situation. For each test edition, the situation and the subsequent questions has been entered manually into the question item bank. Even though 70-80 percent of the text entered was identical, the data entry led to multiple errors due to incorrect input.

This particular testing program previously developed manual procedures to enter test questions in the item bank and the frequency of errors was their measure of success of item entry into the item bank. The executive director reviewed the data on errors and concluded too many had occurred. When the association recognized the problem it took action to prevent these errors in the future by developing a new process.

First, a “gold standard” version of the situation and questions were created. Then they created a small computer program to change selected words and produce the new version of the simulation. Once the computer program was checked, the changes were automated and the incidence of manual errors was eliminated. The new procedure is now documented and available to all employees to avoid losing the knowledge due to staff turnover.

This is a good example of reviewing a previous policy to prevent errors and updating it when evidence shows it is no longer effective. All testing programs spill pickle juice onto their certifiable cake once in a while. The idea is to avoid mixing the pickles into the icing and having them built into the cake!

Reprinted by permission of the National Organization for Competency Assurance. Dr. Wild is currently the president of Wild & Associates, Inc., a consulting firm that helps organizations improve work processes and employee performance to increase the validity of tests and surveys.

Connect with Cheryl on LinkedIn