Making a Certifiable Cake – Part III

Tips for New Certification Organizations:
Making a Certifiable Cake from Start to Finish
Installment 3 – The Measurement Layer

by Cheryl L. Wild, Ph.D.
President, Wild & Associates, Inc

Do you know which of the following standards requires that a job/practice analysis be conducted as a basis for developing an assessment instrument for certification?

  1. ISO/IEC 17024 Conformity Assessment – General Requirements for Bodies Operating Certification of Persons
  2. National Commission for Certificating Agencies Standards
  3. Standards for Educational and Psychological Testing

This is a trick question – all three answers are correct! The stem of the above question was taken from the NCCA Standards which state that “A job/practice analysis must be conducted leading to clearly delineated performance domains and tasks, associated knowledge and/or skills, and sets of content/item specifications to be used as a basis for developing each type of assessment instrument.” The other two standards also include statements about the importance of job/practice analysis. The point is that the appropriate measurement steps for developing a certification test are well documented and generally agreed upon by measurement specialists.

Why is it important to follow these steps? Any test, even a voluntary certification test, feels like a high stakes test to at least some of the candidates taking the test. What happens when an examinee has studied for six months to take your test, fails the test, and feels like the test is unfair? At the very least you will have a letter of complaint from the candidate. Perhaps the examinee’s employer will complain. If more than one examinee feels the same way, you may have many letters of complaint. Responding to letters may be time consuming, but it isn’t serious. However, losing examinee volume or association membership because your test has a reputation for being unfair can be serious. There is also always the possibility of a lawsuit. The best practice measurement procedures for developing a test help you avoid these types of problems and generally lead to a more valid and reliable test.

In some professions, employers start to recognize that employees who have passed a certification examination are better prepared to do their job. Their customers like to know that employees are certified. Employers may start to encourage their employees to take your test and even include your certification in their job advertisements! When this occurs, your examinee volume increases and usually volunteers are more readily found to help with your development process. Job analysis, standard setting, item and test analysis help to increase the likelihood that your certification examination will be well respected in your field!

Today’s Tip: Developing a test is more than writing a bunch of questions and publishing them as a test. The steps in designing a certification test should include:

  • Conduct a job/practice analysis to identify the content outline for your test.
  • Establish your cut score using a standard setting study.
  • Analyze the results of the test to make sure the test questions are working (have one and only one answer) and to assure that your test is reliable.
  • If you use different forms of a test, use some procedure to ensure that the forms are equivalent and the cut score is fair on all forms.
  • Develop procedures to ensure secure and appropriate data retention.

In certification programs with volumes of more than 200 at a test administration, the statistical procedures for setting cut scores and equating are straightforward. A number of resources on these topics are available in the new ICE Resource Directory on the ICE web site.

However, assuring consistent and fair cut scores in small testing programs can be quite difficult. The article by Dr. Steve Downing in this issue of ICE News explains the issues a small testing program must consider in setting passing scores.

Reprinted by permission of the Institute for Credentialing Excellence. Dr. Wild is currently the president of Wild & Associates, Inc., a consulting firm that helps organizations improve the reliability and validity of their tests and surveys.

Connect with Cheryl on LinkedIn