Improving Assessment Quality

Improving Assessment Quality Going Beyond Psychometrics

Testing, which is supposed to increase the scientific accuracy of measuring an individual’s knowledge and skills, may be contributing more inaccuracy than scientific fact. On Sunday, May 20, 2001, a headline in The New York Times read: “Right Answer, Wrong Score: Test Flaws Take Toll.” On October 10, 2001, one of Education Week’s headlines read: “California Score Glitch Throws Wrench Into Bonus Plan.” Although these articles have focused on high stakes tests in the educational arena, the same types of mistakes have been made in certification, licensure, and psychological testing.

The question is, how can organizations prevent such errors from occurring? After we explore the types of errors that make headlines, I will outline five key activities an organization can use to prevent errors and continually improve processes.

What kinds of errors are making headlines? Harcourt reported that one of their scoring irregularities was due to “human error” that resulted in scores being reported as if the test was taken at the middle of the year, rather than the end of the year. NCS Pearson used the wrong key to score six questions on a state graduation test in Minnesota. The questions were scored correctly on other versions of the test. A routine check of the key for the one edition of the test was not conducted. CTB/McGraw-Hill found a programming error resulting in reporting incorrect percentile scores. New York officials also found errors in the Regents mathematics test that “indicated a certain sloppiness or lack of proofreading.”

Are these the same kinds of errors that psychometricians (people who study the statistical and design issues related to testing) discuss? As a “psychometrician,” I attend meetings of the National Council on Measurement in Education, the American Educational Research Association, and the Association for Test Publishers. When I attend sessions, quality usually takes on a very different flavor. Sessions compare the usefulness of various methods to conduct job analysis; they discuss different statistical procedures for equating tests (a statistical procedure to assure that scores on one edition of a test are comparable to scores on another edition). Procedures for preventing the inclusion of “biased” questions on the test are discussed and compared. Studies of how to set cut scores are often discussed. These are all critically important issues, but these technical issues aren’t sufficient to assure the elimination of errors in reported scores.

Psychometric quality is critically important but focusing solely on psychometric quality isn’t enough. Well-designed and documented work processes and quality control procedures are also essential. By designing work processes effectively, we can actually prevent errors, reduce costs, and improve employee satisfaction.

Before summarizing how to avoid introducing “human errors” into the testing process, let’s consider a simple example of a work process. A process is a set of procedures you follow to take some input, work on it, and produce an output. Let’s use the process of making a pot of coffee as an example. The inputs are the coffee and the water. The coffee pot, filter and the measuring utensils are tools you use to create the coffee. The work steps might be: 1) Fill coffee pot with four cups water. 2) Pour water into top of coffee maker. 3) Put coffee pot under drip basket. 4) Put filter into coffee basket. 5) Measure four tablespoons of coffee into filter. 6) Check to see if the coffee pot is plugged in. 7) Turn on coffee pot. 8) Taste before serving to see if the coffee is acceptable. Steps 1 through 8 are a simple process that is illustrated by a flow chart below. Steps 6 and 8 could be considered quality control checks within the process.

If you make your own coffee in the morning, you probably don’t need work instructions to get a good pot of coffee. What if people take turns making coffee? Is it likely to taste the same every morning? Not unless the “cooks” are all following the same formula for combining coffee and water. Written work instructions – possibly above the coffee pot – might result in more consistency each day in the quality of your coffee. How can you prevent errors in the brewing of your morning coffee? Teach each new “cook” where the coffee-making ingredients are, explain the work instructions, agree upon responsibility for brewing the coffee (who does it, what day, what time in the morning). Now we probably have a consistent cup of coffee every morning, but how could we improve that cup of coffee? Look at our inputs – what is the quality of the water? What is the quality of the coffee? Look at our equipment – is it clean and working effectively? Is the formula for the coffee optimal? Explicitly defining your process allows you to discuss and evaluate it.

This is a very simple example. The work processes and quality control procedures involved in developing a test and creating test scores are much more complex. Designing effective work processes for developing tests requires an understanding of the psychometrics of test creation as well as an understanding of work processes. Do your employees really understand their work processes? Do they understand why the checks they are doing are critical to the quality of the scores? Are the quality control checks really necessary?

What might an organization do to assure that their quality control procedures and work processes are effective? You might start by constructing a flow chart of your work processes and describing the work done at each step – be sure you determine what you are doing, not what you think you should be doing. Next, ask yourself some questions. Is everyone doing the job the same way? If not, which way is best? Are mistakes getting through? Why? Once you are clear about what you are doing, invite an independent auditor in to review your quality control procedures. Often we get in such habits of doing things, we don’t think to question what we are doing. To get the most for your money, find someone who is an expert both in test creation and in quality auditing. The auditor will help identify areas where you might improve your work processes and you can set improvement priorities as a result of the external feedback. Or, if your work process isn’t proprietary, do some benchmarking with colleagues in other organizations who also create tests. Compare your processes and share best practices. Everyone benefits.

Connect with Cheryl on LinkedIn