Cost-of-Quality Case Study

Dr. Cheryl Wild

President, Wild & Associates, Inc.

Carol Kovacs
Executive Director (retired)
Educational Testing Service
Princeton, New Jersey 08541


Cost-of-quality measurement provides a yardstick for decision-making. It can be used to identify processes that include excessive waste and rework and estimate potential savings that would result with process redesign. The following article describes the results of a cost-of-quality class given over a four-month period when participants actually applied the tools as they learned them. The results? Ninety percent of the participants indicated they had benefited from the class and recommended it to others. Involvement of employees in the data collection and workflow review resulted in enthusiasm and support for future improvement efforts. Class participants identified potential savings of more than one million dollars.


ETS is a not-for-profit organization whose mission is to provide leadership in educational assessment through testing, research and related services that contribute to improving learning and teaching, individual and institutional decision-making, access and equity, and public information. By far, the biggest part of our work is to provide admissions tests to colleges and universities. These include the Scholastic Aptitude Test (SAT), the Graduate Record Examination (GRE), the Graduate Management Admissions Test (GMAT) and Test of English as a Foreign Language (TOEFL). Our world of testing has begun to change rapidly. Pencil-and-paper tests are quickly becoming obsolete and are being replaced with computer delivered versions. Our customers have new expectations. In a world where a customer can place an order for clothing over the telephone and receive his or her order in less than a week, test takers expect rapid service. The traditional requirement to register to take a test a month before the test administration and then receive test scores six to eight weeks after taking the test seems quite unreasonable.

Recognizing that we are in a time of rapid change, ETS has in the last two years begun to work on a “Corporate Self Renewal” effort. As part of the renewal effort, we have begun to learn quality improvement, quality planning and strategic quality planning methodologies. At the time we began to study the “cost-of-quality,” we had initiated three pilot quality projects. Although our president had named a Council for Strategic Renewal (consisting of company officers and six high-level staff) to provide direction for the renewal effort, the Council hadn’t completed its strategic quality goals. It seemed clear, however, that one of the goals would involve decreasing the costs of providing services to our customers. The cost-of-quality methodology was viewed as a tool to identify projects that would reduce the cost of doing our work.


Our thinking on cost-of-quality methodology was quite similar to that in the cover story of the August 8, 1994 issue of Business Week: “Quality, How to Make it Pay.” The article suggests that the newest emphasis in the quality arena is “return on quality” or “ROQ.” Conducting cost-of-quality studies as a prerequisite to selecting improvement projects is one way to focus efforts on those areas that make the largest contribution to a company’s financial performance.

Most companies use a set of financial controls to monitor the cost of producing products and providing services, and to determine the efficiency of the production processes. The traditional accounting methods to determine cost of goods and services produced and sold fall short in measuring the true cost of producing quality goods and services.

Quality cost measurement provides a yardstick for measuring the cost of processes and the products and services which result, and for identifying opportunities to reduce costs. Quality costs consist of all costs associated with planning the quality system, those associated with verifying requisite quality, and those associated with failures resulting from inadequate systems. The major quality cost categories are:

  • Prevention: All costs associated with activity to keep defects from occurring. Typical examples are machine or process capability studies, vendor surveys, employee training for production work or services, quality designs, etc.
  • Appraisal: All costs associated with activity to maintain quality levels through formal inspections and/or evaluations. Examples are inspections and tests, maintaining test equipment, training associated with inspection, and providing reports.
  • Failure: All costs associated with activity necessary because products or services do not meet specifications or fail to meet customer requirements. Internal Failure includes costs associated with scrap, rework, design changes, excess inventory. External Failure includes warranty costs, customer complaint resolution, field-service training, returns, recalls and liability suits.

The interrelationships among these cost categories provide useful comparisons for deciding whether the money being spent is put to good use. Typical cost studies show woefully low amounts allocated to prevention (0.5 – 5.0 percent), disproportionately high amounts provided for appraisal (10 – 50 percent), and failure costs, surprisingly, several times the appraisal costs. Of course, external failures have an exponentially negative impact because of the ripple effect dissatisfied, angry or poorly served customers can have on a company’s bottom line


For the first offering of the “cost-of-quality” course, our primary objective was to learn the methodology and to identify individuals who might continue offering the course at ETS. As a secondary objective, we wanted to identify potential quality improvement projects. The people we selected for the course came from a number of vice presidential areas and included people who could be future instructors.

Our initial “cost-of-quality” class was offered over a four-month period. There were four half-day classes with homework in between each class. The class proceeded through the following seven steps: Identify a process, prepare a process flowchart, prepare a step procedure (identifying the steps in the process), classify the steps in the process, estimate quality costs, prepare a quality cost report and identify process areas for improvement.

In order to provide a case study of how these seven steps were actually followed, we will follow the history of one process that was studied: answer-sheet processing. An answer sheet is a computer scannable document that the test taker uses to record the answers to test questions when taking a pencil-and-paper test. The answer sheets are then shipped to ETS to be read by a scanner, matched to the test taker’s order (or, “registration form”), and scored. More detail about each of the steps followed in the class are provided below


Each participant was to select a process to study over which they had responsibility. The business process was to include no more than about ten full-time people (in order to make it about the right order of magnitude to study during the course of the class). We also wanted the process to be one that occurred during the time period of the course, so that data about cost would be collected. Before the classes began, participants met with the instructor to discuss the process selected for study, keeping the initial course objectives in mind. Twelve different processes were selected for study.

The answer-sheet processing process was selected after considering a high-level flowchart of processes involved in the test processing. Test processing involves many subprocesses, including for example, conducting test administrations, processing answer sheets, processing registration forms and providing score reports. Within all these processes there are quality inspection and control steps that might represent major commitments of staff time and expense. ETS has relatively few quality problems with test delivery and score reporting; however, whenever a problem occurs, our first impulse has been to add a quality inspection step. Consequently, we have a highly complex quality control system that evolved incrementally over many years. Answer sheet processing is known to involve inspection activities and seemed a good place to start.


Class participants were instructed in flowchart preparation, and then asked to prepare a flowchart of the process they were studying. The flowchart at this stage was still at a macro level but detailed enough to identify all the areas involved in the processing, the sequence of document and data flows, the nature and extent of rework loops, and of course, the supporting quality inspection steps involved in controlling the subprocess. During this stage, it is essential to involve the staff doing and supervising the work–to assure all of the important steps have been identified and to establish a rapport with the staff members you will be working with to obtain the detailed procedural, volume and cost information.

Based on our experience, we would offer a caveat here: Be mindful that most flowcharts focus on the document, information flows, and processing and may omit the preparation or preliminary activities that must occur before the work can begin. Many quality system prevention activities will be unaccounted for if you do not consider the necessary work preceding the actual processing. For example, procedure documentation, employee training, machine setup and testing are all quality system prevention activities that should be included in the process flowchart for the process to be correctly depicted. Because our flowcharts did not include these activities, we had to go back after we had completed much of the study to determine what type of prevention activities, if any, preceded the work. If it is awkward to include these prevention activities in the flowchart, they can be included in the step procedure described below.


Developing a step procedure requires listing the tasks involved in completing a step, at an appropriate level of detail. Here it is important to keep the basic objective of the study in mind: we are looking to identify costs associated with prevention, appraisal and failures inherent in the quality system. The detail should be at the level enabling the categorization of steps as production or as those related to the quality system in place. More detail is unnecessary and complicates the cost-of-quality study.

At the time the tasks are listed, it is also important to identify the position type and level of the staff member(s) accomplishing the task. This information is prerequisite to estimating direct and indirect labor costs associated with the quality system. We were able to get to the appropriate level of detail after we began looking at cost data. Consequently, our step procedure was revised several times during the study.


The initial categorization differentiates production steps from those involved in managing or maintaining the quality system. This was not an easy step for us to accomplish. All work associated with producing the product or providing the service is a production step. All work related to assuring the work is done correctly the first time, or checking that it was done correctly, or assuring that all work is accomplished that should be (input/output controls), or correcting work are all part of the quality system. For each task, we had to ask why the step was performed and what would come out of it. Also, because of the complexity of our work processes, some steps, even at their least common denominator level, included both production and quality system components. In these cases, we used our best estimate of the breakdown. We concluded that the cost-of-quality study was a management tool for the purpose of making relative assessments, and that the data did not have to be precise to be useful.

Once the initial categorization was complete, it was necessary to classify the components within the quality system steps as prevention, appraisal, internal failure or external failure. There may be times when it is not clear in which cost-of-quality category a step or task should be placed. A general rule of thumb is to place the quality cost component into the higher-order category if the step is a consequence of that category. For example, if quality inspection is done on a repaired product that was returned or rejected by a customer, the quality inspection step is part of the external failure, and not an appraisal step. Involving the employees at this step in discussing the appropriate classification can be helpful. One interesting outcome of our work was that employees began to realize that they spent as much or more time checking the work as they did in doing it. They felt there was something wrong with the way the work procedures were set up. This revelation laid the groundwork for their welcoming procedural changes.


In the case of the answer sheet process, we were fortunate to have much of the data needed available in production reports for one test administration. We were able to extrapolate full year data. However, we still had to estimate volumes, labor hours, and the cost of each employee performing work for each quality-related task in the step procedures. Other class participants had to collect data as work was being performed. Designing and piloting of data collection forms was another positive opportunity to involve employees. This step turned out to be more difficult for class participants than anticipated. Since costs hadn’t historically been collected in this way, new data collection often had to be initiated. Many class participants didn’t have experience in designing data collection and had to learn that skill.

Once the data was collected, it was summarized in direct labor dollars. To assure comparability of the cost-of-quality results, we used 1.5 for each direct labor dollar to reflect the indirect costs of benefits. We did not include what we call “cost center load,” an accounting process that generates numbers that can vary from department to department. The use of a standard rate assured that we could all be using the same yardstick when we compared our study results to determine opportunity for quality improvement projects.


Data were then collected into a quality cost report, which is summarized in Figure 1. In the case of the answer-sheet processing example, the report took four pages. Figure 1 includes only the shell of the report and the summary. In the full version of the report, each of the work steps is listed in the left-hand column, indicating the cost of work that is prevention, appraisal, internal failure or external failure. The total cost of the process (including the production costs) is also calculated. For answer sheet processing of this one testing program, 5 percent of the quality costs were due to prevention, 37 percent due to appraisal, 42 percent due to internal failure and 15 percent due to external failure. In terms of total expense, about 56 percent of the costs were cost-of-quality costs.

This data raised many questions. Could we improve the internal and external failure rates by increasing our prevention activities? Would we be able to reduce the appraisal costs if we could improve our prevention activities and preclude errors from happening in the first instance? Could we reduce the 56 percent for the quality system activities, increase our production and lower total costs?

In another case, we were spending 50 percent of our quality system costs on appraisal and 50 percent on prevention. Since there was very little internal error and no external error, could we reduce the internal appraisal cost and reduce total costs overall?

The key point was that we were, relatively speaking, spending a great deal on the quality system and in at least one case more than we needed. If we spent more on prevention we could probably lower total cost and increase production. The results led us to suspect we would find similar situations in the rest of our test processing subprocesses, and that a more coordinated and integrated quality system for the processing would yield great benefit.


At the end of the class, each participant summarized their findings. We then did an analysis of all processes that included more than $10,000 in appraisal and failure (See Figure 2). In most cases, participants had studied a process specific to a test program (such as SAT, GRE or TOEFL). In Figure 2, the asterisks beside some of the projects indicate that a solution to eliminate the costs of poor quality in one program could be transferred to other programs and probably result in savings as much as ten times the number estimated in the class.

Based on the results, several projects were selected to be included in the next round of quality improvement projects. One major one was, of course, the quality inspection and control system in place for answer sheet processing. We expect a much improved quality system to be put in place when the quality improvement team has finished its work.

However, identifying areas of improvement really goes much beyond the identification of the quality improvement projects. Even when the processes studied were too small to be proposed for quality improvement projects, class participants were able to identify ways of improving how they did their work. For example, one individual found that 60 percent of the cost of the process was for inspection and that no internal or external failure was ever found. The inspections were put in when the computer system to do this work was new and being beta tested. Once the “bugs” were all out of the system, no one ever went back to rethink what quality control needed to be done now that the system was improved. The individual was able to go back and eliminate a number of checks in the process, just from having thought about the process differently.

At the end of the class we did a survey of participants. We found that 90 percent of the class indicated that they benefited from taking the class. Ninety percent also recommended that others take the class.

Figure 1

0 Procedure
2,029 2,029
00 Training 978 978
5 Check A/S
with SRF
72 72
7 Key batch
753 753
21 Perform QC 144 144

$3,007 $20,904 $23,741 $8,216 $55,868
as a % of Total Quality Cost
5.4% 34.7% 42.5% 14.7% 100.0%
as a % of Total Unit Expense
3.0% 20.9% 23.8% 8.2% 55.9%

Figure 2

*Other areas use this process.
Cost Reduction potential in this area may be 10 times this amount.


The use of “cost-of-quality” as a tool for looking at one’s work processes has been quite popular with class participants as well as a useful methodology for selecting quality improvement projects. This is true for several reasons. First, it creates a common language across areas to make decisions about costs. For example, when costs for inspection and failure are included in operating costs, managers have traditionally viewed them as the cost of doing business. It can be hard to justify prevention activities when costs of inspection and failure haven’t been tied directly to prevention. Use of this tool encourages a long-term view of costs and realizes the value of prevention. Second, the way the process selection and data collection is designed, the actual “participants” in the process are involved in studying the process. Someone doesn’t come and do this for you: the person responsible for the process participates in the course and involves others who do the work in developing the flowchart and collecting the data. This encourages employees to start thinking about what they are doing and why and to come up with improvement ideas as they go along. Finally, it is a very simple tool to understand and apply. It is something that a manager can understand easily and share with employees in a constructive way.

Dr. Cheryl Wild is currently the president of Wild & Associates, Inc., a consulting firm that helps organizations improve their efficiency and effectiveness.

Reprinted by permission of the Juran Institute.

Connect with Cheryl on LinkedIn