Does Your Examination Communicate Cooperatively?

By Timothy Habick, Ph.D.

Although we usually are unaware of it, we follow a set of “cooperative” rules when we have conversations. People who write high-stakes examinations must become keenly aware of these rules and apply them meticulously. It is each testing program’s duty to ensure that its examinations reach high levels of compliance with the rules of cooperative communication.

So what is cooperative communication?

Cooperative communication means talking and writing in a way that takes into account the overall context of the situation, with particular attention paid to the assumed knowledge and reasonable expectations of the person or people on the other side of the exchange. When people communicate cooperatively, they appear to follow certain “rules.”

The “rules” of cooperative communication were not imposed by someone in authority. Rather, these rules reflect the expectations that people have when they talk to each other. The philosopher Paul Grice observed natural, successful linguistic interactions and inferred from them a few major rules that everyone seems to follow.  He called them maxims.

  • Be truthful.
  • Be clear.
  • Be relevant.
  • Be appropriately informative.

Although these rules appear self-evident, all human communications rely on them. Grice observed that when a person breaks any of the rules – on purpose or not – other people become confused, or sometimes amused.

Confusion and comedy, of course, are completely out of place in examinations. Test questions need to be written as if every word and every sentence will be taken literally. This might or might not be obvious, but it’s a fundamental rule of test development. Examinations are intended to serve one purpose – fair assessment of relevant skills. This is why examinations must be highly cooperative acts of communication.

Grice thought of his maxims as descriptions of the way people typically communicate. He wasn’t trying to teach people how to communicate, or even how to communicate better. Nonetheless, communication skills, like most of language, are learned behaviors. And some people exhibit better skills than others. At the low end of cooperative communication, people with schizophrenia tend to violate the relevance maxim, often choosing to rhyme their sentences, for example, at the expense of meaningful contributions to the conversation. Such conversations, of course, do not represent successful linguistic interactions.

Grice’s maxims sound a lot like admonitions from a writing teacher because people who would normally talk in a perfectly cooperative way have trouble applying their communication skills to writing. This is because the verbal and nonverbal feedback that talking communicators enjoy is normally absent, or long delayed, when we write. One exception is texting, in which writers get instant and continual feedback on brief written thoughts, one major reason for texting’s popularity among young people.

Great writers in the traditional sense consistently keep up their side of the conversation with their readers; they never forget that there are readers on the other side of the exchange. Test question writers must emulate, and even surpass, this one trait of great writers, by being extremely cooperative with their readers.

Let’s take a broader view. Your organization’s certification examination is not just an assessment tool. It is also a significant act of communication, whether or not you intend it to be. Each test question projects a message about your organization to your current and future members. With each question, you make a statement about the knowledge, skills, and abilities that are important to your profession. So each question represents a communication of that type. Some questions regarding actions that are appropriate or inappropriate for professionals in your field can be regarded as official policy statements. By certifying a particular answer choice as correct, your organization is taking a position on that issue.

In addition to content considerations, you might also think about the overall tone and affect of the group of “conversations” that make up your examination. So how cooperatively does your examination communicate? Does it treat your test takers with the same respect and consideration you would give to any professional colleague during a conversation? The only way to know is by evaluating each test question in terms of cooperative principles.

Standardized test questions require an application of cooperative communication standards that is much more rigorous than that needed for normal discourse. Uncooperative writing could be one source of the difficulty that some people have with tests. They say, “I know the subject, but I just don’t test well.” In some cases, they are right. The problem, at least sometimes, is not with the test takers but with the unreasonable, disorienting nature of the questions. Some test-wise candidates overcome poorly written questions by using their intuition and common sense to reject some literal interpretations. They answer the question the test maker intended, not the question as it stands. Such test takers are doing more work than should be asked of them. It is the test maker’s duty to communicate clearly in the first place.

Here is a sentence taken from an actual examination question, as originally submitted by an item writer. (Certain variables in all sample questions have been changed for test-security reasons.)

A township assessed a $9,000 tax on a commercial property for the upcoming calendar year on January 1, 2009, at which time the tax was added to the property’s other encumbrances.

The test taker’s task is to process and interpret each sentence and to link those sentences into a meaningful whole. But what does the quoted sentence mean? Specifically, we wonder what for the upcoming calendar year might mean. If an action occurred on January 1, 2009, then the upcoming calendar year, literally, would be 2010. The writer was trying to refer to the rest of the 2009 calendar year, but the original wording did not say that.

So a cooperative version of the sentence would be something like this:

On January 1, 2009, a township assessed a $9,000 tax on a commercial property for the current calendar year, at which time the tax was added to the property’s other encumbrances.

Another way that question writers can be uncooperative with the reader is by asking a multiple-choice question with answer choices that do not naturally respond to the question. It is true that multiple choice questions are rather unusual acts of communication. But they can become much less peculiar when cooperative principles are followed.

Consider the following question, which is similar in its essential structure to one submitted to a testing program. (There is no actual Standard 402-Y.)

When should inventory managers apply Standard 402-Y to a discontinued product?

(A) Never, because Standard 402-Y does not apply to discontinued products

(B) Only if the product had a serious defect

(C) Only if Standard 401-X also applies to the product

(D) On the day that the last unit of the product was sold to a distributor

This question is uncooperative in several ways. Options (B) and (C) do not answer a when question; they answer a question that asks in which circumstance?

Option (A) also does not answer the question posed in the stem. The question asked when, but (A) tells both when (if at all) and why not. Only (D) directly answers the question, as posed.

A more cooperative version of this question would be something like this:

When, if ever, should inventory managers apply Standard 402-Y to a discontinued product?

(A) Never

(B) One month before the last unit of the product will be produced

(C) On the day that the last unit of the product was manufactured

(D) On the day that the last unit of the product was sold to a distributor

Item writers often stray from cooperative communication while trying to imagine the incorrect answer choices. Test questions can often be vastly improved in terms of communicative compliance merely by changing some of the distractors or by modifying the stem. In this case, we added if ever to the stem to allow for the possibility of a negative response. The original key of (D) can remain as originally conceived.

Some testing programs prefer all the answer choices to be of the same length, so they would want to make (A) longer. Cooperative principles, however, require us to “Be appropriately informative.” In particular, that maxim means to “Say no less than is necessary,” and to “Say no more than is necessary.” If never clearly represents the relevant idea, it is entirely appropriate to use the single word. Notice that all four options are adverbial phrases, with the adverbial phrase in (A) consisting of a one-word adverb.  So the answer choices are parallel in grammatical structure and thus are easy to process as direct answers to the “when” question. Adding additional words to “never” not only serves no useful communicative purpose but also violates the cooperative principle of saying no more than is necessary.

Some questions are even less cooperative than the one presented above. Consider this question from a program that uses five answer choices:

Which of the following is not true about intervention Y?

(A) Treatment for certain types of infections.

(B) 5 hours on and 5 hours off.

(C) Recommended to change every 9 days.

(D) Not associated with local sensory loss.

(E) Approved for up to 4 doses per day.

Answer choice (B), for example, might be a telegraphic version of a common idea in the field, but it still is not a natural way to answer the question. That is, in natural speech, people wouldn’t answer the question in that way. Here is a more cooperative version:

Each of the following statements about intervention Y is correct, EXCEPT:

(A) It is an effective treatment for certain types of infections.

(B) It should be administered five hours on and five hours off.

(C) It should be changed every nine days.

(D) It is not associated with local sensory loss.

(E) It is approved for up to four doses per day.

Now the stem announces that the answer choices will all be statements, and the answer choices are indeed all statements, i.e., complete sentences, with the same essential grammatical structure. This makes the question easier to process, even if the options have a few additional words.

Sometimes an item writer just doesn’t find the right word to express the intended idea. Consider:

Which of the following infections is more suspicious of syndrome F?

The test taker will ask, “Suspicious? Suspicious of what? How can infections be suspicious?” And that would be a reasonable complaint. The item writer actually meant to ask something like this:

Which of the following infections is most indicative of syndrome F?

Cooperative writing, of course, is always good writing, in which each word literally and clearly conveys the intended meaning.

The field of test development is dedicated to the application of principles that promote clarity of communication. Here are some more of these principles, expressed as check-list maxims.

  • Ask just one question per question.
  • Make the answer choices parallel in structure.
  • Use language that ensures efficient information processing.
  • Use language that communicates without undue emotional baggage.
  • Use language that is natural, not strange.
  • Use language that does not call undue attention to itself.
  • Use the same wordings, not synonyms, throughout to avoid confusion.
  • Test specified content, and only that.
  • Don’t test the English language if you’re not testing the English language.
  • Don’t test reading skills if you’re not testing reading skills.
  • Don’t test logic if you’re not testing logic.
  • Don’t play logical tricks on the test takers.
  • Don’t teach while you test.

Are you controlling the content and quality of the communications in your examination as well as you control your other official publications and press releases? Your organization’s future members could resent an unfair, irrelevant, or poorly written test question, even if they earn your certification. Most importantly, as your questions become more cooperative, your examination will become a better, fairer, and more valid assessment tool.

In a future issue, we will look at some common logical tricks (unintentionally) included in test questions, and how to eliminate them from your test.

Connect with Cheryl on LinkedIn