The psychiatrist in a murder trial testified that the defendant’s increase in infrequent responding over the course of an MMPI administration meant that his (the defendant’s) will was overridden over the course of the two-hour police interrogation, so his (the defendant’s) confession should be thrown out. I testified that my profession is justly proud of knowing what most people will do on a given occasion or what a given person will do over many occasions, but it’s ludicrous to think we can tell from current data what a given person will do or did on a specific past occasion.
This confusion is ubiquitous in psychology. The Law of Large Numbers assures us that, on average, the behavior of a group will settle around its norm. This enables us to predict the incidence of crimes or product usage or car crashes, for example. The same reasoning enables us to predict how an athlete will perform over the course of a season, but not on a specific occasion. Even Ted Williams hit into numerous double plays, and even Bucky Dent hit an occasional game-winning home run. When I hear that so-and-so is not the kind of person to do such-and-such, I wince. Indeed, in my experience, it’s often the conviction that they are above misconduct that leads people to wander into temptations they can’t manage.
Two researchers investigated the ability of psychologists to predict outcomes for specific people in a specific task. (Keep in mind that prediction means speaking about something before you know the outcome, not before it happened; claiming that you can tell that a client was sexually abused as a child by observing her as an adult is a prediction.) All candidates for Air Force flight school received an in-depth clinical interview, a Rorschach, a sentence completion test, a figure drawing test, and two other instruments. After flight school was completed, the researchers chose 50 graduates who passed with flying colors and 50 who had above-average pilot skills but flunked out for reasons based on their behavior, excluding those who had a physical illness. Could 19 famous clinical psychologists examine the entry protocols and tell whether a given pilot ended in success or psychological failure?
The answer was no. The psychologists performed no better than chance, not even when their judgments included only the pilots they felt certain about, not even when all three psychologists examining the same pilot agreed on whether he or she was a success or failure, not even when the best (luckiest?) of the psychologists was separated from the others. If Klopfer and Piotrowski (both of whom participated) can’t tell whether a pilot will make it through flight school, can I use a Rorschach to say whether someone will succeed in a job? Can I conduct an interview and tell if the candidate will succeed in graduate school?
Of course, we have better tests nowadays and better methods, the primary one of which is to interpret only unusual data. However, I still see many assessment psychologists trying to make sense of all the data, as in page after page of interpretive statements often derived from computerized services. And the important thing about this one study is how biased the design was in favor of the psychologists (by including only stars and duds).
Take heart; there are two things we can do. We can engage in iterative processes, learning from our mistakes and depending more on our data than on our theories. Thus, after this study was conducted, the researchers examined the actual differences between successful and failed pilots and began the development of an evidence-based rather than a theory-based algorithm. They found, for example, that lifelong interest in flying did not predict success after all. This method works only if you follow up on your assessments and learn from them and if you scrupulously interrogate and debunk your cherished signifiers. The other thing we can do is explain rather than predict behavior. A good explanation is one that gives people ideas about what to try next. I don’t read many psychological evaluations that explain why a parent, say, behaved abusively or neglectfully; I mainly read vague, often tautological (“he beat the child because he is angry and impulsive”) descriptions of the parent’s character.
This is just as true in therapy as it is in assessments. Because every therapy starts with an assessment or case formulation, the implication is that many therapies are organized around the generalities of the patient’s master narrative rather than the specifics of the patient’s ways of relating to the world.
If your clinical judgment is based on your estimation of yourself, you will fail. If it is humbly subordinate to evidence and focuses on what to do rather than on what happened or will happen, you might say something useful.
Holtzman, W. & Sells, S. (1954). Prediction of flying success by clinical analysis of test protocols. Journal of Abnormal and Social Psychology, 49, 485-490.
This post was originally published in The Colorado Psychologist.
