Research Methods for Everyone, Part 4: Experiments, Surveys, and the Polls That Failed

In this post, the fourth in a six-part series on Amanda M. Rosen's Great Courses series Effective Research Methods for Any Project, I follow lectures ten through thirteen, where the abstractions of part three (design, measurement, and sampling) grow teeth: the classic experiment, its quasi-experimental cousins, survey design, and the election polls that put all of it to a very public test.^[1] I will argue that these four lectures teach a single lesson wearing four costumes, namely that the instrument shapes the answer, and I will return to that claim at the end.

Blue eyes, brown eyes #

The day after Martin Luther King Jr. was assassinated in 1968, an Iowa third-grade teacher named Jane Elliott told her class that brown-eyed children were cleaner, smarter, and more civilized than blue-eyed children. Brown-eyed kids got extra recess and second helpings at lunch; blue-eyed kids wore collars and lost their playmates. Within hours, the classroom reorganized itself around the new hierarchy, and the disadvantaged children began performing worse on class exercises. The next day, Elliott reversed the groups, and the cruelty reversed with them. "What had been marvelous, cooperative, wonderful, thoughtful children," she recalled, "turn into nasty, vicious, discriminating little third-graders in the space of 15 minutes."^[2]

Rosen uses Elliott's exercise, later filmed for the Frontline documentary A Class Divided, to anchor the whole discussion of experiments, and then she does something more interesting than praising it: she grades it. A classic experiment has four features: 1) control, 2) manipulation, 3) random selection of participants, and 4) random assignment to groups. "More than anything else," she says, "control defines an experiment," the ability to prevent outside factors from interfering with the outcome. Manipulation means the researcher administers the treatment. Random assignment, giving each participant an equal chance of landing in the treatment or control group, is what defeats selection bias. Measured against that standard, Elliott had manipulation in abundance but no true control group and no random anything. Her exercise is a powerful demonstration, repeated over years with adults, businesses, and correctional facilities, which strengthens its external validity; it is not, strictly, a classic experiment.

That grading vocabulary matters more than the grade. Internal validity is the extent to which you can attribute your result to the independent variable and nothing else; external validity is the extent to which your sample's results generalize to a wider population. The threats to internal validity read like a bestiary: the Hawthorne effect, named for the Chicago factory where 1920s productivity rose with every lighting change because workers knew someone was watching; demand characteristics, where subjects try to give you what they think you want; testing effects, history effects, maturation, dropout, and treatment leaking into the control group. Experimental designs, from post-test-only through the elaborate Solomon four-group design, are engineering responses to those threats.

The Good Samaritan, in a hurry #

Lecture eleven concedes what everyone doing social research knows: the classic experiment is often out of reach. "You can't randomly assign some countries and not others to practice austerity," Rosen notes, and nitric acid, unlike people, "never suspects it's in an experiment and refuses to participate." Quasi-experiments meet some but not all of the four criteria, and her advice is to treat the criteria as "more of a continuum than a dichotomy" and to be honest about where your study falls.

Her showcase is one of social psychology's most quietly devastating studies. In 1973, John Darley and C. Daniel Batson published "From Jerusalem to Jericho" in the Journal of Personality and Social Psychology.^[3] Seminary students at Princeton Theological Seminary completed religiosity questionnaires, then received an assignment to prepare a short talk, for some the parable of the Good Samaritan, for others a talk on seminary careers. On the way to deliver it, each passed a man slumped in an alley, coughing and groaning, planted by the researchers. Some students had been told they had plenty of time, some to head right over, some that they were already late. The result: religiosity did not predict who stopped. Neither did having the Good Samaritan parable literally in hand. Only hurry mattered; situations, not dispositions, carried the day. (For those of us trained in seminaries, the study reads as a parable in its own right, the text of the Good Samaritan failing to produce a single Samaritan.)

The lecture's catalog of quasi-experimental designs is worth keeping at hand. The single-group pre-test and post-test design drops the control group entirely, which makes it weak on its own but valuable as a pilot: if your treatment group shows no change at all, you have saved yourself the expense of a control group. The non-equivalent groups design keeps treatment and control but uses pre-existing groups (classrooms, offices, cities) rather than randomly assigned ones. The interrupted time series measures the outcome at least three times before and after a treatment, which helps rule out the possibility that people were simply changing anyway. And the single-subject design (baseline, treatment, back to baseline) is the formal version of what anyone quitting coffee has done to themselves.

Rosen's treatment is a model of how to read a study. She walks through what the design achieved, control and random assignment, and where it fell short: religiosity was measured, not manipulated; some subjects grew suspicious and were dropped; an assignment error over-filled one condition. None of that sinks the study, cited more than 1,700 times since; it locates the study honestly on the continuum. "You just have to be honest," she says, "about what you have and have not achieved."

The art of the terrible question #

Lecture twelve opens with a trap: a five-question breakfast survey that Rosen has written to be as bad as possible, and then dissects. One question asks whether respondents "eat like birds" or "eat like horses." One asks about eating breakfast on "a major holiday," which could mean Labor Day, Arbor Day, or, she offers helpfully, International Waffle Day. One asks, "Do you always give breakfast to your kids every morning, like a good parent?", a question that manages to prime, moralize, and use an absolute all at once. One is a forced choice between pancakes and eggs. And one buries respondents in jargon, asking about "acute rhinopharyngitis" when it means the common cold. "This question is terrible," she says of one specimen, with evident pride.^[4]

The repair kit is five rules: 1) use plain, neutral language; 2) avoid absolutes like always and never; 3) make closed-ended response options exhaustive and mutually exclusive; 4) never ask a double-barreled question, one that asks two things under the guise of one; and 5) adapt questions from existing, validated instruments rather than writing from scratch, which also buys comparability with prior research. Beneath the rules sits the goal: "Don't try to make them answer the way you want. Instead, aim for honest responses." The lecture also covers the logistics that determine whether anyone answers at all. Rosen tells a story on herself: leaving an Orlando theme park exhausted, she agreed to "a couple of questions," then walked when the couple of questions became a fifteen-minute sit-down survey. "Survey fatigue is a real thing," and two minutes was what she had been willing to give.

What happened in 2016 #

Lecture thirteen is the payoff, applying everything to election polling. The history is a morality play in three acts. In 1896, the Chicago Record predicted Chicago's McKinley-Bryan vote within a tenth of a percentage point off a quarter million ballots, then got everywhere else wrong. In 1936, Literary Digest mailed ten million surveys, received over two million back, and confidently predicted a landslide for Alf Landon over Franklin Roosevelt. Landon won two states. The Digest's sample came from its subscribers plus car and telephone owners, which in the Depression meant the affluent, and the people who bothered to mail back their ballots were not like the people who did not. Sampling bias plus non-response bias equals catastrophe, and the magazine folded in 1938. Meanwhile George Gallup called the election within a point using fifty thousand scientifically sampled respondents, founding the modern polling industry.^[5]

Unscientific polls, meanwhile, never went away; they just became more entertaining. Viewer votes on American Idol and Dancing with the Stars are self-selecting convenience samples. The National Pasta Association once polled pasta preferences by personality, spaghetti beating elbow macaroni 41 percent to 21. And when the UK's Natural Environment Research Council let the internet name a research vessel, the winner was Boaty McBoatface, a result the council partially honored by giving the name to an underwater craft while christening the ship RRS Sir David Attenborough. Open votes measure the enthusiasm of those who show up, nothing more, and Rosen's point is that the same structure underlies every "poll" your social feed serves you.

Good polls, in Rosen's summary, follow five rules: 1) use a random sample; 2) make it large enough (about a thousand respondents for a national poll); 3) minimize sample bias; 4) minimize bias beyond the sample; and 5) publish a transparent methodology. Each rule has its modern failure modes. Landline-only frames now miss the majority who are mobile-only, and they skew older. Social desirability bias, respondents telling interviewers what they think they want to hear, produced the Bradley effect, named for Tom Bradley's 1982 California loss after leading the polls. And herding, pollsters quietly re-weighting their samples to match everyone else's numbers, launders away honest disagreement.

So what about 2016? Rosen's autopsy is more precise, and more interesting, than the folk memory that "the polls were wrong." The major national polls followed rules one and two: ABC News/Washington Post surveyed 2,200 likely voters, the others comparable numbers, all properly randomized. The failures came downstream. Response rates around ten percent amplified non-response bias, under-sampling older rural voters and non-college-educated white men. Likely-voter models misfired. Some respondents apparently would not tell interviewers they supported Trump, social desirability bias in a new key. Media coverage flattened uncertainty: ABC's final tracking poll had Clinton up 47 to 43 with a margin of error of two and a half points, a result within shouting distance of a tie, which the media reported as a "comfortable margin." And above all, the polls were national while the election was not. Clinton won the popular vote by about two points, close to what the polls showed, while losing the Electoral College 304 to 227. "National polls," Rosen concludes, "are simply not a good enough proxy for the statewide polling that's needed to predict Electoral College results."^[6]

The discipline of humility #

What unites an Iowa classroom, a Princeton alley, a breakfast questionnaire, and a polling miss is, as I argued at the outset, the same lesson wearing four costumes: the instrument shapes the answer. Elliott's exercise shows how easily a manipulated situation overwhelms character; Darley and Batson show the same with hurry; the breakfast survey shows that question wording can manufacture any result you like; and 2016 shows that even technically sound instruments answer only the question they were built to ask, a national poll being an excellent answer to a question the Electoral College does not pose. The methods in this stretch of the course are not machinery for producing certainty. They are machinery for knowing exactly how uncertain you are, and for saying so out loud; humility here is not a virtue appended to the method, it is the method, the forecaster's cone of possibilities writ large.

In the next post, I will take up case studies and how to choose them, interpretivism's challenge to the whole positivist edifice, field research from Bali to the ballot box, and research that sets out to change the world rather than merely describe it.

Notes #

---

1. Amanda M. Rosen, Effective Research Methods for Any Project (Chantilly, VA: The Teaching Company/The Great Courses, 2019). This post draws on Lecture 10, "The Classic Experiment," Lecture 11, "The Value of Quasi-Experiments," Lecture 12, "Designing and Conducting a Survey," and Lecture 13, "Understanding Election Polls." ↩︎

2. Jane Elliott, quoted in Rosen, Lecture 10. The exercise is documented in the PBS Frontline film A Class Divided (1985), dir. William Peters. ↩︎

3. John M. Darley and C. Daniel Batson, "'From Jerusalem to Jericho': A Study of Situational and Dispositional Variables in Helping Behavior," Journal of Personality and Social Psychology 27, no. 1 (1973): 100–108, discussed in Rosen, Lecture 11. ↩︎

4. Rosen, Effective Research Methods, Lecture 12. ↩︎

5. On the Literary Digest failure and Gallup, see Peverill Squire, "Why the 1936 Literary Digest Poll Failed," Public Opinion Quarterly 52, no. 1 (1988): 125–33, and Rosen, Lecture 13. ↩︎

6. Rosen, Effective Research Methods, Lecture 13. For the polling industry's own postmortem, see Courtney Kennedy et al., "An Evaluation of the 2016 Election Polls in the United States," American Association for Public Opinion Research (2017). ↩︎