Home / Blog / Research Methods for Everyone, Part 6: Data Analysis, Statistics, Qualitative Coding, and the Write-Up

This post closes my six-part series on Amanda M. Rosen's Great Courses series Effective Research Methods for Any Project. Here I take up the last third of the course (lectures seventeen through twenty-four): preparing and analyzing data, quantitative and qualitative, and then what Rosen calls "the last but perhaps most important step in the research process," telling people what you found.[1] The first five posts traced the arc from question to design to data collection. I will argue that this closing stretch, and the course writ large, teaches a temperament as much as a toolkit; at the end, I will say what I take that temperament to be worth for humanists.

Data is the heart, process is the difference #

Lecture seventeen opens with Michael Scott. In an episode of The Office, Michael learns a coworker served prison time. When the staff muses that prison sounds better than the office, he stages his cautionary alter ego "Prison Mike," whose sources on prison life, he admits, were "50/50 internet." Michael is ridiculous, of course, but Rosen is hunting bigger game: even the real ex-convict's experience is one case. Is it representative? "What differentiates research from informed opinion is process."[2]

Process here means unglamorous things done well. Coding transforms raw data for analysis, assigning variable names and numeric values, documented in a codebook so that others, including your future self, can reconstruct what you did. Her exemplar is the Alliance Treaty Obligations and Provisions dataset, in which Brett Ashley Leeds of Rice University and colleagues coded the contents of treaties from 1815 forward, reducing even the 1939 Germany-Soviet non-aggression pact to a disciplined row of zeros and ones.[3] Inter-rater reliability keeps multiple coders consistent. The machinery of the AP exam shows what that consistency looks like at industrial scale: thousands of teachers trained on a rubric, their scores tracked, about fifteen percent of each reader's essays re-read by table leaders. And, for heaven's sake, back up your work. Rosen tells of a doctoral student whose only raw survey data went into the trash with a departed assistant, ending a PhD, and another whose sole thesis copy drowned with her laptop in a canal. She keeps everything on two cloud services and two external drives. "I may be paranoid, but I've also never had to sheepishly admit to losing my work."

Statistics for people who read words #

Rosen's four statistics lectures are the gentlest on-ramp I have encountered, and their organizing insight is that statistics answer to the same honesty norms as everything else in the course. Descriptive statistics summarize a sample. Means bend to outliers; one person's ten-million-dollar windfall drags a neighborhood's average income to two million (which is why analysts always report household income as a median). Modes and standard deviations (the average distance of scores from the mean) fill out the kit. Z-scores express any case's distance from the mean in standard deviation units, letting you spot the typical and the deviant at a glance. A restaurant sitting four standard deviations from the mean on cleanliness, she notes, is "either the cleanest restaurant imaginable or dirty beyond belief."[4]

Inferential statistics carry you from sample to population, and the logic runs through the null hypothesis from part two of this series. We test the claim that no relationship exists; p-values tell us how likely our results would be by chance alone, with .05 the conventional threshold. For an illustration, Rosen reaches for Settlers of Catan. With two dice, sevens should turn up most often, yet she once lost a game in which nobody rolled a six all night. The dice were not loaded; chance produces flukes, and Type I and Type II errors (false positives and false negatives) are the statistical names for being fooled by them. The tests she surveys (z-tests, t-tests, ANOVA) all ask whether a difference between groups is bigger than the flukes can explain. She reruns both Jane Elliott's classroom and the Good Samaritan study through ANOVA's mill to show how design and analysis lock together.

For categorical data there is the cross tabulation table and the chi-squared test, which Rosen demonstrates on the 2016 election's gender gap. Per Pew Research Center data, 54 percent of women and 41 percent of men voted for Clinton. The chi-squared arithmetic on a simplified version of those frequencies leaves no doubt that the association between gender and vote is real rather than chance.[5] She pairs the technique with a warning label, the ecological fallacy: a group-level result never licenses conclusions about an individual. Being a woman in 2016 did not mean you voted for Clinton. Anyone who has watched demographic punditry collapse individual voters into their categories has seen the fallacy at work.

Lecture twenty delivers the course's most quotable discipline: "Correlation is necessary but not sufficient to establish causality." Establishing causation takes four steps: 1) correlation, 2) a plausible theoretical relationship, 3) temporal order, and 4) the elimination of spurious alternatives. The ice cream and violent crime correlation is real, and a third variable, summer heat, drives it. Rosen does not spare her own field's founders either, pausing to note that Karl Pearson, who gave us the correlation coefficient, was also a prominent proponent of eugenics, "the pseudoscientific effort to breed improvements in the human race."[6] A statistical tool guarantees nothing about the wisdom of its user. Regression, in the final quantitative lecture, is what moves observational research toward causal claims, using control variables to hold rivals constant statistically when you cannot run an experiment. Her closing counsel bears repeating: no matter how large an effect looks, if it is not statistically significant "it should not be reported as a genuine finding," and a small real effect beats a large illusory one.

Qualitative analysis is not the easy road #

Lectures twenty-two and twenty-three give qualitative analysis a rigor manifesto. "Qualitative work done right is neither easier nor more forgiving than quantitative work," Rosen warns. "At best, it's easier to do poorly."[7] The process differs structurally from quantitative work. Collection and analysis run side by side; sample size is not fixed in advance but continues to saturation, "the point at which collecting additional data provides no new insights or value added"; and the enterprise is open to inductive and interpretivist approaches. The tools are coding, applied to "texts" in the widest sense (books, speeches, films, field notes, message boards), and memos, the running interpretive commentary where, she says, "you'll start making connections, generating theories, and seeing patterns."

The dangers are cognitive rather than computational. Confirmation bias comes first: "We're more likely to accept data that agree with our pre-existing beliefs and ignore or discredit data that challenges it." The anecdotal fallacy follows, illustrated by Munich, 1938, one spectacular failure of appeasement forever after treated as a refutation of an entire strategy. And data fitting closes the list: you build your theory from the first fifteen cases and then fail to see the contrary evidence in the next fifteen.

The variations tour in lecture twenty-three is a field guide worth the price of admission: 1) grounded theory, which develops theory rather than testing it, through open, axial, and selective coding; 2) discourse analysis, which reads communication as performative, noting that "pro-fascist speeches have a very different meaning and interpretation in 2018 than in the 1930s"; 3) conversation analysis, exemplified by Don Zimmerman and Candace West's 1975 finding that of 56 recorded interruptions in public conversations, 48 occurred in mixed-gender pairs, and all but two of those were men interrupting women;[8] 4) narrative studies, in the mode of Ronan Farrow's Weinstein reporting; 5) phenomenological research into lived experience; 6) content analysis, which quantifies qualitative material, as when criminologists coding months of cable crime coverage found victims and perpetrators wildly out of proportion to actual crime statistics; and 7) process tracing, Andrew Bennett's detective method for adjudicating between causal explanations of single events like the peaceful end of the Cold War.

The best breakfast in town #

The last lecture opens with four words on a restaurant window: "best breakfast in town." No plaques, no rankings, no evidence of any kind. "It's simple and evocative and it worked," Rosen confesses. "After all, I ate breakfast there the very next day."[9] The sign is her parable for everything research writing is not, and for why the writing matters: unsupported claims, from clickbait to fake news, persuade all the time (humbuggery sells, and it sells breakfast too). Research communication is the discipline of persuading with the receipts attached.

Her second parable cost her more. When her employer asked her to propose a departmental restructuring, she spent days building a beautiful seven-page report with tables and charts. Her boss flipped through it, tossed it on the table, and said, "I didn't have time to read this. What's your proposal?" The lesson: know your audience. A one-page handout would have won the day. The standard report architecture she then teaches (abstract; introduction that states the findings up front, since "Your report isn't an Agatha Christie mystery novel. Tell us early on what you found"; literature review organized by topic rather than source; methodology; results tailored to the audience's statistical literacy; conclusion; references) is a technology for making evidence usable by the people who need it.

And because the course never loses its humanity, it ends on rejection. A colleague's paper collected six rejections before winning a best-paper award. Twelve publishers rejected J. K. Rowling's first Harry Potter book, and one advised her not to quit her day job. Peer review is fallible in both directions, and persistence is part of the method. Rosen's benediction deserves to be quoted whole: "May your questions be interesting, your process systematic, your results significant, and your cocktail parties entertaining."

Closing the series #

I said at the outset that this course teaches a temperament as much as a toolkit, and the closing lectures bear me out. Every lecture circles back to the same commitments: try to disprove rather than prove; state your limitations before someone else finds them; treat process as the thing that separates knowledge from opinion; and remember that other people (subjects, readers, communities) have claims on how you work. Those commitments travel across every boundary the course crosses, from experiments to ethnography, from regression to thick description. Humanists need not convert to statistics to collect the payoff. It lies in recognizing that our own crafts (source criticism, close reading, archival judgment) are methods too, and that naming them with this kind of clarity makes them teachable, contestable, and stronger. Rosen's course is the rare methods text one could hand with equal profit to a doctoral student, a journalist, or a curious reader who just wants to stop being fooled. These six essays are only a map; the territory is the twenty-four lectures, and it is worth the walk.

Notes #


  1. Amanda M. Rosen, Effective Research Methods for Any Project (Chantilly, VA: The Teaching Company/The Great Courses, 2019). This post draws on Lectures 17–24. ↩︎

  2. Rosen, Effective Research Methods, Lecture 17, "Gathering and Preparing Data." The episode is The Office, season 3, "The Convict" (NBC, 2006). ↩︎

  3. Brett Ashley Leeds et al., "Alliance Treaty Obligations and Provisions, 1815–1944," International Interactions 28 (2002): 237–60, and the ATOP project at Rice University, discussed in Rosen, Lecture 17. ↩︎

  4. Rosen, Effective Research Methods, Lecture 18, "Using Statistics to Interpret Data"; Lecture 19, "Statistical Inferences from Data." ↩︎

  5. Pew Research Center, "Behind Trump's Victory: Divisions by Race, Gender, Education" (November 2016), discussed in Rosen, Lecture 21, "From Bivariate to Multivariate Analysis." ↩︎

  6. Rosen, Effective Research Methods, Lecture 20, "Assessing Correlation and Causation." On Pearson and eugenics, see Theodore M. Porter, Karl Pearson: The Scientific Life in a Statistical Age (Princeton: Princeton University Press, 2004). ↩︎

  7. Rosen, Effective Research Methods, Lecture 22, "Foundations of Qualitative Analysis"; Lecture 23, "Qualitative Analysis Variations." ↩︎

  8. Don H. Zimmerman and Candace West, "Sex Roles, Interruptions and Silences in Conversation," in Language and Sex: Difference and Dominance, ed. Barrie Thorne and Nancy Henley (Rowley, MA: Newbury House, 1975), discussed in Rosen, Lecture 23. ↩︎

  9. Rosen, Effective Research Methods, Lecture 24, "The Art of Presenting Your Findings." ↩︎

Tags : research-methods methodology academia statistics writing

Webmentions

No webmentions yet.

Previous

Research Methods for Everyone, Part 5: Case Studies, Fieldwork, and Research That Acts

Kitty Genovese as urban legend, Geertz at the cockfight, and research built to change the world; I argue that each depth tradition makes a trade, and that rigor lies in declaring it.

Next

Review: Hypertext 3.0

I argue that Landow's third edition remains the fullest statement of the hypertext-theory convergence even as its analogical method converts transcendental arguments into product features; read it now as a primary source about its own moment.