Research Methods for Everyone, Part 3: Design, Measurement, and Sampling

In this post, the third in a six-part series on Amanda M. Rosen's Great Courses series Effective Research Methods for Any Project, I cover lectures seven through nine, the architectural stage of research: choosing a design, deciding how to measure your concepts, and selecting the cases you will actually study.^[1] Part two followed the path from topic to question to hypothesis. I will argue that these three lectures form a unit because each one converts an act of judgment into an inspectable procedure; these are the decisions that determine, before you collect a single data point, what you will be entitled to claim at the end.

The menu of designs #

Rosen organizes the design landscape along the quantitative-qualitative axis introduced earlier in the course, while insisting the boundary is soft. Quantitative analysis applies statistical tools to large amounts of numerical data across many cases; qualitative analysis studies a smaller number of cases in more depth, using data that resists reduction to numbers. "Quantitative methods tend to sacrifice depth for breadth," she says, and the reverse holds for qualitative work.^[2]

On the quantitative side of the menu sit experiments, observational designs, and large-scale surveys. Experiments, which get two full lectures of their own (I cover them in part four of this series), earn her strongest endorsement where they are possible: "If you can answer your research question using a proper experiment, you probably should." Observational designs come in cross-sectional form, examining many cases at a single point in time, and longitudinal form, following cases across time; the two can be combined, as when researchers track children from multiple regions from birth to adulthood. On the qualitative side sit case studies, field research, action research, and evaluation research. Here Rosen introduces Clifford Geertz (1926–2006), the American anthropologist whose participant observation at Balinese cockfights produced the "thick description" that became a model for interpretive social science.^[3] We will meet him again in part five.

What I appreciate most in this lecture is its refusal to pretend the choice is purely intellectual. Rosen runs through the practical constraints with unusual honesty. Sometimes you cannot manipulate the variable you care about: "I'd love to be able to convince, say, North Korea to try democracy for a while, just for the sake of research." Sometimes the data does not exist for ethical reasons, as with interviewing domestic violence victims who still live with their abusers. Sometimes the constraint is skill, or money; the National Science Foundation, she notes, funds roughly 11,000 of the 50,000 proposals it receives each year. And sometimes existing data solves your problem: the General Social Survey has been asking Americans about everything "from premarital sex to your television habits" since 1972.^[4] The design you choose should fit your question first, but it must also fit your circumstances; pretending otherwise produces abandoned projects rather than rigorous ones.

Two vocabulary items from this lecture recur constantly in what follows. The unit of analysis is the person, organization, or country about which you gather data, and confusion about it wrecks studies. If you study employee productivity, is your unit the individual or the office? The star employee at the worst office will not save its numbers. And the mixed-method approach, combining quantitative breadth with qualitative depth, gets Rosen's quiet endorsement as often the strongest option of all.

Measurement: from vibes to variables #

Lecture eight opens with a card trick of sorts. Deal six cards and ask how to organize them. By suit? By color? Odd versus even? Primes? "Sixes and not sixes"? The same six cards support a dozen groupings, all valid for different purposes. That exercise is measurement in miniature: the world does not come pre-organized, and "measurement is what takes us from vague and subjective feelings about a concept to something concrete and objective that invites comparison."^[5]

The procedure has three steps. First, define your concept. Defining is harder than it sounds, and her car safety example shows why. Is a safe car one that minimizes injury in a crash, or one that avoids crashes? Those two conceptual definitions point toward different cars, and a dealership selling sheer vehicle weight benefits from your not noticing the difference. Second, operationalize: determine what indicators will show you where a case falls on your variable. Third, evaluate your measure for reliability and validity.

Along the way the course introduces the four levels of measurement that every statistics student eventually memorizes, and it gives them intuitive content. 1) Nominal variables mark difference without rank, like marital status ("Is divorced more or less married than widowed?"). 2) Ordinal variables add rank without distance, like military ranks. 3) Interval variables add meaningful distances without a true zero, like temperature. 4) Ratio variables add a true zero that means absence, like income. The practical rule follows: collect data at the most precise level you can, because you can always collapse categories later but can never recover precision you did not record.

Operationalization is also where hidden values sneak into ostensibly neutral research, and Rosen chooses her examples to sting. Merriam-Webster's first definition of healthy, "free of disease," sounds reasonable until you learn that some ninety percent of Americans carry the virus behind the common cold sore, which would make nearly everyone unhealthy by definition. Partisans who condemn flip-flopping in the other party's candidates while excusing it in their own are running a measurement scam (a small-time grift), applying indicators they never fixed in advance. The remedy is procedural: define your indicators before you see the cases, and hold yourself to them. She also introduces the index, which combines multiple items measured on the same scale into a single number, the way five yes-or-no health questions can sum into a health score. Every composite measure you have ever cited, from democracy indices to well-being rankings, is built this way, and it inherits every choice its makers made.

Reliability and validity get the clearest illustration in the course. A scale that returns five wildly different weights within a minute is unreliable. A new scale that reads 200 pounds five times in a row when you weigh 150 is perfectly reliable and completely invalid. "Reliability is necessary, but not sufficient, for validity." The example that stuck with me, as someone who works on religious texts, is her aside on the camel and the eye of the needle. In Greek, the words for camel and rope differ by a single letter, and in Aramaic they are the same word, so the famous verse may rest on a scribal slip. Rosen frames this transmission puzzle, delightfully, as "a problem of reliability." Textual criticism, in short, is measurement theory with older instruments.

Sampling: why one elephant isn't a herd #

Lecture nine begins with the elephant in the room and immediately asks why we are talking about one elephant and not the whole herd. The population is the universe of cases to which your research applies; a sample is any subset of it; a case is a single unit. The entire enterprise of sampling exists because "we should always be suspicious of simple answers without good data and theory," including the simple answers our own experience whispers to us.^[6]

Rosen's confessional example is perfect. Her feet kept going numb in spinning class. A friend advised watching for inward-collapsing knees, she adjusted, and the pain dropped. Success? She then lists half a dozen rival explanations: she also adjusted her feet, wiggled her toes, changed drills, started a new vitamin, and skipped breakfast. Her thank-you to the friend was a methodologist's thank-you: "I have no idea if your advice is responsible or if it applies to anyone else, but I appreciate it all the same." One case, many variables, no control. Generalizing from it would be bad science, and she draws the line explicitly: generalizing carelessly from small unrepresentative samples to whole groups of people is stereotyping, and "stereotyping is bad science and dangerous."

The machinery for doing better is probability sampling, in which every unit has a known chance of inclusion. Simple random samples give every unit an equal chance; stratified samples divide the population into strata and sample proportionally within them; cluster samples randomly select groupings, like cities, then sample within those; systematic samples take every third or tenth or hundredth unit. All of them depend on a sampling frame, an observable list of your population's members, and any unit missing from the frame has zero chance of selection, which is where bias creeps in. Sample size, meanwhile, is a solved problem for most purposes: for a population like the Fortune 500, a five percent margin of error at 95 percent confidence requires a sample of 217.

Two further concepts round out the toolkit. Sampling error is the difference between a sample statistic and the true population value, and it never fully disappears; honest polls report margins of error for precisely that reason, a thread part four picks up. Qualitative work has its own sampling arithmetic: small-N studies typically run from two to twenty cases, more variables demanding more cases, with the single case study as its own genre. The rule Rosen emphasizes for any probability sample is that once you draw a case, it stays. Discarding the cases you happen not to like, or resampling until the draw looks congenial, reintroduces through the back door exactly the bias randomization exists to expel.

The alternatives, non-probability samples, are not forbidden; they are limited, and the researcher must state the limits. Convenience samples, like street-corner interviews or the call-in votes for American Idol, are self-selecting. Quota samples fill demographic targets but inherit convenience sampling's flaws. Purposive samples deliberately choose cases for a reason, like the Fortune 500's top ten. Snowball samples, in which interviewees supply further names, are sometimes the only way to reach hidden populations; one of Rosen's students used snowball sampling to study Bosnian refugees, a population for which no frame exists. The closing counsel is the course's ethic in one line: transparency about your limitations beats hiding them.

The common thread #

These three lectures form a unit, as I argued at the outset, because each one converts an act of judgment into an inspectable procedure. Design choice makes your trade-offs explicit before you are invested in the results. Operationalization forces you to say in advance what evidence would place a case in one category rather than another, so you cannot quietly move the goalposts. Sampling logic makes the reach of your conclusions a matter of arithmetic rather than rhetoric. None of this machinery removes judgment from research; every step involves choices a reasonable colleague could contest. What the procedures do is make the choices visible, and therefore contestable, and therefore correctable. That visibility, more than any particular technique, is what Rosen means by systematic inquiry.

Humanists have analogues for all of it, mostly unnamed. When a historian decides which archives count, she is building a sampling frame. When a literary scholar defines "sentimental novel" before counting them, she is operationalizing. Making those moves explicit, as this course keeps demonstrating, is not scientism; it is simply showing your work.

In the next post, I will take up the classic experiment and its quasi-experimental cousins, the dark art of survey design, and what actually went wrong with the polls in 2016.

Notes #

---

1. Amanda M. Rosen, Effective Research Methods for Any Project (Chantilly, VA: The Teaching Company/The Great Courses, 2019). This post draws on Lecture 7, "Selecting a Research Design," Lecture 8, "Measuring Concepts and Phenomena," and Lecture 9, "Choosing Populations, Samples, and Cases." ↩︎

2. Rosen, Effective Research Methods, Lecture 7. ↩︎

3. Clifford Geertz, "Deep Play: Notes on the Balinese Cockfight," in The Interpretation of Cultures (New York: Basic Books, 1973), discussed in Rosen, Lecture 7. ↩︎

4. NORC at the University of Chicago, General Social Survey, conducted since 1972, gss.norc.org. ↩︎

5. Rosen, Effective Research Methods, Lecture 8. ↩︎

6. Rosen, Effective Research Methods, Lecture 9. ↩︎