SAT · Problem-Solving and Data Analysis · TOPIC LESSON
Evaluating statistical claims: Observational studies and experiments
Separate how people were selected from how treatments were assigned, then limit the claim accordingly.
SAT35 min study guide9 indexed sections
BY THE END, YOU CAN
Identify populations, samples and likely sources of bias.
Distinguish observational studies from randomized experiments.
Determine which designs support generalization and causal conclusions.
BEFORE YOU START
Meaning of sample, population and variable.
Difference between an association and a causal claim.
Original educational lesson. Work through the examples before revealing the practice answers. This lesson supports SAT Math and Reading and Writing skill study; it does not reproduce official questions or guarantee a score.
LEARN THROUGH ORIGINAL PRACTICE
Original SAT-style worked examples and practice
The teaching passages, worked examples and exercises in this lesson are original. They are not official College Board questions or past-exam reproductions. Blueprint and observed-practice analytics describe separate factual metadata; those source questions are not displayed here.
01 / UNDERSTAND & APPLY
Selection and assignment answer different questions
Random sampling concerns who enters a study. It supports generalizing results to the sampled population when coverage and response are adequate. Random assignment concerns which treatment enrolled participants receive. It helps balance potential confounders and supports a causal comparison when the experiment is well conducted.
A study can have one, both or neither. Random assignment does not turn a convenience sample into a representative population sample. Random sampling alone does not make an observational association causal.
02 / UNDERSTAND & APPLY
Observational studies record rather than assign
An observational study measures existing behavior, exposure or characteristics without assigning the relevant treatment. It can establish an association and motivate hypotheses, but other variables may explain the observed relationship. A confounder is related to both the explanatory and response variables and can distort the comparison.
Reverse causation is another possibility: the proposed response may influence the supposed cause. Strong correlation, a large sample or a small margin of error does not by itself resolve these alternatives.
ORIGINAL WORKED EXAMPLE
A plausible confounding variable
A survey finds that people who buy more running shoes report fewer sick days. Does buying shoes cause better health?
The study records existing behavior; it did not randomly assign purchases.
People who exercise more may both buy more running shoes and have different health outcomes.
The survey supports an association in its sampled group, not the claim that the purchase itself causes better health.
Answer No causal conclusion from this observational association
03 / UNDERSTAND & APPLY
Random assignment supports causal comparison
An experiment assigns a treatment. Random assignment uses a chance process to allocate study units to groups, making treatment groups comparable in expectation. A control or comparison group supplies a baseline; holding other procedures constant helps isolate the treatment difference.
Randomization reduces systematic selection differences but does not guarantee perfectly identical groups in one sample. Noncompliance, attrition, different measurement procedures or other design failures can still weaken the causal interpretation.
ORIGINAL WORKED EXAMPLE
A randomized treatment comparison
Volunteers are randomly assigned to use study method A or B for four weeks, then take the same assessment. What kind of conclusion can a well-conducted comparison support?
The treatment method is assigned rather than merely observed.
Random assignment helps balance preexisting differences between the volunteer groups.
A credible difference can support a causal effect of method in the study setting.
Because volunteers were not randomly sampled from all students, broad population generalization remains limited.
Answer A causal comparison among the studied volunteers, with limited generalization
04 / UNDERSTAND & APPLY
Match the population to the sampling frame
The sampling frame is the list or process from which units can be selected. A random sample of a school’s enrolled students can inform claims about that school, not automatically all students in the country. Undercoverage occurs when part of the target population has no chance to appear in the frame.
A simple random sample gives every possible sample of the chosen size an equal chance of selection. Consequently, each individual unit also has the same chance of being selected; equal individual chances alone do not fully define a simple random sample. A convenience sample takes whoever is easiest to reach. A voluntary-response sample lets motivated respondents choose themselves, often overrepresenting strong opinions.
ORIGINAL WORKED EXAMPLE
Random sampling with a bounded population
A random sample of 300 registered members of a city library is surveyed about opening hours. To which group can results most directly generalize, assuming adequate response?
The sampling frame is registered members of that library.
Random selection supports inference to those registered members.
Residents who are not registered and users of other libraries were not represented by the selection process.
Answer Registered members of the sampled library
05 / UNDERSTAND & APPLY
Recognize systematic errors
Selection bias comes from how participants are chosen. Nonresponse bias occurs when respondents systematically differ from those who were selected but did not respond. Response bias can arise from leading wording, social desirability or inaccurate recall. Measurement bias comes from instruments or procedures that systematically mismeasure.
Random selection at the invitation stage does not eliminate nonresponse bias. Asking a neutral question and pursuing adequate response can matter as much as increasing the number invited.
ORIGINAL WORKED EXAMPLE
A large voluntary poll
A website posts an optional poll about a fee increase and receives 20,000 responses. Why may its result fail to represent all customers?
Participation is voluntary rather than randomly selected from all customers.
People with particularly strong views or frequent website use may be more likely to respond.
The large number of responses does not remove that systematic selection difference.
Answer Voluntary-response and coverage bias may limit representativeness
06 / UNDERSTAND & APPLY
Classify a claim along two separate dimensions
Random sampling plus random assignment can support both population generalization and a causal comparison, subject to sound execution. Random sampling without assignment supports representative association estimates. Random assignment without random sampling supports a causal comparison within a more limited study group. Neither feature usually supports broad generalization or a causal conclusion from the design alone.
A control group without random assignment can still differ systematically from the treatment group. Merely calling a study an experiment does not establish that it used random allocation.
07 / UNDERSTAND & APPLY
Improve the design for the claim you need
To improve representativeness, sample from a frame covering the target population and address nonresponse. To investigate causality when ethical and feasible, randomly assign treatments, use a comparison condition and apply consistent measurement. Blinding can reduce expectation-related behavior or measurement bias where practical.
Some exposures cannot ethically be assigned. An observational design may then be appropriate, but its conclusions still need careful limitations rather than overstated causal certainty.
ORIGINAL WORKED EXAMPLE
Repair the design, not just the sample size
A school wants to know whether a new tutoring program improves test scores. Comparing students who chose tutoring with those who did not may be confounded. What design change would best address this?
Self-selected tutoring groups may differ in motivation or initial achievement.
When ethical and practical, randomly assign eligible consenting students to the tutoring program or a suitable comparison condition.
Use the same outcome measurement and watch for differential dropout.
Random allocation targets confounding; simply adding more self-selected students does not.
Answer Use a well-conducted randomized treatment comparison
08 / UNDERSTAND & APPLY
Keep conclusions no broader than the design
Identify the target population, actual sample, selection method, treatment allocation, measured variables and outcome. Then ask separately: to whom can this result generalize, and does the design support a cause-and-effect claim? Specific wording matters: “was associated with” is weaker than “caused.”
A numerical difference can also arise from sampling variation. Good design creates a basis for inference; it does not make every observed difference automatically meaningful. Use any uncertainty information the study provides.
09 / UNDERSTAND & APPLY
Original practice: what conclusion is justified?
Classify the design before judging the claim.
ORIGINAL PRACTICE · 1
Researchers randomly sample adults and record their existing sleep and exercise habits. Is this a randomized experiment?
Reveal answer & reasoning
No; it is observational
Sampling is random, but researchers did not assign sleep or exercise treatments.
ORIGINAL PRACTICE · 2
Which primarily supports population generalization: random sampling or random assignment?
Reveal answer & reasoning
Random sampling
It addresses whether the studied units represent the target population.
ORIGINAL PRACTICE · 3
Which primarily supports causal treatment comparison?
Reveal answer & reasoning
Random assignment
It helps balance confounders across treatment groups in a well-conducted experiment.
ORIGINAL PRACTICE · 4
A poll asks only people leaving a gym about citywide exercise habits. Name a concern.
Reveal answer & reasoning
Selection or coverage bias
Gym visitors may systematically exercise more than the whole city population.
ORIGINAL PRACTICE · 5
Can a large sample repair a leading survey question?
Reveal answer & reasoning
No
The wording can systematically bias answers regardless of sample size.
ORIGINAL PRACTICE · 6
Volunteers are randomly assigned treatments. Can results automatically generalize to every adult?
Reveal answer & reasoning
No
Random assignment supports the treatment comparison, while volunteer recruitment limits representativeness.
ORIGINAL PRACTICE · 7
A randomly sampled group has very low response, and likely dissatisfied people respond more often. What bias may occur?
Reveal answer & reasoning
Nonresponse bias
Respondents may differ systematically from selected people who do not respond.
ORIGINAL PRACTICE · 8
Ice-cream sales and swimming incidents both rise in hot weather. Name a plausible confounder.
Reveal answer & reasoning
Temperature or season
Hot weather can affect both variables without one directly causing the other.
ORIGINAL PRACTICE · 9
A study randomly samples a school’s students and randomly assigns two teaching methods. What two goals can this design support if well conducted?
Reveal answer & reasoning
Generalization to that school’s students and a causal comparison of the methods
Sampling addresses population representation, while assignment addresses treatment comparability.
KEEP THE BIG IDEAS
Quick revision
Random sampling and random assignment solve different problems.
Observational association alone does not establish causation.
Large sample size does not eliminate systematic bias or expand the sampled population.
The teaching notes, passages, worked explanations and practice are original. Official references identify the assessment framework and skill scope; they do not endorse this lesson or license official question reproduction.
Official skill taxonomy reference. All teaching prose, examples, diagrams and practice in this lesson are original; this is not a past-question collection.
Explore this topic in context
Evaluating statistical claims: Observational studies and experimentsSAT topic analytics ↗
SAT analytics distinguish official blueprint weights from independently classified linear-practice question occurrences. These are not historical administered-exam frequencies.