Sampling, Simulation & Inference — Print
A well-chosen thousand can speak for millions, provided you admit how far off you might be.
What the practice questions cover
These questions use 10 ideas from the course. Follow any of them for a definition, where it is taught, and what it leads to.
- Percent — Parts per hundred. 37% = 37100 = 0.37.
- Repeating — Every fraction becomes a decimal that either stops or repeats forever.
- Random sample — Every member has an equal chance of being picked — the only kind you can trust.
- Mean — Two kinds of centre: the balance point, and the middle value.
- Square root — For k ≥ 0, √k is the nonnegative number whose square is k.
- Factor — A number that divides evenly into another.
- Range — The set of outputs that actually occur.
- Association — A pattern linking the two variables.
- Bias — A flaw in how the sample was gathered that tilts the result in one direction.
- Margin of error — How far the sample result might sit from the truth.
A poll asks a thousand people and reports 52%.
There are millions of voters. Nobody asked them. Yet the number gets printed as though it were a fact.
Ask a different thousand and you would get a different answer — maybe 50%, maybe 54%. The number moves.
So the honest report is not a number. It is a range, together with how confident you are that the truth is inside it.
That is inference: using a sample to say something about a population, and being explicit about the uncertainty. The margin of error is not an admission of failure — it is the point.
Three kinds of study, three kinds of conclusion
A survey describes a population. It can tell you what people think, and nothing about why.
An observational study watches without interfering, so it can find associations but never causes.
Only a randomised experiment, where the treatment is assigned by chance, can establish cause.
The claim must match the design
An observational study showing coffee drinkers live longer cannot say coffee helps. Something else may cause both, and only random assignment rules that out.
Why random matters more than large
A well-conducted simple random sample avoids selection bias by design, and sampling error usually shrinks as the sample grows. Nonresponse and measurement errors can still introduce bias.
A biased sample is wrong in a fixed direction, and collecting more of it makes the wrong answer more precise, not more true.
Asking only people who answer a landline at noon tells you about that group, however many you ask.
Size cannot rescue bias
Two million biased responses beat nobody. A thousand random ones beat two million biased ones.
The margin of error
For a proportion from a simple random sample, a rough conservative margin-of-error shortcut is 1√n at 95% confidence.
A thousand people gives about 1√1000 ≈ 3%, which is why so many polls report exactly that.
To halve the margin you need four times the people, because of the square root.
What 95% confidence means
Not that there is a 95% chance the truth is in this interval. It means that if you repeated the whole process many times, 95% of the intervals built this way would contain it.
Reading a result honestly
A poll at 52% with a 3% margin means the truth is plausibly anywhere from 49% to 55%.
Since that range includes 50%, the poll does not establish a lead. "Too close to call" is the accurate phrase.
Overlapping individual intervals alone do not decide whether two candidates differ. To compare them, analyse the difference directly, accounting for how the estimates are related.
Check whether the interval crosses the line
Whatever value would change the conclusion — 50% here — ask whether it is inside the interval. If it is, the data has not decided.
Simulation instead of formulas
If a formula is out of reach, simulate: model the situation with coins, dice or a random number generator and run it many times.
Choose the alternative first: unusually many heads is one-sided; unusually many heads or tails is two-sided. Count how often the assumed model produces a result at least that extreme. If that is rare, chance is a poor explanation.
This is how a statistician decides whether a coin is loaded, or a treatment really worked.
What "rare" usually means
Under 5% is the common threshold. It is a convention, not a law of nature, and it is worth remembering that one result in twenty clears it by luck.
Worked examples
Reading a poll
A poll of 1000 reports 52% support with a margin of error of 3%. Can the candidate claim a lead?
- The interval is 52 ± 3, so 49% to 55%. Estimate plus and minus the margin.
- That range includes 50%. 50% is the value that decides a lead.
- So the data does not establish a lead. The true support could plausibly be below half.
No — the interval includes 50%
How many to ask
A margin of error of about 3% needs roughly what sample size? What about 1.5%?
- Margin ≈ 1√n, so 0.03 = 1√n gives √n ≈ 33. Rearrange for √n.
- So n ≈ 1100. Square it.
- Halving the margin needs four times the sample: about 4400. The square root is why it is four times, not two.
about 1100, then about 4400
Judging a claim
A coin lands heads 62 times in 100 flips. A simulation of fair coins gives 62 or more heads in 1.0% of trials. What can you say?
- If the coin were fair, this result would show up about once in a hundred runs. That is what the simulation measured.
- 1% is below the usual 5% threshold. Rare enough to doubt the assumption.
- So there is good evidence the coin is not fair. Evidence, not proof — one run in a hundred does happen.
Good evidence the coin is biased
Words to know
- Population
- Everyone or everything you want to know about.
- Random sample
- A sample where every member had an equal chance of being chosen.
- Bias
- A flaw in how the sample was gathered that tilts the result in one direction.
- Margin of error
- How far the sample result might sit from the truth.
- Confidence interval
- The range formed by the estimate plus and minus the margin of error.
Mistakes to watch for
Thinking a big sample fixes bias. The 1936 Literary Digest poll asked two million people and still called the election wrong, because it only reached car and phone owners. Randomness matters more than size.
Claiming cause from an observational study. Finding that coffee drinkers live longer does not mean coffee helps — something else may cause both. Only random assignment of the treatment supports a causal claim.
Reporting the estimate without the margin. "52% support" sounds decisive; "49% to 55%" shows the race is open. The margin is part of the result, not a footnote.
Reading 95% confidence as a probability about this interval. It describes the method: repeat the process many times and 95% of the intervals would capture the truth. This particular one either does or does not.
Using interval overlap as a universal significance test. If one candidate is 49–55% and the other 45–51%, overlap alone does not settle whether they differ. Test the difference itself; estimates from the same poll are related.
The 1936 Literary Digest poll asked two million people and still got the election wrong, because it surveyed car and telephone owners. A huge biased sample is worse than a small fair one.
Quadrupling the sample size only halves the margin of error, because the margin shrinks with the square root. That is why polls stop at about a thousand — beyond it, accuracy gets expensive fast.
Simulation lets you answer questions with no formula at all: shuffle the data ten thousand times and count how often chance alone produces what you saw.
Problem 1
Which study can show that a treatment causes an effect?
- A randomised experiment
- A survey
- An observational study
- A case study
Worked answer
A randomised experiment
Random assignment rules out other explanations
Problem 2
A poll is taken only outside a gym. What is the problem?
- The sample is biased
- The sample is too large
- The margin of error is too small
- Nothing
Worked answer
The sample is biased
Gym-goers are not typical of everyone
The tilt is systematic, not random
Problem 3
A poll reports 48% with a margin of error of 4%. Give the lower end of the interval.
Worked answer
44
48-4=44
Problem 4
Same poll. Give the upper end.
Worked answer
52
48+4=52
Problem 5
Why is a random sample preferred?
- It has no systematic tilt
- It is always larger
- It is cheaper
- It avoids the need for a margin of error
Worked answer
It has no systematic tilt
Random selection gives every member an equal chance
So errors are noise, not bias
Problem 6
Estimate the margin of error for a random sample of 400, as a percent.
Worked answer
5
1√400=120=0.05
Problem 7
Estimate the margin of error for a sample of 2500, as a percent.
Worked answer
2
150=0.02
Problem 8
About what sample size gives a margin of error of 5%?
Worked answer
400
√n = 20
n=400
Problem 9
A poll gives 52% with margin 3%. Can a lead be claimed?
- No — the interval includes 50%
- Yes — 52 is above 50
- Yes — the margin is small
- Not enough information
Worked answer
No — the interval includes 50%
The range is 49% to 55%
It contains 50%, so the data has not decided
Problem 10
To halve a margin of error, by what factor must the sample size grow?
Worked answer
4
Halving 1√n needs √n to double
So n must quadruple
Problem 11
A study finds coffee drinkers live longer. What can be concluded?
- There is an association
- Coffee causes longer life
- Longer life causes coffee drinking
- Nothing at all
Worked answer
There is an association
This is observational
Association can be reported; cause cannot
Problem 12
A sample of 100 gives 60%. Give the lower end of the 95% interval, as a percent.
Worked answer
50
Margin =10%
60-10=50
Problem 13
A simulation of fair coins gives a result as extreme as the observed one in 0.8% of trials. What follows?
- Good evidence the coin is not fair
- The coin is definitely fair
- Nothing can be said
- The simulation was wrong
Worked answer
Good evidence the coin is not fair
0.8% is well under 5%
Chance is a poor explanation for what was seen
Problem 14
What does 95% confidence mean?
- Repeating the process, 95% of such intervals would capture the truth
- There is a 95% chance the truth is in this interval
- 95% of people agree
- The sample was 95% accurate
Worked answer
Repeating the process, 95% of such intervals would capture the truth
Confidence is a property of the procedure over many repetitions
Problem 15
One candidate is 49 to 55%, the other 45 to 51%. Do the intervals overlap? Answer yes or no.
Worked answer
yes
49 to 51 is in both
So the data has not separated them
Problem 16
A 1936 poll asked two million people and still called the election wrong. Why?
- The sample was biased
- The sample was too small
- The margin of error was miscalculated
- The votes were miscounted
Worked answer
The sample was biased
It surveyed car and telephone owners, who were wealthier than average
Size cannot rescue a biased sample