Sampling & Bias

Understanding Sampling & Bias

Before choosing rows, decide whose experience you want to describe. Suppose a service wants the average satisfaction score of everyone whose subscription ended this month, including people who renewed and people who left. That entire set is the target population. The sample is the subset actually measured. The sampling frame is the list from which you can select people. A current-subscriber directory is an incomplete frame for this question because it excludes former subscribers.

Sampling error and selection bias are different problems. Sampling error is the chance variation produced by observing only part of a population. Selection bias arises when the selection process systematically favors observations that distort the quantity being estimated. Increasing a well-designed sample generally reduces sampling uncertainty. Increasing an active-users-only sample does not restore missing former users. A biased method can occasionally give the correct answer by coincidence; bias describes its systematic behavior, not every individual result.

In simple random sampling without replacement, every possible subset of the requested size has the same chance of selection. Each person can appear only once. With a complete frame and accurate measurements, the sample mean is unbiased for the population mean: averaging its estimates across all possible samples gives the population mean. One particular sample can still be far off. Random selection from an incomplete frame only represents that frame, not automatically the intended population.

Stratified sampling first divides the frame into nonoverlapping groups, then samples randomly within each group. Proportional allocation matches the sample's group shares to population shares, subject to integer rounding. Disproportionate allocation can deliberately include more members of a small group to support subgroup analysis. For an overall mean, combine subgroup means using population shares rather than blindly averaging every sampled row. This requires credible group counts and representative observations within each group.

Stratification addresses chosen group differences, not every source of bias. Nonresponse can distort results after random invitations; survivorship bias excludes people or organizations that disappeared before observation. Incorrect answers introduce measurement problems even in a census. Sampling describes populations, but does not establish why groups differ. Always connect the estimate to its population, collection process, and remaining exclusions.

Practical walkthrough

Use Python 3.10 or later; only standard-library modules are needed. Familiarity with lists, comprehensions, and arithmetic means is sufficient. The example creates eight fictional subscribers in memory. Six renewed and have score 8; two left and have score 0. These deliberately uniform group scores isolate the effect of group composition. They are teaching assumptions, not observations about real subscribers.

First calculate the population mean. Usually this is unknown, but a miniature population lets us check estimation methods directly. Next, construct a renewed-only frame and calculate its mean. Even measuring everyone in that frame misses the target because both departed subscribers are absent.

Then enumerate every possible two-person sample from the complete population. There are 28 unordered pairs. Calculating each pair's mean reveals sampling variability without relying on one lucky random draw. Averaging those 28 estimates demonstrates the unbiasedness of this particular simple random sampling design. Enumeration is practical here only because the population is tiny; real sampling does not require listing every possible sample.

Finally, draw three renewed subscribers and one departed subscriber without replacement. This matches the population's three-to-one ratio. Compare it with a deliberately balanced sample containing one person from each group. Correct the balanced estimate by weighting the group means by six-eighths and two-eighths.

The supplied pandas guide organizes related operations under random sampling and split-apply-combine grouping. Here, plain Python exposes the arithmetic directly. Record the frame, allocation, and exclusions alongside any reported estimate: a reproducible selection procedure alone cannot justify claims about people it never had a chance to include.

Worked example

from itertools import combinations
from random import Random
from statistics import mean

# Each tuple represents a distinct fictional subscriber.
population = [(i, "renewed", 8) for i in range(6)]
population += [(i, "left", 0) for i in range(6, 8)]
renewed = [row for row in population if row[1] == "renewed"]
left = [row for row in population if row[1] == "left"]

def score_mean(rows):
    return mean(row[2] for row in rows)

pair_means = [score_mean(pair) for pair in combinations(population, 2)]
rng = Random(7)
proportional = rng.sample(renewed, 3) + rng.sample(left, 1)
balanced = rng.sample(renewed, 1) + rng.sample(left, 1)
weighted = (
    len(renewed) / len(population) * score_mean(balanced[:1])
    + len(left) / len(population) * score_mean(balanced[1:])
)

print(f"Population mean: {score_mean(population):.1f}")
print(f"Renewed-only mean: {score_mean(renewed):.1f}")
print(f"Possible two-person samples: {len(pair_means)}")
print(f"Average of all pair means: {mean(pair_means):.1f}")
print(f"Pair mean range: {min(pair_means):.1f} to {max(pair_means):.1f}")
print(f"Proportional sample mean: {score_mean(proportional):.1f}")
print(f"Balanced sample, unweighted: {score_mean(balanced):.1f}")
print(f"Balanced sample, weighted: {weighted:.1f}")

Expected output

Population mean: 6.0
Renewed-only mean: 8.0
Possible two-person samples: 28
Average of all pair means: 6.0
Pair mean range: 0.0 to 8.0
Proportional sample mean: 6.0
Balanced sample, unweighted: 4.0
Balanced sample, weighted: 6.0

The population total is 48, so its mean is 6. Renewed-only coverage gives 8. Among the 28 pairs, 15 contain two renewed subscribers, 12 contain one from each group, and one contains two departed subscribers. Their means average to (15 × 8 + 12 × 4 + 1 × 0) / 28 = 6. Proportional sampling gives (8 + 8 + 8 + 0) / 4 = 6. Equal group counts instead give 4; population weighting restores 6. Output is deterministic because scores are constant within groups. Real groups vary internally, so stratified estimates generally retain sampling uncertainty.

Common mistakes and how to avoid them

• Treating a large convenience sample as representative.

  Audit who could enter the frame and who responded. Expand coverage or narrow the stated target population; collecting more of the same excluded-group-free data does not fix coverage.

• Assuming stratified sampling always preserves population proportions.

  Specify the allocation explicitly. If smaller groups are oversampled, use population-share-weighted subgroup means for the overall estimate, and retain the group counts needed to calculate those weights.

• Believing a fixed random seed removes bias.

  Use a seed to reproduce selection, not to certify representativeness. Inspect the frame and response process separately. Never search seeds until a sample produces a preferred conclusion.

Review questions

Further reading

• pandas user guide
• scikit-learn user guide