Hypothesis Testing Basics for Analysts

Hypothesis testing is how analysts answer the question behind every A/B test and experiment: is this difference real, or could it easily have happened by chance? When a new landing page converts at 12% versus the old page's 10%, hypothesis testing tells you whether that two-point gap is a genuine improvement or the kind of wobble you would see from random variation alone. It is the formal bridge from "the numbers look different" to "we can act on this."

The procedure has a fixed shape that never changes across tests, only the arithmetic in the middle differs. Learn the shape once and you can read any test result critically. This page walks through each step and finishes with a runnable example.

The framework, step by step

Every hypothesis test follows the same five moves:

1. State the hypotheses. The null hypothesis (H0) is the boring default: no effect, no difference, nothing going on. The alternative hypothesis (H1) is the claim you are actually interested in: there is an effect. For an A/B test, H0 is "the two pages convert equally" and H1 is "they convert differently."

2. Choose a significance level (alpha). This is your evidence threshold, set before looking at the results, usually 0.05. It is the risk you accept of a false alarm.

3. Compute a test statistic. This number measures how far your data sits from what the null predicts, in standardized units. A t-statistic, z-statistic or chi-square value are common choices.

4. Get the p-value. The p-value is the probability of seeing a result at least this extreme if the null were true. It is covered in depth in p-values explained.

5. Decide. If the p-value is below alpha, reject the null — the result is "statistically significant." If not, fail to reject it. That is the entire decision rule.

The logic is deliberately conservative. You assume nothing is happening (the null) and only abandon that assumption when the data is too surprising to ignore. It mirrors "innocent until proven guilty": you never prove the null, you only fail to overturn it.

Why the null is the starting point

Testing is built around the null because it gives you something concrete to compute against. Under "no difference," you can calculate exactly how much random variation to expect, thanks to the central limit theorem making sample means predictable. You then check whether your observed result fits comfortably inside that expected wobble or falls out in the extreme tail. If it is way out in the tail, either something rare happened or the null is wrong — and you bet on the null being wrong.

Two ways to be wrong

Because you are making a decision under uncertainty, two errors are possible:

• Type I error (false positive): rejecting a true null — declaring an effect that is not real. Its probability is exactly alpha.
• Type II error (false negative): failing to reject a false null — missing an effect that is real. Its probability is called beta, and 1 − beta is the test's power.

There is a tension: lowering alpha to avoid false positives makes false negatives more likely, and vice versa. Increasing sample size is the way to reduce both at once, which is why properly powered tests need enough data.

A worked example: a two-sample t-test

Suppose you run an A/B test on time-on-page (minutes) for two page designs. Here is a small, illustrative sample.

from scipy import stats

group_A = [5.1, 4.8, 5.5, 5.0, 4.9, 5.2]   # old design
group_B = [5.9, 6.1, 5.7, 6.3, 5.8, 6.0]   # new design

t_stat, p_value = stats.ttest_ind(group_A, group_B)
print("t-statistic:", round(t_stat, 3))
print("p-value:    ", round(p_value, 5))

Expected output:

t-statistic: -6.574
p-value:     0.00006

Walk through the framework. H0: the two designs give the same average time-on-page. H1: they differ. Alpha = 0.05. The t-test compares the group means (about 5.08 for A and 5.97 for B) relative to the variation within each group. The resulting p-value, roughly 0.00006, is far below 0.05, so we reject the null: the new design's higher time-on-page is statistically significant and very unlikely to be chance. (These are illustrative sample values, not measured production data.)

How analysts use it

Hypothesis testing is the engine of experimentation. Every A/B test dashboard that flashes "significant" is running one underneath. Analysts use it to validate whether a pricing change moved revenue, whether a new feature changed retention, or whether two customer segments truly differ. The discipline it enforces — commit to alpha up front, state the hypotheses, then let the data decide — is what keeps teams from chasing noise. Pairing the p-value with the effect size (how big the difference is) is the professional habit that separates useful analysis from p-value theater.

Common mistakes

Confusing statistical and practical significance. A tiny, useless difference can be statistically significant with a huge sample. Always report the effect size, not just significance.

Treating "fail to reject" as "the null is true." It only means you lacked evidence. A small or underpowered sample often fails to detect real effects.

Choosing alpha after seeing the p-value. Moving the threshold to get significance (p-hacking) invalidates the test. Fix alpha before you look.

Running many tests and reporting only the winners. Test twenty things at alpha = 0.05 and about one will look significant by chance. Correct for multiple comparisons when testing many hypotheses.

In interviews

Hypothesis testing questions are common in analyst interviews. Expect "Walk me through how you would set up an A/B test" — name the null and alternative, alpha, the test statistic, the p-value, and the decision rule. Be ready to define Type I and Type II errors and to explain that failing to reject is not proof of the null. A frequent scenario asks whether a significant result is worth acting on, where you should raise effect size and sample size. Clear command of the framework, not memorized formulas, is what interviewers reward.

Where this fits in your learning path

Hypothesis testing is where probability basics and the central limit theorem come together into a decision procedure. Its single most misunderstood output, the p-value, gets a full treatment in p-values explained. This is the capstone of the inference section of the data analytics learning path, and A/B testing fluency is a headline skill on the data analyst roadmap.