Variance and Standard Deviation

Two datasets can share the exact same mean and still be completely different. One might hug that average tightly while the other swings wildly around it. Variance and standard deviation are the numbers that capture this difference — they measure how spread out the data is. Without a spread measure, an average is only half the story.

Standard deviation in particular shows up everywhere an analyst works: quality control, risk, A/B test results, and the empirical rule for normal distributions all depend on it. This page derives both measures from scratch, computes them by hand on a small dataset, and clears up the population-versus-sample confusion that trips up most beginners.

From deviations to variance

Start with the intuition. To measure spread, you want the average distance of each value from the mean. The naive approach — average the raw differences — fails, because values above and below the mean cancel exactly to zero. The fix is to square each difference before averaging. Squaring removes the sign and, as a bonus, punishes big deviations more than small ones.

That average of squared deviations is the variance. Its one drawback is units: if the data is in rupees, the variance is in rupees-squared, which nobody can interpret. So we take the square root to get back to the original units. That square root is the standard deviation — the measure you actually report.

The formulas:

mean            = sum(x) / n

population variance  = sum( (x - mean)^2 ) / n
population std dev   = sqrt(population variance)

sample variance      = sum( (x - mean)^2 ) / (n - 1)
sample std dev       = sqrt(sample variance)

A full hand calculation

Let us work through a clean, illustrative sample where the numbers come out tidy: the values 2, 4, 4, 4, 5, 5, 7, 9.

Step 1 — the mean. The sum is 2 + 4 + 4 + 4 + 5 + 5 + 7 + 9 = 40, and there are 8 values, so the mean is 40 / 8 = 5.

Step 2 — squared deviations from the mean of 5:

value   (value - 5)   (value - 5)^2
  2         -3              9
  4         -1              1
  4         -1              1
  4         -1              1
  5          0              0
  5          0              0
  7          2              4
  9          4             16
                     sum = 32

Step 3 — variance and standard deviation.

• Population variance = 32 / 8 = 4, so population standard deviation = √4 = 2.
• Sample variance = 32 / (8 − 1) = 32 / 7 ≈ 4.571, so sample standard deviation ≈ √4.571 ≈ 2.138.

A population standard deviation of 2 means that, on average, values sit about 2 units away from the mean of 5 — which matches the eyeball impression that most values fall between 3 and 7.

Verifying in NumPy

import numpy as np

data = np.array([2, 4, 4, 4, 5, 5, 7, 9])

print("mean:", data.mean())                    # 5.0
print("pop var:", data.var())                  # 4.0   (divides by n)
print("pop std:", data.std())                  # 2.0
print("sample var:", data.var(ddof=1))         # 4.5714...
print("sample std:", data.std(ddof=1))         # 2.1380...

Expected output:

mean: 5.0
pop var: 4.0
pop std: 2.0
sample var: 4.571428571428571
sample std: 2.138089935299395

Notice ddof=1 ("delta degrees of freedom") switches NumPy from the population formula to the sample formula. This single argument is the source of countless mismatched-number bugs.

Population versus sample, plainly

Use the population formula (divide by n) only when your data covers every member of the group you care about — all 30 students in one class, for example. Use the sample formula (divide by n − 1) when your data is a sample drawn to estimate a larger population, which is the far more common situation in analytics.

Why n − 1? A sample tends to be a little less spread out than the full population it came from, so dividing by n would systematically underestimate the true variability. Subtracting one from the denominator nudges the estimate upward to compensate. This is Bessel's correction. For large n the difference is tiny, but for small samples it matters, and interviewers love asking about it.

Common mistakes

Mixing library defaults. NumPy defaults to population (n); pandas defaults to sample (n − 1). Compute the same column two ways and you get two answers. Decide which you want and pass ddof explicitly when in doubt.

Comparing standard deviations across different scales. A standard deviation of 500 is huge for exam scores but trivial for annual salaries. To compare spread across variables on different scales, use the coefficient of variation (standard deviation divided by the mean) instead.

Forgetting the units on variance. Reporting "the variance of price is 40,000" is nearly meaningless because the unit is rupees-squared. Report the standard deviation instead so the number lives in the same units as the data.

Letting outliers dominate silently. Because deviations are squared, a single extreme value can inflate the standard deviation dramatically. When outliers are present, report the interquartile range from percentiles and quartiles as a robust alternative.

In interviews

Standard deviation questions test both mechanics and intuition. Expect "Why do we square the differences instead of taking absolute values?" (to keep the math smooth and penalize large deviations, and because absolute differences would sum to zero without squaring). Expect "What is the difference between population and sample variance, and why n − 1?" A live calculation on four or five values is common, so practice the three-step routine: mean, squared deviations, divide. Being able to explain that standard deviation is just "average distance from the mean, in the data's own units" signals real understanding.

Where this fits in your learning path

Variance and standard deviation are the spread half of descriptive statistics, complementing the center measures in mean, median and mode. Standard deviation becomes even more powerful once you reach the normal distribution, where the 68-95-99.7 rule translates it directly into probabilities. All of this sits inside the data analytics learning path, and it is foundational for the statistics section of the data analyst roadmap.