Imputation Techniques for Missing Data

Imputation is the deliberate filling of missing values with estimated ones, chosen because deleting the rows would cost too much data or bias the result. It is the natural next step after handling missing values: once you have decided a gap should be filled rather than dropped, imputation is how you fill it well. The technique you pick — mean, median, mode, forward fill, or a group-based estimate — changes how faithful the filled column stays to reality.

The core tension to keep in mind: every imputed value is a guess. A good technique makes an informed guess that preserves the column's real shape; a careless one flattens the data and hides the truth.

The main techniques

Mean replaces gaps with the column average. It is simple but fragile — a few large values pull the mean upward, so imputing with it can misrepresent a skewed column.

Median replaces gaps with the middle value. It ignores extremes, which makes it the safer default for real-world numeric columns like income, price or spend.

Mode replaces gaps with the most frequent value. It is the go-to for categorical columns, where "average" makes no sense.

Forward / backward fill carries a neighbouring value into the gap. Forward fill (ffill) pushes the last known value down; backward fill (bfill) pulls the next one up. These only make sense on ordered data such as time series.

Group-based imputation fills using a subgroup's statistic — the median salary of a person's department rather than of the whole company. It is the most accurate general approach when groups genuinely differ.

Worked example in pandas

Here is an illustrative messy sample showing several techniques side by side.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name":   ["Aarti", "Bhaskar", "Chitra", "Devan", "Esha", "Farhan"],
    "dept":   ["Sales", "Sales", "Tech", "Tech", "Sales", "Tech"],
    "grade":  ["A", None, "B", "B", None, "A"],
    "salary": [50000, 52000, np.nan, 71000, 49000, np.nan],
})

# median for a skewed numeric column
df["salary_median"] = df["salary"].fillna(df["salary"].median())

# mode for a categorical column
df["grade"] = df["grade"].fillna(df["grade"].mode()[0])

# group-based: fill salary with the median of the person's dept
df["salary_bydept"] = df["salary"].fillna(
    df.groupby("dept")["salary"].transform("median")
)

print(df[["name", "dept", "grade", "salary", "salary_median", "salary_bydept"]])

Expected output:

      name   dept grade   salary  salary_median  salary_bydept
0    Aarti  Sales     A  50000.0        50000.0        50000.0
1  Bhaskar  Sales     A  52000.0        52000.0        52000.0
2   Chitra   Tech     B      NaN        50500.0        71000.0
3    Devan   Tech     B  71000.0        71000.0        71000.0
4     Esha  Sales     A  49000.0        49000.0        49000.0
5   Farhan   Tech     A      NaN        50500.0        71000.0

Look at Chitra and Farhan, both in Tech. The global median fill gave them 50,500 — a number pulled down by Sales salaries. The department-based fill gave them 71,000, the only known Tech salary, which is far closer to reality. That gap is exactly why group-based imputation beats a blind global statistic when subgroups differ. The mode()[0] on grade filled the two missing grades with the most common value, "A".

How analysts use imputation

In practice you rarely impute a whole dataset one way. You go column by column. Numeric columns that are roughly symmetric can take the mean; skewed ones take the median. Categorical columns take the mode, or an explicit Unknown when the missingness itself is meaningful. Time-ordered columns take forward fill. When a natural grouping exists — department, region, product category — group-based imputation almost always produces more believable values than a single global number.

Analysts also protect honesty by adding an indicator column, for example salary_was_missing, before filling. That way any later analysis can see which values were real and which were estimated, and you can check whether "missingness" itself correlates with something interesting.

df["salary_was_missing"] = df["salary"].isna()
df["salary"] = df["salary"].fillna(
    df.groupby("dept")["salary"].transform("median")
)
print(df[["name", "salary", "salary_was_missing"]])

      name   salary  salary_was_missing
0    Aarti  50000.0               False
1  Bhaskar  52000.0               False
2   Chitra  71000.0                True
3    Devan  71000.0               False
4     Esha  49000.0               False
5   Farhan  71000.0                True

The flag column preserves the truth that Chitra's and Farhan's salaries were imputed, so no later reader mistakes an estimate for a recorded value. This small habit is the difference between a defensible analysis and a misleading one.

Common mistakes

• Mean-imputing skewed columns. Income, spend and price are almost always right-skewed. The mean overstates the typical value; use the median.
• Forward-filling unordered data. ffill only makes sense when row order carries meaning, like dates. On a randomly ordered table it copies arbitrary neighbours and invents structure.
• Ignoring groups. Filling every department's missing salary with one company-wide number erases real between-group differences. Use groupby().transform() when groups matter.
• Over-imputing. Filling a column that is 70% missing means you fabricated most of it. Sometimes dropping the column, or flagging rather than filling, is the honest choice.
• Hiding the imputation. Filled values look identical to real ones. Without a flag column or documentation, nobody can audit the result.

In interviews

Interviewers often push past "how do you handle missing data" into "which imputation would you use and why." Be ready to justify median over mean for skewed columns, mode for categoricals, and forward fill only for ordered data. Mentioning group-based imputation with groupby().transform() signals real experience, as does noting that heavy imputation can shrink variance and bias models. If given a coding task, show the column-by-column approach rather than one blanket fillna.

Where this fits in your learning path

Imputation is the deeper half of missing-data handling in the data cleaning cluster. Come here after handling missing values, and make sure the foundations from what is data cleaning are solid. Confident imputation is a distinguishing skill on the data analyst roadmap, because it is where judgement, not just syntax, shows.