Handling Missing Values in Pandas

Missing values are the most common defect in real datasets, and the way you handle them quietly changes your results. A blank in a survey, a sensor that dropped a reading, a field a user skipped — each becomes a NaN that can break an average, distort a chart, or crash a model. Handling missing values well means two things: detecting every gap reliably, and making a deliberate choice about each one instead of a reflex delete.

This topic sits right after what data cleaning is and directly feeds into imputation techniques, where filling gets more sophisticated. Get comfortable here first.

How missing data is represented

In pandas, a missing entry is usually NaN (Not a Number), a special floating point marker. Python's None often becomes NaN inside numeric columns, and the database word "null" describes the same concept. The important practical point: isna() detects all of these consistently, so you have one reliable way to find gaps.

What isna() does not catch are disguised placeholders. If an export wrote the literal text "N/A", "-", or "unknown" into cells, pandas sees ordinary strings, not missing values. The first real step in any cleanup is converting those placeholders into genuine NaN so they can be counted and handled.

Detecting missing values

Always profile before you act. Here is an illustrative messy sample and the standard detection commands.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name":  ["Aarti", "Bhaskar", "Chitra", "Devan", "Esha"],
    "age":   [29, np.nan, 41, np.nan, 35],
    "city":  ["Hyderabad", "Pune", "N/A", "Chennai", None],
    "spend": [1200, 980, 550, 760, np.nan],
})

# convert disguised placeholders to real NaN first
df = df.replace("N/A", np.nan)

print(df.isna().sum())              # missing count per column
print("total missing:", df.isna().sum().sum())

Expected output:

name     0
age      2
city     2
spend    1
dtype: int64
total missing: 5

Now you know exactly where the gaps are: two ages, two cities, one spend. That per-column count is the information you need to choose a strategy — a column that is 2% missing is a very different problem from one that is 60% missing.

Choosing: drop or fill

There is no single correct answer, only a decision based on how much is missing and why.

Dropping is appropriate when only a small fraction of rows are affected and removing them will not bias what remains. dropna() removes any row with at least one NaN by default:

few = df.dropna(subset=["spend"])   # drop only rows missing spend
print(len(df), "->", len(few))

5 -> 4

Using subset keeps you from throwing away rows over an unrelated missing column — a common over-deletion mistake. You can also drop columns with axis=1, or only fully empty rows with how="all".

Filling is appropriate when dropping would cost too much data or skew the result. fillna() replaces gaps with a value you choose:

filled = df.copy()
filled["age"]  = filled["age"].fillna(filled["age"].median())
filled["city"] = filled["city"].fillna("Unknown")
print(filled)

      name   age       city   spend
0    Aarti  29.0  Hyderabad  1200.0
1  Bhaskar  35.0       Pune   980.0
2   Chitra  41.0    Unknown   550.0
3    Devan  35.0    Chennai   760.0
4     Esha  35.0    Unknown     NaN

Numeric gaps were filled with the median age, and missing cities became an explicit Unknown category rather than a silent blank. Choosing the median over the mean here is deliberate — it resists distortion from extreme values, a point covered more in imputation techniques.

How analysts decide in practice

The professional workflow is: convert placeholders to NaN, count per column, then triage. Columns missing a tiny share often get row-dropped. Columns missing a moderate share get filled with a sensible statistic or a category. Columns that are mostly empty are often dropped entirely, because filling 70% of a column invents most of it. Above all, ask why the value is missing. A blank income field because respondents refused is not the same as a blank because a form field was added halfway through collection — and that reason should shape whether you fill, drop, or flag it with an indicator column.

Documenting these decisions in your cleaning script matters. Whoever reads the analysis later — including you — needs to know that "Unknown" cities were imputed and how many rows were dropped.

Common mistakes

• Blanket df.dropna(). Called with no arguments it deletes every row with any missing value, which can wipe out most of your data because of one sparse column. Use subset to target specific columns.
• Ignoring disguised nulls. Forgetting to convert "N/A" or "-" to real NaN means your counts are wrong and your fills miss cells entirely.
• Filling numeric gaps with the mean by default. The mean is pulled by outliers; the median is usually safer for skewed columns like income or spend.
• Filling without recording it. Imputed values look identical to real ones later. Note what was filled, ideally with a separate boolean flag column, so the analysis stays honest.

In interviews

"How do you handle missing data?" is a near-guaranteed data analyst interview question. A weak answer is "I drop them" or "I fill with the mean." A strong answer explains that it depends on the amount missing and the reason, names isna(), dropna(subset=...) and fillna(), distinguishes mean from median for skewed data, and mentions preserving the raw file and documenting choices. You may also be handed a messy sample and asked to profile and fix it in pandas, so practice the detection-then-decision flow above until it is automatic.

Where this fits in your learning path

Handling missing values is one of the first hands-on skills in the data cleaning cluster. It builds on the overview in what is data cleaning and leads directly into the deeper strategies of imputation techniques. Together these form a core competency on the data analyst roadmap.